# Vitna | Documentation Every page, concatenated. Index at https://vitna.ai/docs/llms.txt. # Cite a figure Every price on every board carries a permanent, dated citation. This page is what that citation is, how to get one, and how long it will keep resolving, which is a promise rather than an intention and is written here because it has to exist before the first citation is minted rather than after. ## Getting one Every figure the record holds carries a `cite` mark beside it, on the boards, on a sku's own page and on every period in its ledger. The mark leads to that figure's citation: ``` https://vitna.ai/cite/{id} ``` `{id}` is the [observation](concepts/observation.md) the figure is, which is also what `/api/v1/observations/{id}/proof` takes and what every tape row carries as `observationId`. That page holds the figure, the document it was read from, the moment it was read, the signed day it sits in, and the citation itself in three shapes. A figure with no mark is one this copy of the site cannot cite. Citations resolve out of the record, and an instance serving the committed snapshot has no leaves to name; `vitna.ai` serves the record, so this affects local checkouts and mirrors rather than the site. ## The three shapes They differ by how much identifier each can afford, and the difference is reasoned rather than decorative. | Shape | For | Carries | |---|---|---| | Short reference | A slide, a chart footer | The figure, the date, the url | | Footnote | A vendor review, a memo, a report | The document's short hash and the signed day, and what the signature does not cover | | Contract schedule | A schedule or an annex | The document's full sha256 and the source url | A footnote takes this shape. It is an illustration of the form rather than a live citation: the figure, the hash and the id below are stand-ins, and the real ones come off a board. > Anthropic Claude Sonnet 5, input price: $3.00 per 1M tokens. Read by Vitna > from www.anthropic.com on 23 Aug 2026 at 09:00 UTC, document sha256 4f2a..., > in the signed day 2026-08-23. https://vitna.ai/cite/{id}. No provider signs > its own pricing page: the signature covers Vitna's record of the fetch and > not the provider's page. ## A derived figure carries its arithmetic Some figures are not printed on any page. A provider that sells a sandbox by the 20-minute session has an hourly rate implied and unpublished, and Vitna computes it. The record refuses such a figure without the sum that produced it, and the citation carries that sum in every shape including the short one. One the record holds today, in the clause a citation prints it in: > Derived rather than read: $1.92 per 20-minute session for a 64 GB container, > times 3 sessions per hour. A computed rate quoted on a slide with no arithmetic beside it is the one way this record could put a number into somebody's argument that no document carries. That is why it is in the short form too, where the line is otherwise kept as brief as it can be. ## The promise The url is the citation. Everything else in a pasted line is there so it reads on its own; what has to survive is that `vitna.ai/cite/{id}` keeps answering. 1. **A citation is minted only for a reading the record holds.** No url is handed out that does not resolve on the day it is copied. 2. **It keeps resolving for as long as vitna.ai serves anything.** That is the whole of the commitment on availability, and it is deliberately not dressed up as more. 3. **An id is never reused and never repointed.** `witness.observation` is append-only: the application roles hold no `UPDATE` or `DELETE` on it, ids come from a sequence that is never rewound, and nothing is deleted. If a citation ever stops answering, it will not have come to answer about a different figure, which is the failure that would actually damage somebody quoting this record. 4. **Four things about the answer cannot change**: what was priced, on which meter, at what figure, and the document and moment it was read from. The page around them can be redrawn, the wording can improve, the forms can gain a fourth. Those four are the citation. 5. **A misreading is corrected by a later reading, never by editing this one.** If Vitna read a page wrong, the citation still resolves to what was read, because that is what it says it is; the correction is a later observation, and the sku's own page carries both with the documents behind each. 6. **If the path shape ever changes, `/cite/{id}` keeps answering.** A url in somebody's contract is not a thing to tidy up. ## What a citation does not claim **That the provider published the price.** Nothing here can. No provider signs its own pricing page, and the signature covers Vitna's record of the fetch rather than the page it fetched. What ties a figure to a provider is the archived document named by its sha256, and [verifying a figure](verify-a-figure.md) is fetching that url yourself and comparing hashes. **That the price is still that.** A citation is of one reading on one date. The sku's own page lists every period it has held since, and draws each price that has moved on a true-time axis. **That the provider's page still says it.** A page that has changed since is the ordinary case, and the archive is what makes the disagreement arguable rather than a matter of opinion. **That the day was signed when you copied it.** Roots are cut once a day, after the day closes, so a figure read this morning is provable tomorrow. The citation says which of the two it is rather than implying a root that does not exist yet. ## Citing something that has to outlive us If a figure is going into a document that has to stand on its own, take the proof with it: ```bash curl "https://vitna.ai/api/v1/observations/{id}/proof" ``` That answer carries the leaf's own fields, its hash, the path of siblings up the tree, the root the walk should reach, and the signed statement over it. It is self-contained: a reader holding that file and the published key can check the arithmetic with nothing from this site. The verifier is given away. `platform/witness/attest.mjs` ships inside the Apache-2.0 `vitna-runtime` package, so an independent implementation of the check is a dependency rather than a rewrite. The site that serves these urls is not open source, so what survives Vitna is the checkable artifact and not a mirror somebody else can stand up. Two things are worth archiving beside a citation you intend to rely on: that proof response, and the day's row from [the attestations](api/v1/verification/attestations.md), ideally fetched on a different day from a different network. The mechanism is that the statement was in the open at the time, so an edit made later has to contradict something already published, and a copy somebody else holds is what makes that bite. ## What the record does not archive for you The archived copy of the provider's document is behind a token, because it is a third party's page and republishing it is a question this record does not need to answer. The hash, the url and the fetch time are public, and those are what a check needs. If your argument depends on the page itself, keep your own copy of it: fetch the url the citation names and archive what comes back. --- # One key for the hosted APIs An org holds one key for Vitna's hosted APIs. Search, the Web APIs and the Unblocker each accept it as `Authorization: Bearer vitna_key_...`, and each checks it with vitna.ai, where the key lives as a hash beside the org's other keys. During the beta an operator mints it. ## What it is Every org key in the record holds exactly one scope, and a platform key holds the third: | Scope | Prefix | What it may do | |---|---|---| | `push` | `vitna_push_` | Post a runtime's closed days to `POST /api/v1/usage` | | `router` | `vitna_push_` | Call the hosted router, which spends money at a provider | | `platform` | `vitna_key_` | Call Search, the Web APIs and the Unblocker | A platform key cannot push a ledger or call the router, and a push or router key is not a platform key, so a key leaked out of one pipeline can do only the one thing it was minted for. With it, the org is the service's client, written `org.`, and is limited and metered exactly as any other client of that service is. The keys each service issued before this, `vitna_search_`, `vitna_web_` and `vitna_unblock_`, keep working, and a service tries its own keys first. ## Getting one An operator mints it for the org, at a prompt, and it is shown once: ```bash npm run org:key:create -- --org acme --label "acme, hosted APIs" --scope platform ``` The record keeps its sha256, never the key. The console lists every key an org holds with what each may do; it does not mint or revoke them. Revoking is the same command as for any org key, and a service stops accepting the key within a minute: ```bash npm run org:key:revoke -- 12 --reason "rotated" ``` ## Using it The same key, at each service's own host: ```bash curl https://search.vitna.ai/v1/search \ -H "Authorization: Bearer $VITNA_KEY" \ -H "Content-Type: application/json" \ -d '{"query":"solar system","mode":"index","maxResults":5}' curl https://web.vitna.ai/v1/read \ -H "Authorization: Bearer $VITNA_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/","format":"markdown"}' ``` The playgrounds on [vitna.ai/search](https://vitna.ai/search) and [vitna.ai/apis](https://vitna.ai/apis) take it too, in the field that asks for that product's key. ## How a service checks it A service asks vitna.ai, authenticated by a service token of its own, and sends the key's sha256 rather than the key: ```text POST /api/v1/keys/verify ``` The body is `{"key": ""}` and nothing else. The answer is `{"active": true, "key": {"id": "12", "org": "acme"}}` for a live platform key, and `{"active": false}` for anything else: an unknown key, a revoked one, or a key of another scope, which are not told apart. A service token that is not configured is refused before any lookup, and an instance that cannot read the record answers `503` rather than calling the key inactive. A service keeps a live answer for a minute and a refusal for thirty seconds, so a revocation lands within a minute. When vitna.ai cannot answer, a key the service saw live within the last six hours stays live, and a key it has never seen is answered `503` with `retry-after`, never `401`, because "I could not look" is not "you may not". Lookups of keys a service has not seen are capped per minute, so a flood of made-up keys at a service is not a flood at vitna.ai. ## Setting a service up Mint the service's token, which prints both halves: ```bash npm run service:token:create -- --service search ``` The token and the endpoint go in the service's own environment, as `VITNA_PLATFORM_SERVICE_TOKEN` and `VITNA_PLATFORM_VERIFY_URL`; the line it prints goes in the site's `VITNA_KEY_VERIFIERS`, a JSON object of service name to token hash. A service with neither variable set does not accept platform keys, and one with only one of them refuses to start. ## What the console sees of a request Every request a service admits gets a receipt: its request id, the client, when it started, how long it took, its status or outcome, and the providers or routes it tried. A receipt never carries the query, the page or the answer. Since 2026-10-11 a service that accepts platform keys also sends the receipts for an org's requests to vitna.ai, where the console's Activity lists them, newest first, each one open to inspect: ```text POST /api/v1/receipts ``` It is authenticated by the same service token as key verification, and it names no service in the body: the token says which service is posting. The body is `{"receipts": [ ... ]}`, at most 500 at a time. Each receipt is read through an allowlist, so a field the console has no column for never reaches the record, and a receipt whose client is not `org.` is dropped. A request reported twice is one row. Each receipt is filed under the console product it belongs to as it lands: Search, the Unblocker, the Browser (the Web APIs' `browser` sessions) or Web extraction (the Web APIs' read, scrape, extract, map, crawl, screenshot, pdf, documents, product, company, research, catalog, careers and search operations). A receipt that belongs to none of them is dropped and the answer says why: reading or cancelling a job, whose own receipt arrives when the job ends, and the recipe, monitor, dataset, published, sources, firehose and session routes, which the console has no page for yet. A page of a SERP collection job is recorded under its job's id and page number, since it has no request id of its own. A service whose name is not `search`, `unblocker` or `web` is refused whole with 403, so a verifier added under a new name shows up in the site's journal on its first batch rather than as rows no page shows. A Browser session that ended the way its client asked, by `browser.close()` or by closing its socket, reads as ok whatever close reason the client sent; one the service ended (shutdown, a protocol error, the session, idle or transfer limit) reads as failed. A service sends them in the background, in batches every few seconds, and a request never waits on it. If vitna.ai cannot take a batch, the batch waits and is retried, doubling its wait up to five minutes; past 5,000 queued receipts the oldest are dropped first. A 401 or 403, which means the service's token or its name is not configured on vitna.ai, is retried the same way, because fixing the setting should not lose what was queued; only a batch refused for what it is (400 or 413) is not sent again. The address is `/api/v1/receipts` on the host `VITNA_PLATFORM_VERIFY_URL` names, so releasing a service with this change is what turns it on. `VITNA_PLATFORM_RECEIPTS_URL` points it elsewhere and `VITNA_PLATFORM_RECEIPTS=off` keeps it off. vitna.ai's own Search relay, which vitna.ai/search and the console's Search workspace both use, records the searches it carries for an org's key as well, so a search run from the console is in Activity the moment it finishes. The console's Unblocker workspace sends through vitna.ai's Unblocker relay, which records nothing itself, so its runs are in Activity once the service's next batch lands, a few seconds later. The query and the results of a run made in the console are kept only in the browser that ran it, which is how the inspector can show them beside the record without vitna.ai ever holding them. Of an Unblocker run that copy keeps the page's text up to 20,000 characters, its first hundred links and every attempt, and never the HTML. They are kept for the account that ran them, another account signed in to the same browser never sees them, and signing out from the console forgets them, together with the org key pasted into the workspace. ## What it does not do yet - **Self-service keys.** The console names keys and their scopes and does not mint them. Letting the web process mint and revoke is a decision of its own, since today no role it holds can write a key. - **One statement across products.** Each service meters an org's use on its own; nothing yet totals them. Since 2026-10-16 the console's Plan page counts the Unblocker's delivered pages per org and UTC month from these receipts, the unit its proposed price names, and prints a count, never a price. The receipts are sent best effort (a service past 5,000 queued drops the oldest), so that count is not yet a ledger a statement can be billed from. - **A trace across products.** Each service mints its own request id, so a search followed by a page read and a model call are three unrelated rows in Activity. Joining them needs a trace id every service carries forward. - **Retention.** Nothing prunes the receipts the console reads yet. --- # Quickstart Every read endpoint on `vitna.ai` is public and needs no key. There is no signup, no token, and no rate limit to negotiate. If you can reach the host you can read the record. > [!TIP] > Responses carry an `ETag`. Send it back as `If-None-Match` and an unchanged > record answers `304` with no body, which is the cheapest way to poll. ## The first request ```bash curl https://vitna.ai/api/v1/markets ``` That answers the component classes Vitna prices, how many skus and providers each one carries, and how fresh the freshest and stalest figures in it are. ## The tape for one class ```bash curl https://vitna.ai/api/v1/markets/model/tape ``` One row per priced cell: a sku, a dimension, an amount in USD, the unit that amount is quoted in, when the price started, when it was last confirmed, and the sha256 of the archived document it was read from. Valid classes are `model`, `tooling`, `retrieval`, `extraction`, `translation`, `speech` and `compute`. Every declared class is wired as of 2026-09-10. A class that is registered and not yet read answers an empty tape rather than a 404, and the [markets](api/v1/prices/markets.md) endpoint keeps it in the answer at zero. ## One sku over time ```bash curl "https://vitna.ai/api/v1/skus/gpt-5-pro/history?dimension=input_per_mtok" ``` Price periods, oldest first. A period is a stretch during which the price did not move. ## The spread on one product ```bash curl https://vitna.ai/api/v1/spreads/compute/h100-sxm ``` The same product sold by several providers, cheapest first, with the percentage between the low and the high. ## Subscribe to the log ```bash curl https://vitna.ai/api/v1/feed.json ``` JSON Feed 1.1. Price moves, launches, policy changes and deprecations, plus the triggers that fired and the decisions the engine proposed. ## Check a figure yourself Take an observation id from the record and ask for its proof: ```bash curl https://vitna.ai/api/v1/observations/1/proof ``` You get the leaf's own fields, its hash, the sibling hashes up the Merkle tree, and the signed root the walk should arrive at. Rebuild the hash, fold the path, verify the Ed25519 signature. Nothing in the answer has to be trusted for the check to mean something: if any of it were wrong, the arithmetic would stop agreeing. The full walkthrough, both checks end to end, is in [Verify a figure yourself](verify-a-figure.md). ## Three-minute gateway quickstart (LiteLLM and OpenRouter) Teams already running LiteLLM or OpenRouter in production can adopt Vitna's attested market prices, local unit profiles, and signed routing decisions in three steps without swapping out their gateway first. ### Step 1: Load attested Witness prices in LiteLLM (no key, no signup) ```bash curl -s https://vitna.ai/api/v1/gateways/litellm/model-prices.json | head -c 400 ``` ```bash LITELLM_MODEL_COST_MAP_URL=https://vitna.ai/api/v1/gateways/litellm/model-prices.json \ litellm --config litellm.yaml ``` Every entry translates Witness `input_per_mtok`, `output_per_mtok`, and `embed_per_mtok` figures into per-token USD alongside a `vitna` provenance block (`sku`, `sourceUrl`, `confirmedAt`, `evidenceSha256`, `citationUrl`, and full context `bands`). Responses carry `ETag`, so polling with `If-None-Match` returns `304` when nothing changed. See [LiteLLM cost map](api/v1/prices/litellm-cost-map.md). ### Step 2: Measure your own token unit profile locally Run sample or mirrored traffic through [Relay](runtime/relay.md) (no policy, no vitna.ai account, nothing leaves your machine), then compute your observed token profile from its append-only JSONL ledger: ```bash OPENAI_API_KEY=sk-... npx -p vitna-runtime vitna-relay vitna-runtime-profile --file ~/.vitna/relay/ledger.jsonl vitna-runtime-profile --file ~/.vitna/relay/ledger.jsonl --json ``` `vitna-runtime-profile` prints `--profile inputTokensPerUnit=... --profile outputTokensPerUnit=...` for each lane, separates refusals and unrouted passthrough rows rather than estimating missing counts, and stays completely offline. Operators recording into an Aperture register can also extract p90 latency and sample counts directly via `npm run aperture:evidence:from-run` ([Measuring your unit profile](runtime/profile.md), [Candidates and evidence](cli/aperture/evidence.md)). ### Step 3: Feed CI evals and render signed policies back to your gateway Send existing Promptfoo, Braintrust, or UK AISI Inspect AI results directly to the append-only evidence endpoint ([CI evaluation evidence](api/v1/engine/org-evidence.md)) or enter them via `npm run aperture:evidence:add` ([Candidates and evidence](cli/aperture/evidence.md)): ```bash curl -X POST https://vitna.ai/api/v1/org/evidence \ -H "authorization: Bearer vitna_key_..." \ -H "content-type: application/json" \ -d '{ "adapter": "promptfoo", "workload": "deep-research", "sku": "openai/gpt-5.6-luna", "label": "pr-412-gpt-5.6-luna", "sourceRef": "https://github.com/convexity/app/actions/runs/91823", "report": { "results": { "stats": { "successes": 193, "failures": 7, "errors": 0 } } } }' ``` Once a workload decision is accepted and signed into `policy.json`, render that same policy into native configuration for your existing proxy (`litellm`, `openrouter`, `cloudflare`, or `portkey`) without changing your serving data plane ([Gateway adapters](routing/adapters.md)): ```bash VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target litellm --policy policy.json --out litellm.yaml VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target openrouter --policy policy.json --out presets.json ``` ## Conventions worth knowing before you parse anything - Every response is a JSON object with a `meta` key. See [api/v1/overview.md](api/v1/overview.md). - `meta.source` tells you whether the answer came from the live record (`database`) or from the snapshot committed in the repository (`snapshot`). The public site answers from the record. - Money is USD, quoted in the unit the dimension declares, stored to six decimal places. - Timestamps are RFC 3339. Calendar dates are `YYYY-MM-DD` and mean the day. - Responses carry an `ETag`. Send it back as `If-None-Match` and you will get a `304` when the record has not changed. ## What this API will never do It does not route traffic, hold your keys, proxy inference, or take an action on your behalf. Vitna reads published prices and records decisions about them. Changing what your stack runs is a person's job in another system. --- # Vitna documentation The written surface of Vitna: what the record holds, what the words mean, and how to read it over HTTP without a key. This corpus is Markdown on purpose. It is the source a `/docs` route would render later, it greps from a terminal, and it generates `llms.txt` and `llms-full.txt` beside it so an agent can read the whole set in one fetch. ## Layout | Path | What is in it | |---|---| | `quickstart.md` | Six requests that show the shape of the API | | `concepts/` | The vocabulary: what a sku, a dimension, an observation and a period are | | `api/v1/` | One page per endpoint, with the shape it answers and the ways it fails | | `llms.txt` | Generated index, one line per page | | `llms-full.txt` | Generated single-file bundle of every page | | `build-llms.mjs` | Regenerates both from the Markdown | ## Regenerating the index ```bash node docs/build-llms.mjs ``` It reads every `.md` under `docs/`, takes the first heading as the title and the first paragraph as the summary, and rewrites `llms.txt` and `llms-full.txt`. Run it after adding or retitling a page. ## What this corpus does not cover yet The operator CLI (the `witness:*`, `aperture:*`, `org:*` and `auth:*` command families) and operations (deploy, backup, the units on the box) are documented in `README.md`, `PLATFORM.md` and `HOSTING.md` at the repository root and have no pages here yet. ## Rules for writing in here Three, all inherited from the repository: - No em-dashes, in prose or in code comments. Commas, colons, or a full stop. - Nothing described as built that the code does not do. Every figure, field name, status code and default written in the present tense was read out of `platform/` or `app/`, and a page that drifts from the code is a bug in the page. - A page may describe what is not built yet, and it says so where a reader meets it rather than at the end. The section below is how. ## Writing about what is not built This rule used to read "nothing asserted that the code does not do", flatly, and it was two things at once: a ban on describing unbuilt work, and a ban on describing built work wrongly. Only the second was load-bearing. The first cost the corpus every page worth writing while a thing is being built, which is exactly when a design is worth reading and worth arguing with. What it was protecting still needs protecting, and more than it looks. **This corpus is published**: every page renders at `vitna.ai/docs`, appears in the sitemap, and is concatenated into `llms.txt` and `llms-full.txt` for readers who fetch once rather than crawl. So a page here is read by people who cannot see this repository, and the only thing between a plan and a promise is how plainly the page says which it is. `RUNTIME.md` puts "marketing any mode before its phase ships" out of scope, and a docs page is a public surface for that purpose, so a page describing an unshipped mode is a decision to publish it rather than a place to draft it. One line carries it, and it is the first paragraph under the title: ```markdown # Vitna router **Not built.** `GET /api/v1/router/status` does not exist yet. This page is the design; `RUNTIME.md` phase R5 is where it sits. ``` That one line does three jobs, which is why it is a line of the page rather than a comment above it. **The reader meets it first**, as the opening sentence of the page's lede, before anything they could mistake for a description of software they are running. Not in bold, though, and that is worth knowing before writing one. `app/docs/[...slug]/page.tsx` renders the lede from `page.summary`, the flattened string `build-llms.mjs` computes, so the first paragraph's bold and inline code are stripped on the rendered page. They survive in the Markdown, which is what the test reads, and in `llms.txt`. So write the marker to read correctly as plain prose, because plain prose is how a reader will meet it. The `**` and the backticks are there for the machines and for anyone reading the source. **The index says it too.** `build-llms.mjs` lifts the first paragraph under the title as the page's summary, so `llms.txt` carries "Not built" as well, and an agent that fetches the index once rather than crawling is not misled by a title that sounds shipped. **And it is checked.** `tests/docs-drift.test.mjs` reads the inline code spans out of that same paragraph and asserts each one does **not** exist. The day somebody adds the script or mounts the route, the test fails and names the page to correct. So the falsifier is not a second artifact that can disagree with the sentence: it is the sentence, read by a machine. That is the same shape as `UNDOCUMENTED_ROUTES` in that file, checked for existence so an exemption cannot outlive the route it exempts, and it completes the corpus's directions. Everything named must exist, everything that exists must be named, and everything named as absent must be absent. **A command is named the same way**, as an inline span reading `npm run` and the script, and it has to be inline rather than in a fence. The guard above this one, "every command the docs name is a script", reads fenced blocks and refuses any `package.json` does not define, so a fenced example of an unbuilt command fails it. Which is correct of that guard. The marker is a sentence, and a sentence is where it belongs. **Not an HTML comment**, which was the first design and is wrong twice. `platform/docs/parse.mjs` has no case for one and renders it as a visible paragraph, so it would print raw on the page. And a marker only the repository can see is a marker for the wrong audience: the person who most needs to know this is a plan is the one reading it at `vitna.ai/docs`. **Why a mechanism and not a convention.** The drift test exists because the routing section went on claiming it described unmerged work for a week after it merged, and a person reading is what caught it. A marker nobody checks goes stale in one direction only, and it is the direction that misleads: the page keeps saying "not built" after it is built, which reads as modesty and is wrong. Nothing has ever gone stale the other way. The marker is opt-in and existing pages carry none, so this changes nothing about the corpus as it stands. **What the marker does not catch, learned the same day it shipped.** It watches the thing the page is about. It does not watch the things the page says are still owed. `docs/routing/router.md` went live at 03:46 on 2026-08-31 asserting that `VITNA_PUBLIC_KEY` pinned a single key and that making it accept a set was work still to do; #184 had made it a set at 03:39 and #186 had landed the key registry at 03:43. Seven minutes stale on publication, and the marker was serenely correct throughout, because the router's own script and route still did not exist and those are the only two things it names. So a page's prerequisites go stale faster than its subject, and nothing here watches them. Two habits rather than a mechanism, because the mechanism would be a test asserting the absence of half the codebase. Re-read a not-built page's "what has to happen first" section against `git log origin/main` immediately before merging it, not when you wrote it. And when a page says a thing is owed, prefer naming the file and the behaviour over naming a date or a difficulty, so that the sentence is checkable by somebody reading the file rather than only by somebody who remembers the plan. --- # Verify a figure yourself The whole claim of this record is that you do not have to trust it. This page is that claim carried out, end to end, with nothing taken on faith. There are two independent checks, and they answer different questions. Neither requires a key and neither asks Vitna to vouch for anything. | Check | Answers | |---|---| | [The document](#the-document) | Did the provider publish this price? | | [The signed record](#the-signed-record) | Has Vitna changed this since it was recorded? | They are separate on purpose. A signature says nothing about a provider, and a provider's page says nothing about whether Vitna edited the record afterwards. Together they close both halves. ## Start with a figure Take any row off the tape: ```bash curl "https://vitna.ai/api/v1/markets/model/tape" | head -40 ``` Every row carries the three things the checks need: | Field | Used by | |---|---| | `sourceUrl` | The document check | | `evidenceSha256` | Both | | `observationId` | The signed-record check | All three describe one read, the one that last confirmed this price, so the two checks are checks of the same thing rather than of two neighbouring facts. ## The document The strongest check, and the simplest: fetch the provider's page yourself and see whether it hashes to what the record says it read. First ask the record what it fetched and when: ```bash curl "https://vitna.ai/api/v1/evidence/" ``` That answers publicly, with no token: the URL fetched, the fetch time, the HTTP status, the response headers, the user agent, the adapter, the parser version, the byte count, and every later read that found the same bytes. Now fetch the same URL and hash it: ```bash curl -s "" | sha256sum ``` If the provider has not changed the page since that fetch, your hash equals `evidenceSha256` and the figure is traceable to bytes the provider served. > [!NOTE] > A different hash is the ordinary case on a page that has since changed, and it > is not a failure. It means the document moved on. What the archive records is > what Vitna read at a stated moment, which is why the fetch time is published > beside the hash. The archived copy itself is behind a token, because it is a third party's page and republishing it is a question this record does not need to answer. The hash, the URL and the time are public, and those are what a check needs. ## The signed record Every day of observations is hashed into a Merkle tree and the root is signed with Ed25519. Checking one figure means walking from its leaf to that root, then verifying the signature over it. Take the `observationId` off the row you started from and ask for its proof: ```bash curl "https://vitna.ai/api/v1/observations//proof" ``` That is the whole of the step. The rest of this section is what the answer contains and how to check it. If you would rather start from a day than from a figure, the published roots carry the id range each one covers: ```bash curl "https://vitna.ai/api/v1/attestations" ``` Each row carries `coversOn`, the `root`, the `statement` the signature covers, the `signature`, the `publicKey`, the `leafCount`, and the range of observation ids the day covers as `firstObservationId` and `lastObservationId`. Any id in that range belongs to that day's tree. Either way, the proof response carries the leaf's own fields as `leaf`, the exact string its hash is computed over as `canonicalLeaf`, its hash, its index, the sibling hashes up the tree as `proof`, the root the walk should reach, and the signed attestation. ### The four steps 1. `leafHash = sha256(0x00 || canonicalLeaf)`, over the UTF-8 bytes of `canonicalLeaf` exactly as served. It must equal the `leafHash` given. 2. Check that `canonicalLeaf` says what `leaf` says: `leaf` written as JSON with the keys in the order shown, no whitespace, amounts as the strings given, and `bandFromTokens` and `bandToTokens` left out where they are `0` and `null`, is `canonicalLeaf` byte for byte. A figure priced once is hashed without its band; a figure priced in a context band is hashed with both. 3. Fold the path. Each step carries a `side` and a `hash`. Compute `sha256(0x01 || hash || running)` when `side` is `left`, and `sha256(0x01 || running || hash)` when it is `right`. An odd node is promoted unchanged, so it has no step. 4. The result must equal `attestation.root`, and `attestation.statement` must verify against `attestation.publicKey` with Ed25519. The algorithm is named in the response as `sha256-merkle-rfc6962`, and the statement carries the prefix `vitna-attestation-v1` so a routing config signature can never be read as an attestation. ### Why the two prefix bytes A leaf is hashed under `0x00` and an interior node under `0x01`. Without the distinction the two are drawn from the same space, and a crafted leaf could stand in for a whole subtree. This is the standard defence and it is why the algorithm string names RFC 6962. ### Check the key, separately A valid signature proves whoever holds that key signed the statement. It does not prove the key is Vitna's, and no amount of arithmetic inside one response can establish that. Compare `publicKey` against the one published on `/witness` and on `/api/v1/attestations`, ideally from a copy you obtained on a different day. The mechanism is that the statement was in the open at the time, so an edit made later has to contradict something already published. ### Two roots, published together The proof also carries `computedRoot`, which is what the endpoint calculates from the record as it stands right now, beside `attestation.root`, which is what was signed. They should be equal. Publishing both is what lets you see a disagreement rather than take the record's word that there is none. ## The one case where the path stops `observationId` is `null` when an instance is serving a committed snapshot written before the ids were carried. There is nothing to walk from, and the row says so rather than offering an id that would prove something else. `vitna.ai` serves the record, so this affects local checkouts and any mirror still running an older snapshot. The document check has no such gap and works from any tape row. ## Quoting a figure somewhere else Every figure on every board carries a `cite` mark, and it leads to a permanent dated citation of that figure in the three shapes people paste: a footnote, a short reference for a slide, and a schedule line for a contract. [Cite a figure](citing-a-figure.md) is what those are, and how long the urls keep resolving. The two checks above are what a citation is worth. A quoted figure that nobody can take back to a document is a number in a slide; this page is why one here is not. ## What neither check proves **That the price is a good deal, or one you could get.** The record holds published list prices. Negotiated rates, committed-spend discounts and private agreements are invisible to it. **That Vitna read the page correctly.** The hash proves which bytes were read. The `locator` and `parserVersion` on the observation say where in the document the parser looked and which version looked there, which is what makes a misread arguable against an archived document rather than a matter of opinion. ## Doing it in bulk ```bash npm run witness:attest -- --verify ``` Rechecks every root ever signed. Because the inputs are append-only and the rule has no free parameters, an independent implementation reading the same public endpoints should reach the same answer, and disagreeing with this command is the interesting result. --- # The model Vitna is a record of what AI infrastructure costs, and an engine that decides when a change to a production stack is worth making. Two systems, one repository, one database with a schema each. ## Witness, the record Witness reads pricing pages that providers publish, archives what it read, and accumulates a price history from it. The chain runs one way and never doubles back: ``` source -> fetch -> observation -> price period -> market event ``` A [source](source.md) is a fetchable document. A fetch is one read of it, with the bytes archived under their sha256. An [observation](observation.md) is one figure parsed out of that document, pinned to the fetch it came from. A [price period](price-period.md) is a stretch during which that figure did not move. A [market event](market-event.md) is a published statement that something happened. Everything in that chain is append-only. A price does not get corrected by editing it; a correction is a later row that names the one it replaces. ## Aperture, the engine Aperture holds workloads: a thing you run, the sku it currently runs on, the candidates in play, and the evidence measured about them. It arms triggers, watches the record for them, and when one fires it produces a brief arguing for or against a change. It proposes. It does not act. See [decision.md](decision.md). ## The honesty contract The rules the whole system is built to keep, because a price record that is occasionally wrong is worth less than no price record at all: **Nothing is asserted without a document behind it.** Every figure on the tape names the sha256 of the archived page it was read from, and the URL a human should read. **A figure read once is not yet a price.** Change detection publishes a move on the second consistent read, so one bad parse of one malformed page does not enter the record as a price change. **Absence is stated, never filled.** A cell no source publishes is missing from the tape rather than present and zero. A class with no wired market answers an empty tape. An endpoint that cannot reach the record answers `503` rather than an empty list, because an empty list reads as "nothing is there" when the truth is "I could not look." **The record signs itself.** Each day's observations are hashed into a Merkle root and signed. See [attestation.md](attestation.md). **Derived figures say so.** A number Vitna computed rather than read carries `derived: true` and a `derivation` string saying how. ## Two ways of serving The same code answers from two places, and `meta.source` on every response says which: | `meta.source` | Meaning | |---|---| | `database` | The live record. This is what `vitna.ai` serves. | | `snapshot` | The JSON committed under `app/data/`, a picture of one moment. | A checkout with no `DATABASE_URL` runs entirely from the snapshot, which is how the tests run and how `npm run dev` works. Endpoints that can only be answered from the record (history, attestations, proofs, proposals, watches) say so rather than improvising an answer from the snapshot. ## What Vitna is not It is not a broker, a proxy, or a router. It holds no keys to anything and sends no traffic anywhere. It reads published list prices, which is a narrower claim than "what you will pay": no negotiated rate, committed-spend discount or private agreement is visible to it, and it says so rather than implying its numbers are yours. --- # Attestation Each day's observations hashed into a Merkle root and signed with Ed25519, so that an edit made afterwards has to contradict something already published. ## The mechanism Every morning, the previous day's observations are hashed into a tree: - A leaf is `sha256(0x00 || canonical JSON of the observation)`, with the field order fixed in `platform/witness/attest.mjs`. - An interior node is `sha256(0x01 || left || right)`. An odd node is promoted unchanged. - The remaining root goes into a statement, and the statement is signed. The two distinct prefix bytes matter: without them, a leaf hash and an interior hash are drawn from the same space, and a crafted "observation" could stand in for a subtree. ## Reading the roots ```bash curl https://vitna.ai/api/v1/attestations ``` Newest first, each carrying the statement, the signature, and the public key that made it. The response also carries the algorithm, the statement prefix, and the three steps a check involves, served beside the data rather than linked from it. ## Checking one figure A root by itself is a claim. The path from a single figure to that root is what makes it checkable: ```bash curl https://vitna.ai/api/v1/observations//proof ``` That answers the leaf's own fields, its hash, its index, the sibling hashes up the tree, and the signed root the walk should reach. Rebuild the leaf, fold the path, compare, then verify the signature. The response also carries `computedRoot`, which is what the endpoint calculates from the record as it stands right now. It should equal the signed root. Publishing both is what lets a reader see a disagreement rather than take Vitna's word that there is none. ## What a signature proves **It proves** the record for that day has not changed since it was signed. > [!WARNING] > A signature does not prove the provider published the price. No provider signs > its own pricing page. What ties a figure to a provider is the archived > document, and that records what Vitna fetched. See [evidence](evidence.md). Both sentences ship inside the API responses, because this is the claim most easily overstated by someone quoting the record. ## Today is not yet signed Roots are signed the following morning. A figure read today becomes provable tomorrow, and until then the proof endpoint answers with `attestation: null` and an `unattested` string explaining it, rather than failing. A day older than the newest signed day that still has no root is a different case: a gap, not a wait. The proof says that instead, names the newest signed day, and promises nothing. See [a day with no root](../api/v1/verification/proof.md#a-day-with-no-root). ## Doing it end to end [Verify a figure yourself](../verify-a-figure.md) walks the whole path with the commands, alongside the separate check against the provider document. ## Verifying the whole chain ```bash npm run witness:attest -- --verify ``` Rechecks every root ever signed. `--new-key` prints a keypair and touches no database. --- # Component class The top-level division of the market. A class is a kind of thing you buy, with one unit it is ultimately bought in. Classes are registry rows in `witness.component_class`, seeded from `platform/witness/seed-data.mjs`. | id | Label | Unit | Wired | |---|---|---|---| | `model` | Language models | One successful task | yes | | `tooling` | Agent tools | One tool call | yes | | `retrieval` | Embedding and rerank | One answered query | yes | | `compute` | GPU compute | One GPU-hour delivered | yes | | `image` | Image generation | One accepted image | yes | | `extraction` | Document extraction | One correctly extracted document | yes | | `translation` | Machine translation | One accepted translation | yes | | `speech` | Speech recognition and synthesis | One usable transcript; one accepted utterance | yes | ## A class can hold two units Most classes name one unit. `speech` names two, because it holds two things bought together and sold by overlapping sellers: a transcript, billed by the minute of audio, and an utterance, billed by the million characters of text. What the two halves do not share is a quality story, and the class says so rather than leaving a reader to assume. Transcription quality is word error rate over audio the customer already has, which a laptop computes. Synthesis quality is a listening panel, which is people in a room. Both are measurable, which is what admits the class at all: a class whose quality cannot be measured has nothing to weigh a price against. Only one of them is cheap. ## The unit is not the meter The `unit` column is deliberately not what providers bill in. A language model is billed per million tokens; what you actually buy is a completed task. A GPU is billed by the hour; what you buy is an hour that delivered something. The gap between those two is the whole point of the record. The unit names what you are buying. The [dimension](dimension.md) names what you are billed by. ## A class a language model is eating is still a class `translation` is the clearest case of the gap this record exists to price. Dedicated translators sell by the million characters. General models sell by the million tokens and do the same job. A buyer choosing between them is asking the stay, test or switch question in its plainest form, and almost nobody publishes the comparison. The record holds both and refuses to convert between them. See [dimension](dimension.md#a-character-is-not-a-token) for the arithmetic that does not exist, and note that the absorption is visible inside the class as well as across it: two of its five products are large language models billing input and output separately, in the translator unit. Image generation is the class where that gap is widest, and it is the class that made the gap visible in the data rather than only in this paragraph. What you buy is an image somebody accepts. What you are billed for is an image at a particular output size and a particular quality setting, and the same model can cost six tenths of a cent or twenty-one cents depending on nothing else. So an image [sku](sku.md) carries the size and the quality its figure is for, and a seller who charges one price whatever comes back says that too, with a null and a sentence rather than by leaving the question unanswered. Quality is what makes the class admissible at all, and it follows the same rule every other class follows: it is measured on the customer's own work or not at all. For an image that is the acceptance rate on the customer's own briefs, judged by the people who already review them. Nothing in this record scores a picture. ## Wired `wired` says whether Vitna has live sources for a class yet. An unwired class is registered, appears in `/api/v1/markets`, and answers its tape endpoint with an empty array and a `200`. That is deliberate. A `404` would say the class does not exist. An empty tape says the class exists and Vitna is not yet reading it, which is true. ## Where classes appear - `/api/v1/markets` lists every class with its counts and freshness. - `/api/v1/markets/{class}/tape` takes a class id in the path. An id not in the table above answers `404 unknown_class`. - `/api/v1/spreads/{class}/{product}` likewise. - Every [sku](sku.md) and every [dimension](dimension.md) belongs to exactly one class. --- # Workloads, triggers and decisions Aperture, the engine half. Witness records what things cost; Aperture decides whether a change is worth making, and records that nobody has acted on it yet. ## Workload A thing you run: a name, the [sku](sku.md) it currently runs on, the unit it is measured in, a priority, and a usage profile. The registry refuses a workload whose incumbent is unregistered, whose incumbent belongs to another market, whose profile names a key its class does not spend, or whose incumbent has no profile to spend it against. Those refusals are the point: a workload the engine cannot reason about should fail at entry rather than produce a confident brief later. A workload is a registry row and not a record, so it can be corrected with `aperture:workload:edit`. Evidence and decisions cannot: those stay append-only. ## Candidate A sku in play against the incumbent for a given workload. ## Evidence A measurement about a candidate: a kind, a label, a date, a sample size `n`, one or more metrics, and a source reference that is a URL or a commit. Recording a metric the engine never reads is refused. A measurement nothing consumes is a number that looks like evidence and functions as decoration. ## Trigger An armed watch on the record. `price_move` with a threshold percentage is the worked example. A trigger fires when the record moves past its condition, and firing spends it: it will not fire again until a decision arms a new one. That is what stops one volatile cell from producing the same notification every hour until it is muted, and muted watches are how monitoring systems stop being read. ```bash curl "https://vitna.ai/api/v1/watches?state=armed" ``` ## Decision What the engine proposed, and what a person did about it. | Status | Meaning | |---|---| | `proposed` | The engine produced a brief. Nobody has ruled on it. | | `accepted` | A person accepted it, arming the next review. | | `superseded` | A person refused it, with the reason kept on the row. | ```bash curl "https://vitna.ai/api/v1/proposals?status=proposed" ``` `meta.awaitingReview` counts the proposals waiting on a person, which is the number worth watching in this record. The third row is the one worth reading twice, because the word invites the opposite reading. `superseded` is written by one act only, a person refusing a proposal, from the prompt or from the console, and both keep the reason on the row. A decision that is merely overtaken by a later one **keeps its own status**: an accepted decision stays accepted forever, because what it records is that somebody accepted it at the time, and that stayed true. Which decision a workload is currently running is the chain of supersessions, never this column. This table said "it was overtaken without being accepted" until 2026-08-26, which described neither. ## One decision has an address Every decision an org publishes is a page: ``` https://vitna.ai/decisions/{id} ``` It carries what `/api/v1/proposals` carries plus the brief the run wrote, the chain on either side of it, and a section naming the columns the row does not hold and why each one is allowed to be empty. The gate is the same one this whole register is published under. A decision has a page exactly when it has JSON: `org.feed_public` decides, and an org whose feed is private has neither. A decision belonging to such an org answers 404, identically to an id nobody ever issued, because which of the two it was is information about somebody else's register. ## Inputs are published Every decision carries the exact `witness.price_period` and `aperture.evidence` ids the run read. All of them are append-only, so a brief can be regenerated and checked at any distance. Publishing the verdict without them would be publishing an opinion. ## Exporting a signed decision pack Step 8 of model change control. Any accepted or recorded decision exports as a signed decision pack: ```bash npm run aperture:decision:pack -- --out ./audit-pack.json ``` A decision pack bundles: 1. The decision and the provenance absences saying what was not recorded. 2. The exact `witness.price_period` records and observation leaves with their Merkle leaf hashes. 3. The exact `aperture.evidence` measurements and SHA-256 evidence digests. 4. The canonical payload digest and an Ed25519 signature from the authority. It is verifiable offline without network or database access: ```bash npm run aperture:decision:pack -- --verify ./audit-pack.json --key ``` Public decisions can also be downloaded directly from `/decisions/{id}` or via `GET /api/v1/decisions/{id}/pack`. ## The engine proposes and does not act Worth being unambiguous, because "decision engine" invites the opposite assumption. Running the engine writes a proposed decision. Nothing is accepted, and nothing is routed. Accepting is a person's act through the CLI, and acceptance is refused if the record moved underneath the proposal, so a stale brief cannot be accepted into force. This is enforced below the application, not just in the UI, and the exact grants are worth stating because they are narrow rather than absent. **Two application roles hold `UPDATE` in `aperture`, each on named columns of one table and nothing else:** | Role | Table | Columns | Since | |---|---|---|---| | `vitna_collector` | `aperture.trigger` | `fired_at`, `fired_by_event_id` | `0009` | | `vitna_console` | `aperture.decision` | `status`, `note`, `ruled_by`, `ruled_at` | `0021` | The first records that a watch fired. The second records a person's ruling and who made it. Nothing holds `UPDATE` on a workload, a candidate or a measurement at all, and nothing anywhere can touch a decision's outcome, thresholds, inputs, reason or brief: those are fixed at insert, so a correction is a new decision that supersedes the old one and never an edit. So the property is not that a decision cannot be changed. It is that the only thing anything can change about a decision is what a person decided about it, and widening that needs a migration granting a privilege that does not exist. Until 2026-08-26 this page claimed the schema held no such grant at all. That was true when it was written and `0009` ended it one phase later, which is the argument for a table of grants over a sentence about them: the table is wrong in a way somebody notices, and `tests/docs-drift.test.mjs` now fails if any page or module comment goes back to claiming the absolute. A fired trigger is a question put to a person. Vitna records the answer. Changing what production runs happens in another system. --- # Dimension What a price is charged by. A [sku](sku.md) does not have "a price"; it has a price per dimension, and most skus carry several. Dimensions are registry rows in `witness.price_dimension`, each belonging to one [component class](component-class.md). ## The registered dimensions **Language models** (`model`): | id | Label | Unit | |---|---|---| | `input_per_mtok` | Input price | USD per 1M tokens | | `cached_input_per_mtok` | Cached input price | USD per 1M tokens | | `cache_write_5m_per_mtok` | Cache write price, 5 minute retention | USD per 1M tokens | | `cache_write_1h_per_mtok` | Cache write price, 1 hour retention | USD per 1M tokens | | `output_per_mtok` | Output price | USD per 1M tokens | | `batch_input_per_mtok` | Batch input price | USD per 1M tokens | | `batch_output_per_mtok` | Batch output price | USD per 1M tokens | | `priority_input_per_mtok` | Priority input price | USD per 1M tokens | | `priority_output_per_mtok` | Priority output price | USD per 1M tokens | | `flex_input_per_mtok` | Flex input price | USD per 1M tokens | | `flex_output_per_mtok` | Flex output price | USD per 1M tokens | | `output_per_image` | Output price, per image | USD per image | | `batch_output_per_image` | Batch output price, per image | USD per image | A model that emits images is billed by the image rather than by the token, so the class carries two units rather than one. That is why the unit travels on the figure and is not a property of the class. **Agent tools** (`tooling`): | id | Label | Unit | |---|---|---| | `tool_call_per_1k` | Per 1,000 calls | USD per 1,000 calls | | `page_fetch_per_1k` | Per 1,000 pages | USD per 1,000 pages | | `container_hour` | Container hour | USD per container-hour | | `session_hour` | Session hour | USD per session-hour | | `storage_per_gb_day` | Storage | USD per GB-day | A fetched page is its own meter rather than a kind of call. One call can return one page or several, and three of the four sellers on `page_fetch_per_1k` price the page rather than the call. Putting both on one scale would invite a reader to divide one by the other. `session_hour` is a hosted session billed for as long as it runs, and neither seller on it uses that name: one sells a managed agent session and the other sells a browser hour. The id names the meter, as `input_per_mtok` does; sharing one is not a claim that two products are the same, which is what a [sku](sku.md)'s product slug says and these two do not share. **Embedding and rerank** (`retrieval`), both USD per 1M tokens: | id | Label | |---|---| | `embed_per_mtok` | Embedding price | | `rerank_per_mtok` | Rerank price | **Document extraction** (`extraction`), both USD per page: | id | Label | |---|---| | `page_ocr` | Text from a page | | `page_layout` | Structure from a page | A page is not one thing across these providers. `page_ocr` is a page read for its characters; `page_layout` is a page read for its tables, form fields and layout blocks, and what that includes differs by provider. So every extraction sku carries a `page` attribute saying which kind of page its figure is for, and the boards print it under the rate. Two per-page figures compared without it are two different products under one heading. **Speech recognition and synthesis** (`speech`): | id | Label | Unit | |---|---|---| | `audio_minute_batch` | Pre-recorded transcription | USD per audio-minute | | `audio_minute_streaming` | Streaming transcription | USD per audio-minute | | `tts_per_mchar` | Synthesis price | USD per 1M characters | Transcription is two meters because every seller on the board sells it twice, once for audio that has already been recorded and once for audio arriving live, at two different prices. The two share a unit and are still two meters, which is the same distinction `embed_per_mtok` and `rerank_per_mtok` carry: a unit is what a figure is measured in and a meter is what is being sold. That is also the admissibility test for this class, and it is stricter than it sounds. A seller quoting one recognition rate covering both is quoting neither of these meters, so the record declines it rather than filing the figure under whichever of the two looks closest. The audio-minute is the audio's own minute, not the wall clock. A pre-recorded service that returns in ten seconds still bills the minute it read. **GPU compute** (`compute`): | id | Label | Unit | |---|---|---| | `gpu_hour_on_demand` | On-demand list price | USD per GPU-hour | **Machine translation** (`translation`), both USD per 1M characters: | id | Label | |---|---| | `translate_input_per_mchar` | Input price | | `translate_output_per_mchar` | Output price | Only the pay-as-you-go tier is registered. Every seller on that board also prices a volume commitment, and a price for a promise about next year is not a price a reader can take today. The two meters are never summed. Three of the five products bill the characters you send and carry no output figure at all; the other two are large language models billing both ends. Google states on its own page that its ten dollar each way product is cost equivalent with its twenty dollar one way product, which holds at a one to one character ratio and at no other, so the board prints the ends a buyer will actually send rather than an equivalence resting on an assumption nobody made. **Image generation** (`image`): | id | Label | Unit | |---|---|---| | `image_per_image` | Per image | USD per image | | `batch_image_per_image` | Batch price, per image | USD per image | | `image_per_megapixel` | Per megapixel | USD per megapixel | Two units again, and for a sharper reason than the model class has. Some sellers charge one price for an image whatever size it is; others charge by the area of what comes back. At a one-megapixel output the two can print the same number and still be different products, because at two megapixels one of them has doubled. So `image_per_megapixel` is a meter of its own rather than the same meter converted, and no spread is ever drawn across the two. `image_per_image` is not `output_per_image`. The latter is the model class's meter for a language model whose output happens to be a picture, priced beside its token rates on a token page; the former is an image product sold on its own. A figure lives under one of them and never under both. **Video generation** (`video`): | id | Label | Unit | |---|---|---| | `video_per_second` | Per second | USD per second | One meter, because one seller is read so far and it prices one way. What the figure buys is on the sku rather than in the unit, and two things are on it that move a price further than anything else in this class: the resolution, where one model spans $0.05 to $0.60 a second, and whether audio is included, because a rate for video with audio is not a rate for video without it. The same page states a per-second equivalence for token-billed video and calls it approximate. It is not read. An equivalence the seller itself hedges is not a list price for this unit, and the rule that refuses a from-priced GPU refuses this for the same reason. **Music generation** (`music`): | id | Label | Unit | |---|---|---| | `music_per_track` | Per track | USD per track | One meter, for the same reason as video: one seller is read so far and it prices one way. The page bills per request and a request returns one track, so the two are the same thing counted from either end. The length is on the sku rather than in the unit, and nothing divides by it. The same seller charges $0.04 for a thirty second clip and $0.08 for a full song, and the record carries those as two products rather than one rate at two durations. Turning the clip into a price per second would invent a unit the page does not sell: it sells a clip, not thirty seconds. The seller writes "per song" against the clip row too, which is its wording rather than a mistake this record corrects. ## A row carries only the dimensions its source publishes If a provider's page does not quote a batch rate, that sku has no `batch_input_per_mtok`, and the cell is absent from the tape rather than present and zero. Absence is a fact about the market and gets reported as one. ## Sanity bounds are data, not parser code Each dimension carries `sanityMinUsd` and `sanityMaxUsd`. A parsed figure outside them is rejected as a probable misread rather than entering the record. These live on the registry row rather than inside the adapter so that widening one is a data change with a reason attached, not a code change buried in a parser. Two of them have already been widened for exactly that reason: the model input ceiling was $100 until OpenAI listed o1-pro at $150, which made the bound a limit on Vitna's coverage rather than on what a parser could plausibly have got wrong. ## Why every floor is above zero The floor is not a claim that nothing is free. It is there because zero is what a broken read returns. A label that matched a navigation menu, a window that stopped before the figures, a cell index that ran past the end of a row: each produces a nothing that arithmetic turns into a nought, and each would otherwise put a free model on a public board with no error raised anywhere. A page that genuinely publishes $0.00 is a different thing and a real fact about a market. The two are told apart by evidence rather than by declaration: a figure carries the substring the document actually printed, so a zero has to be shown to have been read rather than asserted to have been. ## An add-on is its own meter or it is absent Every speech seller on the board prices diarisation, redaction, keyterm prompting and the rest as separate per-minute rates beside the transcription rate. None of them is registered and none of them is folded into the figure above it, which is what makes the headline rate readable as a headline rate. The rule generalises past speech: an add-on gets a dimension of its own when the record decides to carry it, and until then the record carries the base rate and says so. Folding one in would produce a number no document prints. ## A cache read and a cache write are two meters, and a write is two more `cached_input_per_mtok` is the cached **read** rate: what a request pays for tokens it found already in the cache. Filling the cache is a separate charge, priced above uncached input, and it is the two `cache_write_*` meters. Those two are one meter split by how long the seller promises to keep what was written. The same tokens go in either way; a five minute window and an hour are two prices for two promises, printed side by side on the same row of the same page. That is the definition `batch_*`, `flex_*` and `priority_*` already use, so a retention window joins the rule about service tiers below rather than starting a new one. **Why two dimensions rather than one carrying the window as an attribute.** Three things decide it and none is taste. A sku's `attrs` is a property of the sku, and one model carries both rates at once. Putting the window there means one sku per window, which is the shape [the context band](#a-context-band-is-not-a-dimension) refused for the same reason: it doubles every count the boards print and makes a succession field describe a lineage that is really one product. A [spread](../api/v1/prices/spreads.md) groups by dimension. One dimension would put a five minute write and an hour's write in one group and print the gap between them as a dispersion. It is not one. It is the distance between two products, and the record's own machinery would have manufactured exactly the misreading this meter has to be safe from. And the dimension id is what a figure carries everywhere it goes, into the tape, into a bill line, into a citation. With the window in the id there is no reading of $6.25 that could be the hour's rate. The two names are names, not one name with a number in it. A seller pricing a thirty minute window gets a third dimension rather than a third value of a field, which is not hypothetical: Novita's pricing payload already carries fields for a five minute and a thirty minute write, both empty of figures on 2026-09-10. **One seller is on these meters. Four more publish a write rate and none of them is filed**, for reasons worth keeping apart because only one of them is about the market. Read 2026-09-10. - **Anthropic** prints both windows in its own pricing table, and is read. - **OpenAI** prints a cache-writes column in each of its four tier tables and names no retention window anywhere on the page, so there is no registered meter its figure belongs to. The temptation is the part worth recording: every row prints its write at exactly 1.25x its own input rate, and 1.25x is the five minute multiplier at every seller who publishes both windows. An inference that clean is exactly the kind this record does not make. - **DeepInfra and Novita** carry theirs inside a script payload the HTML readers strip before any adapter sees it. A different parse strategy, not a new field. - **Azure** lists cache-write meters in the Retail Prices API that the registered query does not ask for, and its meter names say nothing about a window either, so it is held by both of the reasons above at once. Only OpenAI's is a document that has been read and does not say enough. The other three are documents nobody has asked the right question of yet. Novita has since been asked. Its pricing page now prints a write column at five minutes and at an hour, and the adapter has read both since 2026-11-02, so two sellers are on these meters and the second is the first in a peer set. **Google is absent from these meters and is not one of those cases.** It prices cache **storage**, per million tokens per hour, which is rent for holding a cache rather than a charge for filling one, and a third unit as well as a third meter. An hour appears in that figure and in `cache_write_1h_per_mtok`, and the two have nothing to do with each other. `rerank_per_mtok` is still the only rerank meter. Some providers sell reranking per search instead. That is not a point on the same scale, so it gets its own dimension when a source publishes one, rather than being converted into this one at an assumed document count. ## What a cache write can and cannot be set against Asked because a meter that draws no board should say so rather than have somebody find out by drawing one. **Against a cache read: never.** A write and a read are different events at different prices and the dispersion between them is not a fact about a market. Nothing has to remember this. The spread groups by dimension and they are two. **Against a write at another window: never.** Five minutes of retention and an hour of it are two products, and the gap between them is the price of the longer promise rather than a disagreement between sellers. This is guarded the same way and only because the window is in the id. **Against another seller's write at the same window: yes, with a note.** Both meters are in the model board's own unit, so nothing bars them from a spread. A dispersion on this board is several sellers of one product, grouped by the product slug an open-weight row carries. Novita's write columns reached three such peer sets in the snapshot of 2026-11-03 (Kimi K3, Qwen3.8 Max and Qwen3.8 Flash), and Novita is the only seller filing a write in each, so no write spread is drawn yet. The reason is coverage rather than category. The note is the one thing that had to be settled before the first such spread, and it was settled when those peer sets arrived. Sellers cache in blocks, and the block size is theirs to choose: DeepInfra's own page states a per-model granularity running from 1,024 to 8,192 tokens with the remainder billed as standard input. Two sellers quoting the same rate per million tokens at the same retention window are therefore not necessarily selling the same thing, in the way two per-page extraction figures are not. So every answer that sets one seller's write against another's carries it as `blockNote`: the [spread](../api/v1/prices/spreads.md#cache-writes) and the peer range a [quote check](../api/v1/prices/quote-check.md) reads. No other meter does. ## A character is not a token `translate_input_per_mchar` is per million characters. `input_per_mtok` is per million tokens. The record does not convert between them and will not. A conversion needs a characters-per-token ratio, and no such constant exists. It is a property of a tokenizer and a language at once: near four characters to the token for English, far denser for Chinese, Japanese, Korean and Thai. Translation crosses languages by definition, so a single ratio would be wrong by a multiple in exactly the work this class is about. The meters are not counting the same events either. Google states that markup, whitespace and untranslated characters are all charged, and that a request with no content is charged one character. No token meter behaves that way. So both prices stand in their own unit, side by side, and the ratio between them is a measurement somebody takes on their own corpus. That is the same rule the record already applies to quality: measured on your own documents, or not claimed. A dedicated translator priced per million characters standing beside a general model priced per million tokens is a comparison worth having, and it is worth having in two units rather than in one invented one. ## A service tier is a dimension Providers sell the same model at more than one speed. A batch tier runs the work whenever there is room and costs less; a priority tier runs it sooner and costs more; a flex tier sits between standard and batch. Each is a separate rate the provider publishes on the same page, and each is registered as its own meter. That is what `batch_input_per_mtok` has always been, so the six tier meters are one rule rather than three special cases: `batch_*`, `flex_*` and `priority_*` are the same model billed under a different service promise, and a buyer chooses between them per request. `priority_*` is named for the meter rather than for what any one seller calls it this year. OpenAI's page prints "Fast pricing data" and states in its own footnote that the tier was called Priority processing until 30 July 2026 and that `service_tier: "priority"` is still accepted; Anthropic prints "Fast mode pricing" and takes `speed: "fast"`. An id tracking the marketing word would have needed renaming once already, so the id is Vitna's and the [locator](observation.md) quotes the page. Sharing a dimension is not a claim that two sellers' tiers are one product. A dimension is the meter; a peer set is the product, and these two sellers share none, so no spread is drawn across them. `container_hour` already carries that distinction for OpenAI's sized containers against Anthropic's unsized one. ## A context band is not a dimension Some providers price one model differently by how much context a request carries. That is a property of the price rather than a new meter, so it travels on the figure as a band rather than as a dimension of its own. The other option was to register a second long-context dimension beside `input_per_mtok`, and it was refused because it would have meant every consumer learning a second name for the same meter, and every unbanded sku carrying a column it never fills. See [the tape](../api/v1/prices/tape.md#context-bands). ## Money Amounts are `numeric(14,6)` USD, in the unit the dimension declares. Six decimal places because a per-token price is quoted per million tokens and dividing back to a single token has to stay exact. --- # Evidence The archived document a figure was read from, addressed by the sha256 of its bytes. Every priced cell on the tape carries an `evidenceSha256`. That hash is the link between a number on a page and a document that existed. ## What is public and what is gated Deliberately split: **Public**, for any hash, with no key: - the URL that was fetched - the time it was fetched - the HTTP status and response headers - the user agent, the adapter, the parser version, the byte count - what robots.txt said - every subsequent read that found the same bytes **Gated** behind a bearer token: the archived copy of the document itself. ```bash curl https://vitna.ai/api/v1/evidence/ ``` ## Why that split The public half is what a verification actually needs. Given the URL and the fetch time, you can fetch the page yourself and compare hashes. Vitna does not have to be trusted for that check to work, which is the entire design goal. The archived copy is a third party's page. Republishing it wholesale raises a copyright question that this record does not need to answer in order to be checkable, so it is held and not served openly. ## What a hash proves That the bytes Vitna read hash to this value, and that a figure was parsed out of them at a stated place by a stated parser version. It does not prove the provider published the price. Nobody signs their own pricing page. What ties a figure to a provider is the archived document plus the URL and time it was fetched from, and those record what Vitna fetched. That distinction is stated in the API responses themselves rather than only here, because it is the claim most likely to be overstated by someone quoting the record. ## Where the archive lives On disk, at `VITNA_EVIDENCE_DIR`, defaulting to `/srv/vitna/evidence`. Each document has a sidecar holding the fetch metadata above. Documents are stored gzipped. The hash is of the document as fetched, before compression, and the byte count is its uncompressed length, so the body the endpoint returns hashes to the key you asked for. Documents archived before 2026-10-14 were stored uncompressed, and both forms are read. The read that first archived a document is on the sidecar. Every fetch that has since found the same bytes is in the database, and both come back together from the evidence endpoint. --- # Market event A published statement that something happened in the market. The rows the market log renders and the feed carries. ## Kinds Constrained in the database to five: | Kind | Meaning | |---|---| | `launch` | Something became available | | `price` | A price moved | | `policy` | Terms, limits or conditions changed | | `deprecation` | Something is going away | | `availability` | Access or region changed | ## Origins | Origin | Written by | |---|---| | `curated` | A person, through `npm run event:add` | | `detected` | Change detection, from two archived documents | ## The inclusion rule for curated events A curated event is refused unless its `sourceUrl` sits on one of the provider's own registrable domains, and that domain list is a column in `witness.provider`, not a habit. The consequence is worth stating plainly: an accurate news report of a price change cannot become a curated event. Only the provider's own page can. The rule is a constraint precisely so that it cannot be relaxed for one entry because someone was confident. `sourceVerifiedOn` records the day the URL was checked and found to answer. ## The inclusion rule for detected events A detected `price` event must name both sides of the comparison it came from, `fromPeriodId` and `toPeriodId`, and a database constraint enforces it. A detected move that cannot say what it moved from is not a move, it is an assertion. Detected launches name neither, because there is no before: a launch row is written when a person promotes a [candidate](sku.md) the collector saw. ## Corrections are new rows `supersedesId` names the event a later row replaces. Nothing is edited. The log is what the record said and when it said it, including when it was wrong, and a log that quietly rewrote its own history would not be evidence of anything. A correction is a curated row, and it keeps the date of the row it restates, saying the true date in its text where the two differ. It may restate a curated row or a detected one. The committed log names the row by its day and headline, and a detected row by its origin as well, because a detected row is in the database alone: the seed finds it there by origin, provider, day and headline, and leaves the correction unpointed rather than guessing when it finds none or more than one. A superseded row stays in the log, marked, and leaves every summary of it: the week digests, the sized moves and the homepage count only rows that still stand. Curated rows carry a partial unique index on `(occurredOn, title)`, so seeding the log is idempotent and can run after every deploy without inserting duplicates. Detected rows are excluded from that index, because two genuine moves on one day for two cells can share a headline by coincidence and the record should keep both. ## Reading events ```bash curl "https://vitna.ai/api/v1/events?kind=price&since=2026-01-01" curl https://vitna.ai/api/v1/feed.json ``` See [events](../api/v1/log/events.md) and [feed.json](../api/v1/log/feed.md). --- # Observation One figure, parsed out of one document, at one moment. The atom of the record. `witness.observation` is append-only. An observation is never edited and never deleted, so anything derived from it can be recomputed and checked later. ## Fields | Field | Meaning | |---|---| | `id` | Serial id, the thing `/api/v1/observations/{id}/proof` takes | | `fetchId` | The read this came out of | | `skuId` | What was priced | | `dimensionId` | What it was priced by | | `amountUsd` | The figure, `numeric(14,6)` | | `observedAt` | When it was read | | `derived` | Whether Vitna computed this rather than read it | | `derivation` | How, when `derived` is true | | `locator` | Where in the document it was found | | `parserVersion` | Which version of the adapter read it | ## One figure per cell per read A unique index on the fetch, the sku, the dimension and the [context band](../api/v1/prices/tape.md#context-bands) means one read of one document yields at most one figure for any one cell. A page that lists the same model twice cannot produce two conflicting observations from a single fetch. ## Derived figures announce themselves Some numbers are not printed on the page. A provider that prices a 20-minute sandbox session has an hourly rate implied but not published, and Vitna computes it. A database constraint requires `derivation` to be non-null whenever `derived` is true, so a computed figure cannot enter the record without saying how it was computed. `derived` travels all the way out to the API, on every tape row and every spread entry. ## `locator` and `parserVersion` exist for the argument you will have later When a figure looks wrong, the question is always the same: did the provider change the price, or did Vitna misread the page? `locator` says where in the document the parser looked and `parserVersion` says which parser looked there, so that question is answerable a year later against an archived document rather than debated. ## From observation to price An observation is a reading, not yet a price. Readings accumulate into [price periods](price-period.md), and change detection publishes a move only on the second consistent read. --- # Price period One stretch during which a price did not move. The history unit. Where an [observation](observation.md) is a reading, a period is the claim built from readings: this sku cost this much, for this dimension, from here to there. ## Fields From `witness.price_period`: | Field | Meaning | |---|---| | `skuId`, `dimensionId` | The cell this period belongs to | | `amountUsd` | The price during the period | | `validFrom` | When it started | | `validTo` | When it ended, `null` while current | | `firstObservationId` | The reading that opened it | | `lastObservationId` | The most recent reading confirming it | | `confirmed` | Whether a second consistent read has confirmed it | | `flap` | Whether the value moved and moved back | ## `validTo IS NULL` is the tape The current price of any cell is its one open period. A partial unique index enforces exactly one open row per `(skuId, dimensionId)`, so the record cannot hold two current prices for one thing. That is a constraint rather than an assumption, because the alternative is a tape that is quietly self- contradictory. A second constraint requires `validTo >= validFrom`, so a period cannot run backwards. ## Confirmation `confirmed` is what separates a reading from a price. A figure seen once is recorded, but a move is published only on the second consistent read. One malformed page, one partial deploy on the provider's side, or one transient parse failure therefore does not enter the log as a price change. The cost is latency: a real price cut shows up on the second read rather than the first. That is the trade the record makes on purpose, because a log that is occasionally wrong is worth less than one that is slightly late. ## `flap` Set when a value moved and moved back. Providers do deploy a wrong price and withdraw it, and a record that reported both moves as decisive would be reporting noise as signal. ## Reading history ```bash curl "https://vitna.ai/api/v1/skus/gpt-5-pro/history?dimension=input_per_mtok" ``` History exists only in the live record. A checkout serving the committed snapshot answers with the sku, an empty `periods` array, and a `meta.note` saying why, rather than inventing a single period out of the current price. --- # Provider term What a provider published about **how** it handles the work, beside what it charges for it. Retention, training use, processing regions, whether a key you hold is accepted. A workload can require one, and a candidate whose provider does not meet it is excluded before anything is weighed. ## It is not a price A [price](price-period.md) is a figure with a unit. A term has neither: it is a yes, a no, or a list of regions. It passes the same admissibility rule the tape applies, because it is a statement on a document the provider publishes on its own domain that can be fetched, archived and dated. It fails the tape's shape, because there is nothing to divide. So it sits beside the tape and never on it: no board shows one, no spread compares one, no [reference rate](reference-rate.md) is computed from one. ## It is curated, not parsed A policy page is prose and the answer is a yes or a no. An adapter deciding what "we do not retain your data" means would be interpolating exactly where the honesty contract forbids it. So a term is entered by a person with `npm run term:add`, under the four refusals the [curated log](market-event.md) already applies: no source URL no row, http is not a source, a host that is not the provider's is not the provider, and a URL that does not answer 200 today is not verified today. ## The four kinds | Kind | Value | Answers | |---|---|---| | `zero_retention` | `yes` or `no` | Whether prompts or responses are kept after answering | | `no_training` | `yes` or `no` | Whether customer data is trained on | | `regions` | `us,eu` | Which regions the work is processed in | | `byok` | `yes` or `no` | Whether a key the customer holds is accepted | Each kind is named for what a `yes` means, so `zero_retention = yes` is a provider that retains nothing. `observed_on` is the date somebody read the page, which is the date the term is true as of. It is never a date the provider states: policy pages rarely carry one, and a date nobody published would be invented here. ## What a workload does with one ```bash npm run aperture:workload:edit -- acme/support --constraint requiresZeroRetention=true ``` | Constraint | Meaning | |---|---| | `requiresZeroRetention=true` | The provider must publish `zero_retention = yes` | | `requiresNoTraining=true` | The provider must publish `no_training = yes` | | `requiresRegions=us,eu` | Every region the provider processes in must be one of these | | `allowsProviders=anthropic,openai` | Who may see this work at all | `allowsProviders` is not a term, and that is deliberate: no provider publishes a document saying whether this customer approves of it. It is a fact about the organisation, so it lives on the workload alone. A list that does not name `openrouter` keeps the substrate out as well. Since vitna-runtime 0.3.0 the signed policy then carries no OpenRouter name for that workload's lanes, fallbacks or rules, so no runtime can route its work through OpenRouter whatever keys it holds. ## A requirement the record cannot check is not satisfied This is the whole of the design and the only part worth arguing about. A workload that requires zero retention, weighed against a provider whose retention the record has never read, could be answered three ways: pass it, fail it, or ignore the requirement. The first two both invent a fact. **The third is the one that looks harmless and is worst**, because the brief would then be silent about a constraint the customer wrote down. So the candidate is excluded, and the refusal names what is missing: > this workload requires that its provider retains no prompt or response data, > and the record holds no zero_retention term for fireworks. A requirement the > record cannot check is not satisfied, and assuming either answer would put a > fact in the record that nothing supports. That is the same rule `costPerUnit` already applies to a price the tape does not carry. ## What it closes Before this existed, exactly one of these was representable, and it was `VITNA_ALLOW_PROVIDERS`: an environment variable an operator typed on a box. The [runtime](../runtime/serving.md) honours it before every route including a fallback, which is the right enforcement, and nothing in the record knew it existed. An operator who wrote down a compliance boundary had said something no decision recorded and no brief could cite. Part of the decision was living outside the decision record. A decision made under a requirement now names the `witness.provider_term` rows that answered it, in `inputs.termIds`, and the brief prints them. "Why was this candidate allowed to serve" resolves to a document somebody read on a date, exactly as a figure resolves to a period. --- # Reference rate One number per product per day: the median list price among the providers publishing one, by a rule with no judgement in it. ```bash curl https://vitna.ai/api/v1/rates ``` ## The rule 1. A series exists for a product that at least three distinct providers publish a confirmed price for on the day. 2. The rate is the median of those prices: the middle one, or the mean of the two middle ones, with the halfway case rounded up. 3. Arithmetic is exact, in whole millionths, the precision the record stores. 4. A price read once and not yet confirmed by a second read does not count. 5. A series is never restated. A day is published once, and a correction is a new row on a later day. Rule 5 is the one that gives the series its value. A benchmark that can be quietly recomputed after the fact is not a benchmark, it is an opinion with a timestamp. ## What a rate is not > [!WARNING] > A reference rate is not a price anyone can buy at, not a volume-weighted > average, and not a forecast. All three are reasonable assumptions about > something called a reference rate, and none of them is true. Published inside the API response itself, in an `isNot` array, because all three are things a reader could reasonably assume and none is true: - **not a price anyone can buy at.** It is a median of list prices. No seller is obliged to be at it. - **not a volume-weighted average.** The record holds no volumes. Vitna sees published prices, not transactions. - **not a forecast.** It is a statement about a day that has already happened. ## Why the method travels with the numbers The rule is served in the same response as the rates rather than linked from it. A benchmark whose method a reader has to go and find is one most readers will use without reading, and the entire claim being made here is that the rule is fixed and checkable. ## Recomputing ```bash npm run witness:rates -- --verify ``` Recomputes every day ever published and reports disagreement. `--dry-run` writes nothing. Because inputs are append-only and the rule has no free parameters, an independent implementation reading `/api/v1/skus/{slug}/history` should arrive at the same series. That is the intended check. --- # Sku The unit Vitna prices: one thing, sold by one provider, that you could buy. `gpt-5-pro` from OpenAI is a sku. The same open-weight model served by three different providers is three skus, because three sellers are quoting three prices for it, and the difference between those prices is exactly what the record exists to show. ## Fields From `witness.sku`: | Field | Meaning | |---|---| | `id` | Stable internal id | | `slug` | The public identifier, unique across every class | | `name` | What it is called | | `classId` | Its [component class](component-class.md) | | `providerId` | Who sells it | | `attrs` | Context window, form factor, tier, and for compute the product row it prints on | | `succeedsSkuId` | The sku this one replaces, where a provider named one | | `featured` | Whether it appears in the three-row tape on the homepage | | `status` | `active` by default | `slug` is what you pass to the API: ```bash curl https://vitna.ai/api/v1/skus/gpt-5-pro/history ``` ## Sku against product A sku is a seller's offer. A **product** is the thing several sellers are offering, and it lives in `attrs` rather than as its own table. `H100 SXM` is a product. Six providers renting one are six skus grouped on it. That grouping is what makes a [spread](../api/v1/prices/spreads.md) computable: without it there is no peer set to take a minimum and a maximum over. Product slugs are lowercased and dashed, so `H100 SXM` becomes `h100-sxm`. ## Featured is a property of the sku Which three rows the homepage tape shows is stored on the sku, not chosen in the page. The marketing surface and the market therefore cannot drift: there is no slice taken at render time that could disagree with the record. ## How a sku comes to exist Not by being typed in. The collector reports priced rows it saw that the registry does not track, as candidates in `witness.sku_candidate`. It counts sightings and holds `UPDATE` on the counting columns only. Promoting a candidate into a sku is a person's act, and it is what writes the `launch` [market event](market-event.md) recording when Vitna first read the row. Registering a back catalogue that a provider did not just launch uses `--backfill`, which registers the sku and writes no event, because claiming a launch that did not happen would put a false statement in the log. --- # Source One fetchable document, and the adapter that knows how to read it. Sources are registry rows in `witness.source`. ## Fields | Field | Meaning | |---|---| | `id` | Stable source id | | `providerId` | Who publishes it | | `url` | What the collector fetches | | `publicUrl` | The page a citation should point a human at | | `adapter` | The parser that reads this document | | `enabled` | Whether the collector reads it | | `cadenceMinutes` | How often, default 60 | | `robotsStatus` | What robots.txt said | | `notes` | Free text | ## Why `url` and `publicUrl` are two columns What a machine should read is not always what a person should read. Anthropic's pricing parses cleanly from the documentation's Markdown mirror; Google's needs a locale-pinned URL to answer consistently. Citing those would send a reader to a mirror or to a URL carrying a locale parameter they did not choose. So the tape cites `publicUrl` and the collector fetches `url`, and a citation never points at scaffolding. ## Provider domains are the inclusion rule `witness.provider` carries a `domains` array, and it does real work. A curated [market event](market-event.md) is refused unless its source URL sits on one of the provider's own registrable domains, subdomains included. That turns "is this the provider's own announcement?" from a judgement call made per entry into a constraint the database enforces. A press summary, a news article or a well-informed blog post cannot become a curated event, however accurate it is. ## Health ```bash curl https://vitna.ai/api/v1/sources/health ``` Public, deliberately. A source that has stopped answering is part of the record: a tape that silently stopped updating because a page moved looks exactly like a market that stopped moving. The endpoint flags a source `stale` past its cadence and `failing` at three or more consecutive failures. It answers `503` when the record cannot be read at all, rather than reporting green from a committed file, which would be the one lie this system exists to prevent. ## Fetch outcomes Every read writes one `witness.fetch` row with a final status, inserted once after both the fetch and the parse have been decided, so nothing about a fetch is ever rewritten: | Status | Meaning | |---|---| | `ok` | Fetched and parsed. Has a `contentSha256`. | | `fetch_failed` | Could not be retrieved. Has an error. | | `parse_failed` | Retrieved, but the adapter could not read it. Has an error. | | `robots_blocked` | robots.txt disallowed the read. | Two database constraints hold the shape: an `ok` row must have evidence, and a non-`ok` row must have a reason. --- # API v1 JSON over HTTP. Reading the public record takes no key, no signup and no quota. Base URL `https://vitna.ai/api/v1`. Everything public here stays that way. What belongs to one organisation is reached with a credential of its own: its [own feed](engine/org-feed.md), whose address is the credential and which an owner gets from `/console`, and, with the organisation's push key, its [signed artifacts](engine/artifacts.md) and the [usage it pushes](engine/usage.md), the one write this API mounts. The console itself is behind sign-in, and it is not part of this API. ## The index ``` GET /api/v1 ``` ```bash curl https://vitna.ai/api/v1 ``` Every route, each with its method, its path, the page documenting it and one address that answers as it stands. A route that needs a value another answer gives, such as an evidence hash, names where to take it from in `from` instead. Any other path under `/api/v1` answers `404 unknown_route` in the error shape below, never an HTML page. A route asked with a method it does not answer, such as a `GET` on a `POST` route, refuses with `405 method_not_allowed` in the same shape, naming the methods it does answer. ## The envelope Every response is a JSON object carrying a `meta` key alongside its payload. There is no single `data` wrapper: the payload key is named for what it is (`classes`, `tape`, `events`, `periods`, `proposals`). ```json { "class": "model", "tape": [ ... ], "meta": { "generatedAt": "2026-08-23T09:14:02.881Z", "source": "database", "count": 918 } } ``` `meta.generatedAt` is always present. Everything else in `meta` is per-endpoint. **Changes are additive.** A key can join a response. A key a caller already reads is not renamed, retyped or dropped within v1. Response examples are checked against a body the endpoint serves, so an example names no key that is not there, on every page except five: watches, proposals, evidence, artifacts and usage. Those read the engine's register, an on-disk archive or a key, or answer a write, and no test can serve them, so their examples are kept by hand. Bodies are pretty-printed with two-space indentation and end with a newline, so the API is readable in a terminal without piping it through a formatter. ## `meta.source` | Value | Meaning | |---|---| | `database` | The live record | | `snapshot` | The JSON committed in the repository | | `archive` | The on-disk evidence store | `vitna.ai` answers from the record. A checkout with no `DATABASE_URL` answers from the snapshot, and says so rather than pretending. ## Caching and ETags Four policies: | Header | Used by | |---|---| | `public, max-age=60, stale-while-revalidate=3600` | Most endpoints | | `public, max-age=60` | `/keys`, `/keys/policy-key.pub` | | `no-store` | `/health`, `/evidence/{sha256}`, `/sources/health`, `/proposals`, `/watches` | | `private, no-store` | `/feeds/{capability}/*`, refusals included | The uncached set is uncached on purpose. A proposal is accepted by a person and a watch is spent by a collector run, so a minute of staleness there reports that nobody has acted when somebody has. Source health has the same problem in reverse: a cached green is a claim about the past. An organisation's own feed is `private` as well as `no-store`, so that no shared cache keeps one organisation's activity even if a rule in front of the site is ever set to store what `no-store` alone would let it. The key register keeps its minute and gives up the stale window, because a key retired after a compromise must stop being offered to pin within that minute rather than within the hour. Every response carries an `ETag`, computed as a sha256 over the body **with `meta.generatedAt` removed**, truncated to 32 hex characters. Hashing the timestamp too would make every response a fresh entity and the header useless. What a caller wants to know is whether the record changed, not whether time passed. ```bash curl --compressed -H 'If-None-Match: "a1b2c3..."' https://vitna.ai/api/v1/markets ``` A match answers `304` with no body. Send back the tag you were given, whichever form it took. The proxy in front of the site may hand you `"a1b2c3..."` or `W/"a1b2c3..."`, depending on the encoding it chose for the response, and which request gets which is its choice rather than the API's. Both forms match, and so does a comma-separated list of tags or `*`. Only a read that would succeed is weighed against the tag. An error answers as an error whatever `If-None-Match` says, so `*` on an unknown class is still a `404`, never a `304`. ## Reading from a browser Every keyless response carries `Access-Control-Allow-Origin: *` and exposes `ETag` and `Age`, so a page on any origin can read the record and revalidate what it read. A preflight `OPTIONS` to any other `/api/v1` path answers `204`, allowing `GET` and the `If-None-Match` header. It allows no `Authorization` header, so the two routes that take a key cannot be called from another origin's page. The one prefix outside that is `/api/v1/feeds/`, an organisation's own feed. Its answers carry no `Access-Control-Allow-Origin`, and its preflight answers `204` granting no origin and no method, so a page on another origin cannot read it. ## Errors ```json { "error": { "code": "unknown_class", "message": "no component class named \"models\"" }, "meta": { "generatedAt": "2026-08-23T09:14:02.881Z" } } ``` Errors are always `no-store`. Codes in use: | Code | Status | Cause | |---|---|---| | `unknown_route` | 404 | Not a v1 route; `GET /api/v1` lists them | | `method_not_allowed` | 405 | The route does not answer that method; the message names the ones it does | | `unknown_class` | 404 | Not a registered component class | | `unknown_product` | 404 | No such product in that class | | `unknown_dimension` | 404 | Product or sku exists, not metered that way | | `unknown_sku` | 404 | No sku with that slug | | `unknown_provider` | 404 | No registered or priced provider with that id | | `unknown_kind` | 404 | Not one of the five event kinds | | `unknown_evidence` | 404 | Nothing archived under that hash | | `unknown_observation` | 404 | Not an observation id | | `unknown_model` | 404 | No catalog model matches that slug | | `unknown_board` | 404 | No mirrored benchmark board matches that id | | `unknown_manifest` | 404 | No verified Anchor DMA manifest matches that slug | | `unknown_dataset` | 404 | Not one of the five export datasets | | `unknown_feed` | 404 | Not a live address for an organisation's feed, for any reason | | `not_found` | 404 | No decision of a public feed has that id, for any reason | | `no_policy_key` | 404 | No key is registered to sign policies, so there is no `.pub` file to serve | | `bad_since` | 400 | `since` is not an ISO date | | `bad_term_kind` | 400 | Not `zero_retention`, `no_training`, `regions` or `byok` | | `missing_challenge` | 400 | `GET /router-challenge` called with no POST challenge body | | `bad_challenge` | 400 | Router challenge payload missing required workload, incumbent, router, or evidence fields | | `missing_decision_input` | 400 | `GET /decide` called with no POST decision payload | | `bad_decision_input` | 400 | Stateless decision check missing workload, candidates, or evidence fields | | `missing_reconciliation` | 400 | `GET /reconcile` called with no POST reconciliation payload | | `bad_reconciliation` | 400 | Stateless reconciliation missing window, unitProfile, or valid readings | | `missing_watchdog_input` | 400 | `GET /watchdog` called with no POST rollout watchdog payload | | `bad_watchdog_input` | 400 | Stateless rollout watchdog missing workload or valid rule/telemetry fields | | `missing_pipeline_input` | 400 | `GET /pipeline` called with no POST compound pipeline payload | | `bad_pipeline_input` | 400 | Stateless compound pipeline missing stages or valid profile/ledger fields | | `missing_audit_pack_input` | 400 | `GET /audit-pack` called with no POST audit pack or inline Stackfile payload | | `bad_audit_pack_input` | 400 | Stateless audit pack request missing valid `pack` or `stackfile` fields | | `bad_status` | 400 | Not `proposed`, `accepted` or `superseded` | | `bad_state` | 400 | Not `armed` or `spent` | | `unauthorized` | 401 | Evidence body requested without a valid token, or an artifact without an org key | | `record_unavailable` | 503 | The record could not be read | | `no_record` | 503 | Serving the snapshot, and this needs the record | | `not_configured` | 503 | This instance holds no key to check an organisation's feed address with | ## 503 rather than an empty list The rule the whole API keeps: an endpoint that cannot reach the record answers `503`, never `[]`. An empty list reads as "nothing is there". When the truth is "I could not look", that is a lie, and it is exactly the lie a record like this exists to prevent. It matters most on `/sources/health`, `/proposals` and `/watches`, where an empty answer would report a healthy quiet system. The same principle shapes partial answers. `/feed.json` reports its engine half as **absent** rather than empty when it cannot read it, in `_vitna.engineActivity`. ## Types - Money: USD, decimal, in the unit named alongside it. Six decimal places. - Timestamps: RFC 3339, UTC. - Dates: `YYYY-MM-DD`, meaning the whole day. In the feed a day is rendered as its own midnight UTC, since RFC 3339 requires an instant. - Hashes: 64 lowercase hex characters. ## Endpoints Grouped by what they answer about, which is how the navigation lists them too. **Prices**, the boards as data: | Page | Endpoint | |---|---| | [Markets](prices/markets.md) | `GET /markets` | | [Tape](prices/tape.md) | `GET /markets/{class}/tape` | | [Spreads](prices/spreads.md) | `GET /spreads/{class}/{product}` | | [Sku history](prices/sku-history.md) | `GET /skus/{slug}/history` | | [Snapshot](prices/snapshot.md) | `GET /snapshot` | | [Reference rates](prices/rates.md) | `GET /rates` | | [Providers](prices/providers.md) | `GET /providers`, `GET /providers/{id}` | | [Provider terms](prices/terms.md) | `GET /terms`, `GET /providers/{id}/terms` | | [LiteLLM cost map](prices/litellm-cost-map.md) | `GET /gateways/litellm/model-prices.json` | | [Quote check](prices/quote-check.md) | `POST /quote-check` | | [Catalog, benchmarks and capability](prices/catalog.md) | `GET /catalog/models`, `GET /catalog/models/{slug}`, `GET /catalog/models/{slug}/sellers`, `GET /catalog/skus/{slug}/capabilities`, `GET /catalog/benchmarks`, `GET /catalog/capability` | | [Historical data exports](prices/exports.md) | `GET /exports/{dataset}`, Deploy | **The log**, what happened and when: | Page | Endpoint | |---|---| | [Events](log/events.md) | `GET /events` | | [Feed](log/feed.md) | `GET /feed.json` | **Verification**, checking a figure without taking anyone's word for it: | Page | Endpoint | |---|---| | [Attestations](verification/attestations.md) | `GET /attestations` | | [Observation proof](verification/proof.md) | `GET /observations/{id}/proof` | | [Citations](verification/citations.md) | `GET /citations/{id}` | | [Evidence](verification/evidence.md) | `GET /evidence/{sha256}` | | [Signing keys](verification/keys.md) | `GET /keys`, `GET /keys/policy-key.pub` | **The engine**, what was decided and what is being watched, and the routes an organisation's own credential reaches: | Page | Endpoint | |---|---| | [Proposals](engine/proposals.md) | `GET /proposals` | | [Watches](engine/watches.md) | `GET /watches` | | [Decision pack](engine/decision-pack.md) | `GET /decisions/{id}/pack` | | [Router challenge](engine/router-challenge.md) | `POST /router-challenge` | | [Stateless decision check](engine/decide.md) | `POST /decide` | | [Stateless invoice reconciliation](engine/reconcile.md) | `POST /reconcile` | | [Stateless rollout watchdog](engine/watchdog.md) | `POST /watchdog` | | [Stateless compound pipeline analysis](engine/pipeline.md) | `POST /pipeline` | | [Stateless audit attestation pack](engine/audit-pack.md) | `POST /audit-pack` | | [Relay spreads and Anchor manifests](engine/relay-anchor.md) | `GET /relay/spreads`, `GET /anchor/manifests`, `GET /anchor/manifests/{slug}` | | [Model Context Protocol (MCP)](engine/mcp.md) | `GET /mcp`, `POST /mcp` | | [Your organisation's feed](engine/org-feed.md) | `GET /feeds/{capability}/feed.json`, `GET /feeds/{capability}/rss.xml`, private | | [Collecting a signed artifact](engine/artifacts.md) | `GET /artifacts`, `GET /artifacts/{id}`, with the organisation's push key | | [Pushing usage](engine/usage.md) | `POST /usage`, with the organisation's push key | | [Organisation workloads](engine/org-workloads.md) | `GET /org/workloads`, `POST /org/workloads`, `POST /org/workloads/{slug}`, key-scoped | | [Organisation Stackfile](engine/org-stackfile.md) | `GET /org/stackfile`, `POST /org/stackfile`, key-scoped | | [Workload counterfactual cost](engine/org-counterfactual.md) | `GET /org/workloads/{slug}/counterfactual`, key-scoped | | [Organisation dependency exposure](engine/org-exposure.md) | `GET /org/exposure`, key-scoped | | [Organisation rollout watchdog](engine/org-watchdog.md) | `GET /org/watchdog`, key-scoped | | [Organisation compound pipeline](engine/org-pipeline.md) | `GET /org/pipeline`, key-scoped | | [Organisation audit attestation pack](engine/org-audit-pack.md) | `GET /org/audit-pack`, key-scoped | | [CI evaluation evidence](engine/org-evidence.md) | `GET /org/evidence`, `POST /org/evidence`, key-scoped | | [Headless artifact signing requests](engine/org-artifact-requests.md) | `GET /org/artifact-requests`, `POST /org/artifact-requests`, `GET /org/artifact-requests/{id}`, key-scoped | | [Policy drift webhooks](engine/org-webhooks.md) | `GET /org/webhooks`, `POST /org/webhooks`, `POST /org/webhooks/{id}/test`, `POST /org/webhooks/{id}/disable`, Deploy | | [Organisation members](engine/org-members.md) | `GET /org/members`, `POST /org/members`, `POST /org/members/remove`, Deploy | | [Organisation API keys](engine/org-keys.md) | `GET /org/keys`, `POST /org/keys`, `POST /org/keys/{id}/revoke`, Deploy | **The Unblocker**, the browser-backed web access alpha, which reads no part of the record: | Page | Endpoint | |---|---| | [Unblocker](../../unblocker/overview.md) | `POST /unblock`, with an Unblocker client key, relayed to the hosted service at https://unblock.vitna.ai, `503` only where the relay is not configured | **One key**, the question the hosted APIs ask about an org's platform key, which only their services may ask: | Page | Endpoint | |---|---| | [One key for the hosted APIs](../../one-key.md) | `POST /keys/verify`, with a hosted API's service token | | [One key for the hosted APIs](../../one-key.md) | `POST /receipts`, with a hosted API's service token: the receipts for an org's requests, for its console | **Status**, whether the record and its sources are answering: | Page | Endpoint | |---|---| | [Health](status/health.md) | `GET /health` | | [Source health](status/sources.md) | `GET /sources/health` | --- # Collecting a signed artifact `GET /api/v1/artifacts` lists what has been signed for your organisation, and `GET /api/v1/artifacts/{id}` hands over the bytes, exactly as they were signed. ``` GET /api/v1/artifacts GET /api/v1/artifacts/{id} ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." https://vitna.ai/api/v1/artifacts curl -H "authorization: Bearer vitna_push_..." https://vitna.ai/api/v1/artifacts/12 -o policy.json ``` This is how a signed policy reaches the machine that runs it without anybody at Vitna running a command. Before it existed, `aperture:sign` wrote the bytes to a row and only an operator could read them out, which is a different operator doing the same manual step. ## Two acts, two credentials, and neither can do the other Asking for an artifact is a write and it does not happen here. An owner of the organisation asks at `/console/deployment`, signed in, through a role that may write on their say-so. The signer answers on its own timer, rebuilding the document from the record rather than signing anything it was handed. Collecting is a read, and it happens here, with the organisation's own key. A person signed in to the organisation, owner or member, collects the same bytes from `/console/deployment` instead, which also prints the key that signed each file. This route is for a machine with no session: a CI job, or a deploy script. The route checks the key through the one database role that checks org keys, and reads the artifacts through a role that is read-only at the role level and cannot see a key at all. Neither can ask for an artifact, settle a request or write one. A leaked org key can take away what was already signed. It cannot make Vitna sign anything. ## The key An org key with the `push` scope, minted once by an operator: ```bash npm run org:key:create -- --org convexity --label "CI, collecting policies" ``` A router key is refused here and says so. The two are separate capabilities on purpose: a router key spends money at a provider, and reading what your register decided is not that. A push key presented to the router is refused the same way. A request with no key, or with a key the record does not hold or has revoked, answers `401` with `"code": "unauthorized"` beside the `refusals`, the same code the rest of v1 uses for the same refusal. A revoked key is told the date it was revoked and not the reason. The likeliest bearer of a revoked key is whoever found it, and a member of the organisation reads the reason beside the key at `/console/settings`. ## What a listing answers ```json { "org": "convexity", "artifacts": [ { "id": 12, "kind": "policy", "outcome": "signed", "at": "2026-08-31T12:04:11.000Z", "digest": "9f2c...", "decisions": [41], "publicKey": "MCowBQYDK2Vw...", "refusedBecause": null, "body": "/api/v1/artifacts/12" } ], "note": "every signed row is bytes you can take away, and every refused row is a question that was asked and declined." } ``` **Refused rows are in the listing.** An organisation that asked for an artifact and did not get one is owed the reason, and a listing showing only successes would answer "what have we signed" rather than "what did we ask for". A refused row carries `refusedBecause` and no digest, and collecting it answers `409` with the reason rather than handing over a file that is not one. `publicKey` is which key signed. It is the public half, the record publishes it at `/witness`, and you need it to know which of the keys the record acknowledges to check this against. The newest fifty are listed, newest first. ## What a collection answers The bytes, with `content-type: application/json`, a filename in `content-disposition`, and the digest in `x-vitna-digest` so a caller can check the transfer without parsing the document first. **Byte for byte.** Not re-serialised, not pretty-printed, not wrapped in an envelope. The signature covers these exact bytes, so anything that touched them on the way out would break the one check you are meant to run: ```bash npm run aperture:policy -- --verify policy.json ``` An id belonging to another organisation answers `404`, not `403`. A `403` would confirm the id exists, which turns a list of guesses into a census of who has signed what. ## What "short-lived" means here, precisely `RUNTIME.md` open decision 9 calls this an authenticated short-lived download, and it is worth saying what that turned out to be rather than implying a mechanism that is not here. There is no signed URL and no download token. Every request carries the organisation's key and is authorised on the spot, so **there is no artifact URL that works without the credential**, and therefore nothing to expire. That is a different property from a short expiry on a link anybody holding it can use, and a stronger one, so it is described rather than borrowed. ## What this does not do **It does not fetch a policy for a running runtime.** Nothing here is in a request path and no runtime calls it while serving. A runtime loads its policy from local disk at boot; putting the file there between deploys is a management-plane act, and a runtime that fetched its policy per request would put Vitna's availability inside its own, which is the dependency the whole architecture refuses. **It does not sign.** The key is not in this process, and `tests/environment-boundary.test.mjs` fails if anything a page can import so much as names it. --- # Stateless audit attestation pack Verify a `vitna.audit-pack.v1` document across all six cryptographic and governance layers, or assemble an inline audit pack from a `vitna.stackfile.v1` payload in memory with no API key and zero retention. ``` POST /api/v1/audit-pack ``` ```bash curl -X POST https://vitna.ai/api/v1/audit-pack \ -H "content-type: application/json" \ -d '{ "mode": "verify", "pack": { ... }, "trustedRecordPublicKey": "MCowBQYDK2VwAyEA...", "requirePackSignature": false, "failOnRollback": true, "maxMonthlyCostUsd": 5000 }' ``` ## Modes `POST /api/v1/audit-pack` operates in memory (`Cache-Control: no-store`) with two modes: 1. **Verify mode** (`verify`, default when `pack` or a direct `vitna.audit-pack.v1` body is posted): runs [`verifyAuditPack`](../../../routing/audit-packs.md) across canonical SHA-256 `digest` integrity, verbatim `claims.citationBoundary` preservation, outer Ed25519 `packSignature` verification, `stackfileDigest` + signed `policy.json` + `vitna.lock.json` lock invariants, RFC 6962 Witness observation Merkle proofs and signed daily roots, provider compliance term and evaluation digests, and optional CI governance gates (`requirePackSignature`, `requireAttestedPrices`, `failOnRollback`, `maxMonthlyCostUsd`). 2. **Assemble mode** (`assemble`, when `stackfile` is posted): validates an inline `vitna.stackfile.v1` alongside optional `policyText`, `policyLock`, `ledgerRows`, `evidence`, `terms`, `monthlyRuns`, and `discoverPeers`, evaluates rollout watchdog and compound pipeline BOMs against Witness rates, and returns an unsigned `vitna.audit-pack.v1` stamped with its canonical SHA-256 `digest`. Payloads and pinned keys are never persisted. Also exposed to MCP clients as `vitna_verify_audit_pack` on [`POST /api/v1/mcp`](mcp.md). ## Verification response shape ```json { "mode": "verify", "ok": true, "verdict": "verified", "orgId": "convexity", "generatedAt": "2026-11-06T00:00:00.000Z", "digest": "4f3b9a8e01c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8", "checks": { "envelope": { "schemaValid": true, "digestMatch": true, "citationBoundaryIntact": true }, "packSignature": { "present": true, "verified": true, "signerPublicKey": "MCowBQYDK2VwAyEA...", "pinnedKeyMatched": true }, "stackfileAndPolicy": { "stackfilePresent": true, "stackfileDigestMatch": true, "policyPresent": true, "policyVerified": true, "lockPresent": true, "lockVerified": true }, "witnessPriceAttestations": { "totalProofs": 4, "attestedProofs": 4, "unattestedProofs": 0, "verifiedProofIds": ["101", "102", "103", "104"], "pinnedRecordKeyUsed": true }, "evidenceAndTerms": { "providerTermCount": 4, "customerEvidenceCount": 2 }, "governanceGates": { "watchdogVerdict": "healthy", "totalEstimatedMonthlySpendUsd": 1220 } }, "failures": [], "warnings": [], "privacy": "Evaluated in memory with zero retention; audit attestation packs, inline Stackfiles, policy envelopes, ledger rows, and pinned public keys are never stored.", "meta": { "generatedAt": "2026-11-06T00:00:00.000Z", "mode": "verify", "verdict": "verified" } } ``` --- # Stateless stack decision check Evaluate an inline workload, candidate model list and evaluation measurements against live Witness prices and provider compliance terms in memory, with no API key and zero retention. ``` POST /api/v1/decide ``` ```bash curl -X POST https://vitna.ai/api/v1/decide \ -H "content-type: application/json" \ -d '{ "workload": { "id": "ci/support-triage", "classId": "model", "incumbentSkuId": "openai/gpt-5.6-luna", "unitLabel": "tickets", "unitProfile": { "inputTokensPerUnit": 2000, "outputTokensPerUnit": 500 }, "monthlyVolume": 100000, "constraints": { "qualityFloor": 94.0, "economicThresholdPct": 15, "edgeCaseVeto": ["pii-leak"], "requiresZeroRetention": true }, "priority": "economics" }, "candidates": ["anthropic/claude-sonnet-4-6"], "evidence": [ { "skuId": "openai/gpt-5.6-luna", "measuredAt": "2026-09-14", "metrics": { "successRatePct": 96.0 } }, { "skuId": "anthropic/claude-sonnet-4-6", "measuredAt": "2026-09-14", "metrics": { "successRatePct": 95.5 } } ] }' ``` ## How candidates are weighed Every candidate runs through the same pure `decide()` engine used by `aperture:decide`: 1. **Eligibility** against `witness.provider_term` (`requiresZeroRetention`, `requiresNoTraining`, `requiresRegions` and `allowsProviders`), [three-provenance capability and runtime speed bounds](../../../routing/capabilities.md) (`requiresCapabilities` and `minContextTokens` over `declared_mirror`, `minThroughputToksSec` and `maxTtftMs` over `measured_third_party`, and `requiresVerifiedCapabilities` over customer `verifiedCapabilities` evidence carrying `n` and `sourceRef`), citing the exact term IDs, mirror readings, and customer evidence rows inspected. 2. **Quality floor** (`candidate.successRatePct >= constraints.qualityFloor`). 3. **Critical edge-case vetoes** (intersection of reported `edgeCaseFailures` with `constraints.edgeCaseVeto`, downgrading otherwise economic candidates to `test`). 4. **Cost per successful unit** over the Witness market tape (`savingsRatePct >= economicThresholdPct`). Inline workload profiles, candidate lists and measurements are evaluated purely in memory (`Cache-Control: no-store`) and never written to disk or the database. ## Response shape ```json { "workload": { "id": "ci/support-triage", "name": "Stateless CI check", "classId": "model", "incumbentSkuId": "openai/gpt-5.6-luna", "incumbentName": "GPT-5.6 Luna", "unitLabel": "tickets", "monthlyVolume": 100000, "priority": "economics", "activeRequirements": ["requiresZeroRetention"] }, "recommended": { "action": "switch", "skuId": "anthropic/claude-sonnet-4-6", "name": "Claude Sonnet 4.6", "savingsRatePct": 24.5, "annualSavingsUsd": 3420, "reason": "The candidate clears the quality floor with no critical failures and improves cost per successful task by 24.5%." }, "evaluations": [ { "skuId": "anthropic/claude-sonnet-4-6", "name": "Claude Sonnet 4.6", "evaluable": true, "outcome": "switch", "reason": "The candidate clears the quality floor with no critical failures and improves cost per successful task by 24.5%.", "figures": {}, "thresholds": {}, "checks": [], "inputs": { "pricePeriodIds": [101, 102], "enteredPriceIds": [], "evidenceIds": [], "evidenceSources": [], "termIds": [8] } } ], "privacy": "Evaluated in memory with zero retention; inline workload profiles, candidates, and evaluation figures are never stored.", "meta": { "generatedAt": "2026-09-14T18:35:00.000Z", "source": "database", "count": 1 } } ``` --- # Decision pack One accepted or recorded decision as a standalone, signed audit pack: the decision, the exact price periods and evidence it read, the observation leaves with their Merkle hashes, and the canonical digest of the whole, so the pack can be verified offline without the record. ``` GET /api/v1/decisions/{id}/pack ``` ```bash curl -O -J https://vitna.ai/api/v1/decisions/42/pack ``` The answer is served as a file (`Content-Disposition: attachment`), named `vitna-decision-{id}-pack.json`, and is cacheable: a pack is assembled from append-only rows, so the same id answers the same pack. ## Path parameters | Parameter | Meaning | |---|---| | `id` | The decision's id, up to eighteen digits, as `/decisions/{id}` prints it | ## Who can read one Only a decision belonging to an organisation whose feed is public. A decision of an organisation with a private feed answers `404 not_found`, identically to an id that was never issued, so the endpoint cannot be used to enumerate tenants. An id that is not digits answers the same `404` without touching the record. ## What the pack carries - The decision itself, and the provenance absences saying what was not recorded. - The `witness.price_period` rows the decision read, and the observation leaves behind them with their Merkle leaf hashes. - The `aperture.evidence` measurements it read, with their SHA-256 digests. - `digest`: the SHA-256 of the pack's canonical payload, which is the pack with its `signature`, `statement`, `digest` and `signerPublicKey` fields stripped and the rest serialised canonically. ## Errors | Status | Code | Cause | |---|---|---| | 404 | `not_found` | No decision of a public feed has that id: never issued, not digits, or a private feed's | | 503 | `record_unavailable` | This instance has no record, or the record could not be read for this pack | Errors are `no-store`. A `503` carries no pack at all: when a read of the record fails, nothing is assembled in its place. The same pack, signed by the authority's Ed25519 key, is what `npm run aperture:decision:pack` writes, and `--verify` checks a written pack against a public key with no network and no database. The concept page, [Decisions](../../../concepts/decision.md), says what a decision is and how one is accepted. --- # Model Context Protocol (MCP) Keyless Streamable HTTP MCP endpoint (`2025-03-26` and `2024-11-05`) exposing ten read-only and zero-retention tools over Vitna's first-party Witness market record, provider compliance terms, three-provenance SKU capabilities, cryptographic citations, stack decision engine, rollout watchdog, compound pipeline analyzer, and offline audit pack verifier. ## Server discovery ``` GET /api/v1/mcp ``` Returns the server descriptor and tool schemas as cacheable JSON. ```json { "serverInfo": { "name": "vitna-mcp", "version": "1.0.0" }, "protocolVersion": "2025-03-26", "transport": "streamable-http", "endpoint": "/api/v1/mcp", "tools": [ { "name": "vitna_check_quote", "description": "Evaluate a vendor AI pricing quote in memory against Witness list prices and peer spreads with zero retention.", "annotations": { "readOnlyHint": true, "openWorldHint": false }, "inputSchema": { ... } } ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "count": 10 } } ``` ## JSON-RPC 2.0 Streamable HTTP calls ``` POST /api/v1/mcp ``` Accepts standard MCP JSON-RPC 2.0 methods (`initialize`, `notifications/initialized`, `ping`, `tools/list`, and `tools/call`) with zero authentication required. Available tools: - `vitna_check_quote` - evaluate a vendor quote against Witness list prices and peer spreads - `vitna_get_market_tape` - read current Witness list prices by component class and provider - `vitna_get_sku_history` - inspect append-only price period history for a SKU - `vitna_get_provider_terms` - read curated provider retention, training, region, and BYOK terms - `vitna_cite_observation` - resolve an observation ID into its SHA-256 evidence digest and copy-ready citation strings - `vitna_get_sku_capabilities` - inspect a Witness SKU's three-provenance capability and runtime speed profile - `vitna_decide_stack` - run a stateless, zero-retention stack migration evaluation - `vitna_check_rollout` - evaluate standing-decision drift, shadow rule promotion readiness, and routed canary health with zero retention - `vitna_analyze_pipeline` - compute a multi-stage, multi-class AI pipeline Bill of Materials, trace fan-out attribution, and spend optimization with zero retention - `vitna_verify_audit_pack` - verify a `vitna.audit-pack.v1` procurement, FinOps, and governance attestation pack across all six layers or assemble one inline with zero retention ```bash curl -X POST https://vitna.ai/api/v1/mcp \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "vitna_check_quote", "arguments": { "sku": "gpt-5.4", "quotedAmountUsd": 2.10, "dimension": "input_per_mtok" } } }' ``` --- # Headless artifact signing requests Queue an asynchronous request for the isolated signer to rebuild and Ed25519-sign your organisation's standing routing policy or environment config, and poll until the artifact settles. ``` GET /api/v1/org/artifact-requests POST /api/v1/org/artifact-requests GET /api/v1/org/artifact-requests/{id} ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -X POST https://vitna.ai/api/v1/org/artifact-requests \ -H "authorization: Bearer vitna_push_..." \ -H "content-type: application/json" \ -d '{ "kind": "policy", "decisionId": 18, "requestedBy": "github-actions/deploy-routing" }' curl -H "authorization: Bearer vitna_push_..." \ https://vitna.ai/api/v1/org/artifact-requests/7 ``` ## Ask-only security boundary Reads (`GET`) accept either a `platform` or `push` key. Queueing an ask (`POST`) requires a `platform` key (`vitna_key_...`) and answers `202` when accepted onto `aperture.artifact_request`. A signing request names only `kind` (`policy` or `config`) and an optional `decisionId` ("as of this accepted decision"). It never accepts document bytes: supplying `body`, `bytes`, `document`, `policy`, `config` or `signature` answers `400 bytes_forbidden`, so the isolated `vitna_signer` daemon rebuilds every signed artifact from accepted decisions in the database rather than acting as a signing oracle. At most one `pending` request sits on the queue per organisation at a time (`409 already_pending` returns the existing `pollUrl`). Once `state` moves to `signed`, fetch the canonical signed bytes from `artifact.body` (`/api/v1/artifacts/{id}`). ## Response shape ```json { "orgId": "convexity", "request": { "id": 7, "orgId": "convexity", "kind": "policy", "decisionId": 18, "requestedBy": "api-key:4 (github-actions/deploy-routing)", "state": "signed", "createdAt": "2026-09-14T18:30:00.000Z", "settledAt": "2026-09-14T18:30:04.000Z", "pollUrl": "/api/v1/org/artifact-requests/7", "artifact": { "id": 12, "kind": "policy", "outcome": "signed", "at": "2026-09-14T18:30:04.000Z", "digest": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "decisions": [18], "publicKey": "MCowBQYDK2VwAyEA...", "refusedBecause": null, "body": "/api/v1/artifacts/12" } }, "meta": { "generatedAt": "2026-09-14T18:30:05.000Z" } } ``` --- # Organisation audit attestation pack Export a read-only `vitna.audit-pack.v1` procurement, FinOps, and model-governance attestation pack across an authenticated organisation's registered workloads, standing decisions, signed policy, and attested Witness prices. ``` GET /api/v1/org/audit-pack ``` ```bash curl "https://vitna.ai/api/v1/org/audit-pack?monthlyRuns=10000&discoverPeers=true&download=1" \ -H "Authorization: Bearer vitna_key_..." ``` Accepts either an organisation platform key (`vitna_key_...`) or push key (`vitna_push_...`). Queries `aperture.*` and `witness.*` through the read-only site role (`vitna_site`) and mutates nothing. Like [`GET /api/v1/decisions/{id}/pack`](decision-pack.md), this route serves an unsigned pack stamped with its canonical SHA-256 `digest`; operators who sign packs with their own Ed25519 key run [`npm run aperture:audit:pack -- --sign`](../../../cli/aperture/audit-pack.md). See [Audit and procurement attestation packs](../../../routing/audit-packs.md) for the six-layer offline verification contract. ## Query parameters | Parameter | Meaning | |---|---| | `stages` | Optional comma-separated `[:][:bg]` stage list for the compound pipeline BOM | | `monthlyRuns` | Optional positive integer monthly pipeline execution volume (alias `runs`) | | `discoverPeers` | When `true`, include active Witness class-peer exploratory spend levers (`false` by default) | | `includePolicy` | Include the organisation's latest signed `aperture.artifact` policy envelope (`true` by default) | | `window` | Optional `YYYY-MM-DD..YYYY-MM-DD` UTC date filter for Witness observation Merkle proofs | | `label` | Optional human-readable release or procurement review label | | `download` | When `1` or `true`, set `Content-Disposition: attachment; filename="vitna--audit-pack.json"` | ## Response shape ```json { "schema": "vitna-audit-pack-v1", "format": "vitna.audit-pack.v1", "orgId": "convexity", "orgName": "Convexity", "label": "q4-procurement-review", "generatedAt": "2026-11-06T00:00:00.000Z", "sentence": "Audit pack for convexity covers 3 active workloads, 2 standing decisions, 4 attested Witness observations, estimated spend $940.00/mo, rollout status healthy.", "summary": { "workloadCount": 3, "standingDecisionCount": 2, "pricePeriodCount": 4, "attestedObservationCount": 4, "unattestedObservationCount": 0, "enteredPriceCount": 0, "providerTermCount": 4, "capabilitySkuCount": 3, "evidenceCount": 2, "pipelineCount": 1, "totalEstimatedMonthlySpendUsd": 940, "verifiedMonthlySavingsUsd": 0, "watchdogVerdict": "healthy", "policyIncluded": true, "lockIncluded": false }, "claims": { ... }, "governance": { "stackfile": { ... }, "stackfileDigest": "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b", "standingDecisions": [ ... ], "policyEnvelope": { ... }, "policyLock": null }, "pricingAndProvenance": { "witnessPricePeriods": [ ... ], "observationProofs": [ ... ], "enteredPrivatePrices": [] }, "complianceAndCapabilities": { "providerTerms": [ ... ], "capabilityReadings": [ ... ], "admissibilityRules": { ... }, "customerEvidence": [ ... ] }, "operations": { "rolloutWatchdog": { ... }, "compoundPipelines": [ ... ] }, "digest": "4f3b9a8e01c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8", "meta": { "generatedAt": "2026-11-06T00:00:00.000Z", "workloadCount": 3, "attestedObservationCount": 4 } } ``` --- # Workload counterfactual cost Compute what a recorded usage window of one of your organisation's workloads would have cost at historical Witness list prices had it run on another SKU. ``` GET /api/v1/org/workloads/{slug}/counterfactual ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." \ "https://vitna.ai/api/v1/org/workloads/deep-research/counterfactual?window=2026-08-01..2026-08-31&candidate=anthropic/claude-sonnet-4-6" ``` ## Bounds and caveats Accepts either a `platform` or `push` key. Requires `window` (`YYYY-MM-DD..YYYY-MM-DD`, or `from` and `to`) and `candidate` (SKU ID or slug in the same component class). When a price moved inside the window and the recorded usage window carries no daily split, both `actual` and `would` are reported as exact `[minUsd, maxUsd]` bounds rather than point estimates, and `caveats` states every coverage boundary. No candidate requests are run and no unmeasured quality figure is claimed. ## Response shape ```json { "orgId": "convexity", "workload": { "id": "convexity/deep-research", "slug": "deep-research", "name": "Deep research", "classId": "model", "unitLabel": "runs" }, "candidate": { "id": "anthropic/claude-sonnet-4-6", "name": "Claude Sonnet 4.6", "classId": "model" }, "requestedWindow": { "start": "2026-08-01", "end": "2026-08-31" }, "counterfactual": { "ok": true, "servedUnits": 50000, "actual": { "minUsd": 420, "maxUsd": 420, "exact": true }, "would": { "minUsd": 290, "maxUsd": 290, "exact": true }, "candidateSkuId": "anthropic/claude-sonnet-4-6", "difference": { "minUsd": -130, "maxUsd": -130, "exact": true }, "moves": 0, "caveats": [] }, "sentence": "50,000 units over 2026-08-01 to 2026-08-31 cost $420.00 at list, and $290.00 on anthropic/claude-sonnet-4-6: $130.00 less.", "meta": { "generatedAt": "2026-09-14T18:40:00.000Z" } } ``` --- # CI evaluation evidence Record evaluation measurements from CI pipelines and read back historical measurements for your organisation's workloads, with built-in normalization for Promptfoo, Braintrust and UK AISI Inspect AI outputs. ``` GET /api/v1/org/evidence POST /api/v1/org/evidence ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." \ "https://vitna.ai/api/v1/org/evidence?workload=deep-research&kind=eval" curl -X POST https://vitna.ai/api/v1/org/evidence \ -H "authorization: Bearer vitna_push_..." \ -H "content-type: application/json" \ -d '{ "adapter": "promptfoo", "workload": "deep-research", "sku": "openai/gpt-5.6-luna", "label": "pr-412-gpt-5.6-luna", "sourceRef": "https://github.com/convexity/app/actions/runs/91823", "report": { "timestamp": "2026-09-14T18:22:00Z", "results": { "stats": { "successes": 193, "failures": 7, "errors": 0 } } } }' ``` ## Adapters and append-only rules Reads (`GET`) accept either a `platform` or `push` key and filter by `workload`, `sku`, `kind` (`eval`, `production` or `market`) and `limit` (`1` to `200`, default `50`). Writes (`POST`) require a `platform` key (`vitna_key_...`). Supported `adapter` values: - `vitna` (default): native JSON carrying `workload`, `sku` (or `skuLabel`), `label`, `measuredAt` (`YYYY-MM-DD`), `n`, `sourceRef` and `metrics` (`successRatePct`, `latencyP90Ms`, `edgeCaseFailures`). - `promptfoo`: normalizes `npx promptfoo eval -o output.json` rows or `stats` into `n`, `successRatePct`, sorted p90 latency (`latencyP90Ms`) and failed test case identifiers (`edgeCaseFailures`). - `braintrust`: normalizes Braintrust `Eval()` experiment summaries or row dumps (`scoreKey` selects a named scorer when an experiment tracks several). - `inspect`: normalizes UK AISI `inspect_ai` JSON logs (`inspect eval --log-format json`), extracting sample count, accuracy, p90 working time in milliseconds and failed sample IDs. Evidence is append-only and unique on `(workload_id, label)`. Re-submitting an existing label answers `409 duplicate_evidence` rather than overwriting the measurement a prior decision cited. ## Response shape ```json { "orgId": "convexity", "supportedAdapters": ["vitna", "promptfoo", "braintrust", "inspect"], "measurements": [ { "id": 42, "workloadId": "convexity/deep-research", "skuId": "openai/gpt-5.6-luna", "isPrivateSku": false, "skuLabel": null, "label": "pr-412-gpt-5.6-luna", "kind": "eval", "measuredAt": "2026-09-14", "n": 200, "metrics": { "successRatePct": 96.5, "latencyP90Ms": 1420 }, "sourceRef": "https://github.com/convexity/app/actions/runs/91823", "notes": null, "recordedAt": "2026-09-14T18:22:10.000Z" } ], "meta": { "generatedAt": "2026-09-14T18:22:10.000Z", "count": 1 } } ``` --- # Organisation dependency and deprecation exposure Read your organisation's live workload-to-SKU-to-provider dependency graph, provider blast-radius concentration, stale standing decisions and upcoming model deprecations as structured JSON. ``` GET /api/v1/org/exposure ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." \ https://vitna.ai/api/v1/org/exposure ``` ## What is included Accepts either a `platform` or `push` key. Archived workloads are excluded by default; pass `?includeArchived=true` to include retired lanes in the graph. - `exposure` and `sentence`: total active workloads, distinct SKUs, distinct providers, single-provider workloads with no cross-provider fallback, and workloads with no standing decision. - `providers`: per-provider blast-radius breakdown showing how many workloads reach each provider as primary, fallback or conditional traffic-split rule. - `staleDecisions`: standing decisions aged past review thresholds or challenged by newer market launches. - `upcomingDeprecations`: armed or spent deprecation and date watches affecting your workloads. - `graph`: full node and directed-edge topology (`serves`, `fallback`, `rule`, `provider`). ## Response shape ```json { "orgId": "convexity", "orgName": "Convexity", "includeArchived": false, "exposure": { "workloads": 2, "skus": [ { "id": "openai/gpt-5.6-luna", "label": "GPT-5.6 Luna", "private": false, "serves": 2, "reaches": 2 } ], "providers": [ { "id": "openai", "label": "openai", "private": false, "serves": 2, "reaches": 2 } ] }, "sentence": "openai is reachable by 2 of 2 workloads, 100 percent, and 2 of them run it now. A deprecation there is the largest single market event this organisation is exposed to.", "providers": [ { "providerId": "openai", "label": "openai", "skuCount": 2, "skus": ["openai/gpt-5.6-luna", "openai/gpt-5.6-mini"], "workloadCount": 2, "workloads": ["convexity/deep-research", "convexity/support-triage"], "primaryWorkloadCount": 2, "fallbackWorkloadCount": 0, "ruleWorkloadCount": 0 } ], "staleDecisions": { "staleCount": 0, "totalDecisions": 2, "decisions": [], "sentence": "Every standing decision has an armed watch and no unresolved market challenges." }, "upcomingDeprecations": { "count": 0, "deprecations": [], "sentence": "No upcoming deprecation reviews are scheduled on armed date triggers." }, "graph": { "columns": ["workload", "primary", "reachable", "provider"], "edgeLabels": {}, "rows": 2, "nodes": [], "edges": [] }, "meta": { "generatedAt": "2026-09-14T18:50:00.000Z", "workloadCount": 2, "providerCount": 2 } } ``` --- # Your organisation's feed The triggers that fired on your organisation's workloads and the decisions the engine proposed for them, newest first, as a feed a reader or a chat channel can subscribe to. It is private: the address is the permission, and an owner of the organisation gets it from the console. ``` GET /api/v1/feeds/{capability}/feed.json GET /api/v1/feeds/{capability}/rss.xml ``` ```bash curl https://vitna.ai/api/v1/feeds/your-org.NOT-A-REAL-CAPABILITY/rss.xml ``` That address is not a real one, and answers the `404` described below. A real `{capability}` is your organisation's id, a dot, and 43 characters. ## Getting the address An owner of the organisation opens `/console/settings`, finds the Feed panel, and presses "Issue a feed address" the first time. After that the panel shows both addresses behind "Show the feed address". - **RSS** (`rss.xml`) is the one to paste into a feed reader, or into Slack's RSS app if your workspace has it installed. - **JSON Feed** (`feed.json`) is the same items for a program. Members of the organisation are not shown the address. A member can read everything the feed carries in the console already; what the address adds is reading it without signing in, and it keeps working after somebody's membership ends. So it stays with the people who can rotate it, and a member asks an owner. ## What a capability URL is The address is a credential in a URL. Be plain with yourself about what that means: - **Anyone holding it reads the feed**, without signing in and without anything else. There is no second factor. - **Anything that sees the path can log it.** Your feed reader's servers, a chat app's servers, a proxy between you and this site, Cloudflare in front of it, and the web server on the box all see the full path of every request. Vitna's own process writes the address to no log, and that is the only part of the path it controls. - **If it has been somewhere it should not have been, rotate it.** A pasted screenshot, a public channel, a shared feed reader account: rotate. It is in the path rather than in a header because the things this feed is for cannot send a header. A feed reader and Slack's RSS app take an address and nothing else. ## Rotating An owner presses "Rotate the address" in the same panel. The old address stops working at that moment, with no grace period, and the panel shows the new one. Every reader subscribed to the old address gets the `404` below on its next poll and needs the new address. Rotating is the only way to end an address. There is no switch that turns a feed off and leaves the address in place. ## How the address is made It is not stored. The record keeps one number per organisation, which starts empty and goes up by one each time an owner issues or rotates an address, and the capability is an HMAC-SHA256 of the organisation id and that number under a key the site holds outside the database. So a database backup carries no address, and there is no table of addresses for a bug to read out. The trade that comes with it: that key is one secret for every organisation's feed. If it ever leaks, it is replaced, and replacing it changes every address at once, so every owner hands out a new one. ## Response The JSON Feed rendering. Items are shaped exactly as they are on the [public feed](../log/feed.md), and an item that appears on both carries the same `id` on both, so a reader following both sees it once. ```json { "version": "https://jsonfeed.org/version/1.1", "title": "Vitna: acme", "home_page_url": "https://vitna.ai/console", "feed_url": "https://vitna.ai/api/v1/feeds/acme.NOT-A-REAL-CAPABILITY/feed.json", "description": "The triggers that fired on acme's workloads and the decisions the engine proposed for them, newest first. ...", "language": "en", "items": [ { "id": "https://vitna.ai/api/v1/feed.json#decision-41", "url": "https://vitna.ai/console/workloads/acme/invoice-extraction", "title": "A cheaper candidate clears the threshold", "content_text": "The candidate is 31 percent cheaper per successful unit. Nobody has ruled on this yet. ...", "date_published": "2026-09-12T08:00:00.000Z", "tags": ["decision", "switch", "proposed"], "_vitna": { "kind": "proposal", "decisionId": 41, "workload": "acme/invoice-extraction", "outcome": "switch", "status": "proposed", "candidate": "openai/gpt-5.6-luna", "firedByTriggerId": 17 } } ], "_vitna": { "org": "acme", "source": "database", "engineActivity": "included" }, "meta": { "generatedAt": "2026-09-13T09:14:02.881Z", "source": "database", "count": 2 } } ``` The example shows the proposal. The trigger that wrote it is the second item, and its `_vitna` carries a different set of keys: | `_vitna.kind` | Other keys in `_vitna` | |---|---| | `proposal` | `decisionId`, `workload`, `outcome`, `status`, `candidate`, `firedByTriggerId` | | `trigger_fired` | `triggerId`, `triggerKind`, `workload`, `eventId` | Served as `application/feed+json; charset=utf-8`. The RSS rendering is `application/rss+xml; charset=utf-8`, carries the same items, and uses each item's `id` as its `guid`. Fifty items at most, newest first. Only your organisation's workloads are read, whether or not its register is published on the public feed. Market events are not in this feed; the [public feed](../log/feed.md) carries those. Two differences from the public feed's items, both about where a link goes: - A proposal's `url` is its workload's page in the console, not `/decisions/{id}`. A decision of a private register has no public page, and a subscriber who is a member can open the console one. - A trigger carries a `url` too, to the same page. On the public feed it has none, because no public page shows a trigger. ## Refusals Every refusal is the v1 error shape, in both renderings, because the thing asking is a program: ``` { "error": { "code": "unknown_feed", "message": "..." }, "meta": { ... } } ``` | Status | `error.code` | When | |---|---|---| | `404` | `unknown_feed` | The address is not a live one | | `503` | `record_unavailable` | This instance cannot read the register | | `503` | `not_configured` | This instance holds no key to check an address with, so it serves no private feed | **One `404` for every wrong address.** Malformed, an organisation that does not exist, an organisation that never issued an address, a wrong capability and a rotated one all get the same status, code and message. Which of those it was would be information about somebody else's organisation. A malformed address is refused before the database is asked anything, and a capability with any of its characters percent-encoded counts as malformed: every character an address holds is safe in a path unencoded, so it has one spelling. **`503`, never an empty feed.** A feed saying nothing fired, from an instance that could not look, would be the lie this record exists to prevent. ## Caching and CORS Every answer, refusals included, carries `cache-control: private, no-store`, so no shared cache and no browser keeps an organisation's activity. The JSON Feed rendering carries an `etag` and still answers `304` to an `If-None-Match` that matches it, for a reader that keeps the tag itself. There is no CORS on this route, unlike every keyless read in v1. A page on another origin cannot read it, and its preflight is answered without an allowed origin. --- # Organisation API keys List active and revoked organisation API keys without exposing key hashes, and soft-revoke compromised or retired keys with an audit reason. ``` GET /api/v1/org/keys POST /api/v1/org/keys POST /api/v1/org/keys/{id}/revoke ``` ``` Authorization: Bearer vitna_key_... ``` ```bash curl -H "authorization: Bearer vitna_key_..." https://vitna.ai/api/v1/org/keys curl -X POST https://vitna.ai/api/v1/org/keys/9/revoke \ -H "authorization: Bearer vitna_key_..." \ -H "content-type: application/json" \ -d '{"reason": "Rotated after CI runner migration"}' ``` ## Scopes and privilege isolation Each key carries one immutable `scope`: - `push` (`vitna_push_...`) submits usage ledgers (`POST /api/v1/usage`), collects signed artifacts (`GET /api/v1/artifacts`) and reads control-plane state (`GET /api/v1/org/*`). - `router` (`vitna_push_...`) calls the hosted router and holds zero control-plane access. - `platform` (`vitna_key_...`) calls Vitna's hosted APIs and reads and writes the organisation control plane (`GET` and `POST /api/v1/org/*`). No web-facing database role holds `INSERT` on `site.api_key` (`drizzle/0043`), so `POST /api/v1/org/keys` answers `403 passkey_ceremony_required`. Owners mint `platform` keys (`vitna_key_...`) directly in `/console/settings` by confirming a request with an approved WebAuthn passkey (settled by `vitna_minter`, sealed to the browser's ephemeral key), or an operator mints a key at the prompt: ```bash npm run org:key:create -- --org convexity --scope platform --label "control-plane, owner" ``` `vitna_auth` holds column-level `SELECT` on `site.api_key` excluding `token_sha256`, so no query on this surface can read a key hash back after minting. Revoking a key (`POST /api/v1/org/keys/{id}/revoke`) sets `revoked_at` and `revoked_reason` without deleting the audit row, and a database trigger prevents reviving a revoked row. ## Listing response ```json { "data": [ { "id": 9, "orgId": "convexity", "label": "ci-policy-sync", "scope": "platform", "status": "active", "createdAt": "2026-09-14T12:00:00.000Z", "revokedAt": null, "revokedReason": null } ], "meta": { "generatedAt": "2026-09-14T12:00:00.000Z", "orgId": "convexity", "count": 1, "activeCount": 1 } } ``` --- # Organisation members List active organisation members, add or re-admit GitHub accounts on any plan, and soft-remove members with last-owner protection. ``` GET /api/v1/org/members POST /api/v1/org/members POST /api/v1/org/members/remove ``` ``` Authorization: Bearer vitna_key_... ``` ```bash curl -H "authorization: Bearer vitna_key_..." https://vitna.ai/api/v1/org/members curl -X POST https://vitna.ai/api/v1/org/members \ -H "authorization: Bearer vitna_key_..." \ -H "content-type: application/json" \ -d '{"github": "octocat", "role": "member"}' ``` ## Unlimited seats and last-owner protection Every Vitna plan admits as many members as you like (`seatAllowance: null`): Vitna counts workloads, never seats. Reads (`GET`) accept either a `platform` or `push` key; writes (`POST`) require a `platform` key (`vitna_key_...`). The invited GitHub handle must have signed in at `/auth/github/start` at least once so `site.user` holds a verified account id (`404 user_not_signed_in` otherwise). Removing a member via `POST /api/v1/org/members/remove` dates `removed_at` and `removed_reason` rather than deleting the row, and refuses to remove the sole active `owner` (`409 last_owner`) so an organisation can never orphan itself. ## Listing response ```json { "data": [ { "orgId": "convexity", "userId": 1, "login": "garrett", "githubId": 1001, "name": "Garrett", "role": "owner", "addedAt": "2026-09-14T12:00:00.000Z" } ], "meta": { "generatedAt": "2026-09-14T12:00:00.000Z", "orgId": "convexity", "count": 1, "seatAllowance": null } } ``` --- # Organisation compound pipeline Inspect the cross-layer Bill of Materials, bottlenecks, decision-ready switches, and eligible class-peer exploratory levers across an authenticated organisation's registered workloads. ``` GET /api/v1/org/pipeline ``` ```bash curl "https://vitna.ai/api/v1/org/pipeline?stages=web-search:6,doc-parse:10,deep-research:1&monthlyRuns=10000&discoverPeers=true" \ -H "Authorization: Bearer vitna_key_..." ``` Accepts either an organisation platform key (`vitna_key_...`) or push key (`vitna_push_...`). Queries `aperture.*` and `witness.*` through the read-only site role (`vitna_site`) and mutates nothing: every migration or peer evaluation recommendation emits the exact [`aperture:*`](../../../cli/aperture/pipeline.md) operator command. See [Multi-class unit profiles and compound pipelines](../../../routing/pipelines.md) for how stage costs, provenance splits, and optimization tiers are computed. ## Query parameters | Parameter | Meaning | |---|---| | `stages` | Optional comma-separated `[:][:bg]` list; defaults to all active workloads in the organisation | | `monthlyRuns` | Optional positive integer monthly pipeline run count (alias `runs`) | | `runLabel` | Optional plural unit label for one run (default `runs`) | | `name` | Optional human-readable pipeline name | | `discoverPeers` | When `true`, scan active Witness SKUs in each stage's component class for eligible lower-cost exploratory levers | ## Response shape ```json { "orgId": "convexity", "orgName": "Convexity", "pipeline": { "id": "convexity/pipeline", "name": "Convexity compound pipeline", "orgId": "convexity", "runLabel": "runs", "monthlyRuns": 10000, "stageCount": 3, "criticalPathStageCount": 3 }, "totals": { "costPerRunUsd": 0.094, "monthlyCostUsd": 940, "annualCostUsd": 11280, "pricedStageCount": 3, "unpricedStageCount": 0, "measuredQualityStageCount": 3, "compoundSuccessRatePct": 91.2, "strictAllCallsSuccessRatePct": 84.5, "costPerSuccessfulRunUsd": 0.10307, "criticalPathLatencyP90Ms": 2840, "latencyComplete": true }, "breakdown": { ... }, "bottlenecks": { ... }, "optimization": { ... }, "stages": [ ... ], "runAttribution": null, "caveats": [ ... ], "recommendedCommands": [ ... ], "sentence": "Pipeline Convexity compound pipeline costs $0.094000 per run ($940.00/mo across 10,000 runs) across 3 priced stages; ...", "meta": { "generatedAt": "2026-11-06T00:00:00.000Z", "stageCount": 3, "discoverPeers": true } } ``` --- # Organisation Stackfile Export your organisation's live register as a canonical `vitna.stackfile.v1` JSON document, preview a deterministic reconciliation plan (`dryRun: true`), or apply declarative register updates over HTTP. ``` GET /api/v1/org/stackfile POST /api/v1/org/stackfile ``` ``` Authorization: Bearer vitna_key_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." https://vitna.ai/api/v1/org/stackfile curl -X POST "https://vitna.ai/api/v1/org/stackfile?dryRun=1" \ -H "authorization: Bearer vitna_key_..." \ -H "content-type: application/json" \ -d '{ "stackfile": { "schema": "vitna.stackfile.v1", "org": "convexity", "privateSkus": [], "workloads": [ { "slug": "support-triage", "name": "Support triage", "classId": "model", "status": "active", "priority": "economics", "unitLabel": "tickets", "incumbentSkuId": "openai/gpt-5.6-luna", "monthlyVolume": 50000, "unitProfile": { "inputTokensPerUnit": 1800, "outputTokensPerUnit": 400 }, "constraints": { "qualityFloor": 90, "allowsProviders": ["openai", "anthropic"] }, "delivery": { "policyAlias": "support-triage" } } ] }, "dryRun": true }' ``` ## Scopes and reconciliation rules - `GET /api/v1/org/stackfile` accepts either a `platform` (`vitna_key_...`) or `push` (`vitna_push_...`) bearer credential and returns the canonical [Stackfile](../../../routing/stackfile.md) alongside its SHA-256 `digest`. - `POST /api/v1/org/stackfile` requires a `platform` key (`router` and `push` keys answer `403 forbidden_scope`). - Pass `"dryRun": true` (or `?dryRun=1`) to validate the document and compute the exact action plan (`workload.create`, `workload.edit`, `workload.status`, `private.add`, `private.price`, `candidate.add`, `fallback.remove`, `fallback.add`, `rule.add`, `watch.arm`) without mutating the database. - Every action executes through `registerStore`, enforcing metered workload allowances (`402 plan_allowance_exceeded`), archived-workload immutability (`409 workload_archived`), and org scope (`403 org_mismatch`). Workloads omitted from the payload are never deleted; they are reported in `plan.unmanaged.workloads`. ## Export response (`GET`) ```json { "stackfile": { "schema": "vitna.stackfile.v1", "org": "convexity", "privateSkus": [], "workloads": [ { "slug": "support-triage", "name": "Support triage", "classId": "model", "status": "active", "priority": "economics", "unitLabel": "tickets", "incumbentSkuId": "openai/gpt-5.6-luna", "monthlyVolume": 50000, "unitProfile": { "inputTokensPerUnit": 1800, "outputTokensPerUnit": 400 }, "constraints": { "qualityFloor": 90, "allowsProviders": ["openai", "anthropic"] }, "delivery": { "policyAlias": "support-triage" }, "notes": null, "candidates": [], "fallbacks": [], "rules": [], "watches": [] } ] }, "digest": "6f3e4b2c1a9d8e7f0123456789abcdef0123456789abcdef0123456789abcdef", "meta": { "generatedAt": "2026-09-14T12:00:00.000Z", "orgId": "convexity", "workloadCount": 1, "privateSkuCount": 0 } } ``` --- # Organisation rollout watchdog Inspect standing-decision drift, shadow rule promotion readiness, and routed canary health across an authenticated organisation's active workloads. ``` GET /api/v1/org/watchdog ``` ```bash curl https://vitna.ai/api/v1/org/watchdog \ -H "Authorization: Bearer vitna_key_..." ``` Accepts either an organisation platform key (`vitna_key_...`) or push key (`vitna_push_...`). Reads `aperture.*` and `witness.*` through the read-only site role (`vitna_site`) and mutates nothing: every promotion, rung expansion, or rollback recommendation emits the exact [`aperture:*`](../../../cli/aperture/watchdog.md) operator command to execute. See [Rollout and decision watchdog](../../../routing/watchdog.md) for all workload and rule verdicts. ## Query parameters | Parameter | Meaning | |---|---| | `workload` | Optional workload slug (`deep-research`) or qualified id (`convexity/deep-research`) | | `window` | Optional closed UTC date range (`YYYY-MM-DD..YYYY-MM-DD`); defaults to each workload's newest `deployment_observation` window | | `minShadowMatches` | Minimum shadow matches before `promote_ready` (default `50`) | | `minCanaryUnits` | Minimum served canary units before `expand_or_adopt_ready` (default `100`) | | `maxUnservedRatePct` | Maximum unserved attempt rate percentage (default `5`) | | `maxFallbackSharePct` | Maximum fallback share of served units percentage (default `10`) | | `maxPriceDriftPct` | Maximum unit-cost increase vs standing decision percentage (default `10`) | ## Response shape ```json { "orgId": "convexity", "orgName": "Convexity", "summary": { "workloads": 2, "healthy": 1, "unruled": 0, "attention": 1, "decisionDrifted": 0, "rollbackRecommended": 0, "promoteReadyRules": 0, "expandReadyRules": 1, "rollbackRules": 0 }, "thresholds": { "minShadowMatches": 50, "minCanaryUnits": 100, "maxUnservedRatePct": 5, "maxFallbackSharePct": 10, "maxPriceDriftPct": 10 }, "sentence": "convexity: all standing decisions hold; 0 shadow rules are ready to promote and 1 canary is ready to advance.", "recommendedCommands": [ "npm run aperture:rule:add -- --workload convexity/deep-research --kind traffic_share --share 25 --sku anthropic/claude-sonnet-4-6" ], "workloads": [ ... ], "meta": { "generatedAt": "2026-11-05T12:00:00.000Z", "workloadCount": 2, "window": "latest" } } ``` --- # Policy drift webhooks Register, test and soft-disable HMAC-SHA256 signed webhook endpoints for policy drift, newly signed policy bundles and fired price watches on Deploy and agreement organisations. ``` GET /api/v1/org/webhooks POST /api/v1/org/webhooks POST /api/v1/org/webhooks/{id}/test POST /api/v1/org/webhooks/{id}/disable ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -X POST https://vitna.ai/api/v1/org/webhooks \ -H "authorization: Bearer vitna_push_..." \ -H "content-type: application/json" \ -d '{ "url": "https://ops.example.com/webhooks/vitna", "events": ["policy.drift_detected", "policy.bundle_signed"], "description": "Production CI drift receiver" }' ``` ## Entitlement and SSRF safety Registering and testing webhooks require Deploy (`$79` a month) or recorded agreement terms; metered organisations on the free allowance receive `402 plan_upgrade_required` pointing to `/console/billing`. Destination URLs must use `https://` on a public internet hostname. Loopback, localhost, RFC1918 private networks, link-local metadata addresses and `.local`/`.internal` suffixes are refused at registration and before delivery. ## Signature verification At registration `signingSecret` (`whsec_vitna_...`) is returned once and sealed at rest with AES-256-GCM under `VITNA_CONNECTION_KEY`. Subsequent `GET /api/v1/org/webhooks` listings expose only `secretFingerprint` (sha256 hex). Every delivery serialises its JSON body with deterministic lexicographic key ordering and signs the exact UTF-8 wire bytes with HMAC-SHA256 in `x-vitna-signature-256: sha256=`, alongside `x-vitna-event` and `x-vitna-delivery`. ## Registration response ```json { "data": { "id": 4, "orgId": "convexity", "workloadId": null, "url": "https://ops.example.com/webhooks/vitna", "description": "Production CI drift receiver", "events": ["policy.drift_detected", "policy.bundle_signed"], "secretFingerprint": "5a91e8c4b0f1a2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7", "status": "active", "createdAt": "2026-09-14T12:00:00.000Z", "disabledAt": null, "disabledReason": null, "signingSecret": "whsec_vitna_example_shown_once" }, "meta": { "generatedAt": "2026-09-14T12:00:00.000Z", "orgId": "convexity", "signatureHeader": "x-vitna-signature-256", "note": "signingSecret is returned once at registration. Store it in your receiver and verify x-vitna-signature-256 over the raw request body." } } ``` --- # Organisation workloads List, create, edit and transition workloads in your organisation's register over HTTP, including reversible pausing and permanent archiving that free active workload slots without erasing decision history. ``` GET /api/v1/org/workloads POST /api/v1/org/workloads GET /api/v1/org/workloads/{slug} POST /api/v1/org/workloads/{slug} ``` ``` Authorization: Bearer vitna_push_... ``` ```bash curl -H "authorization: Bearer vitna_push_..." https://vitna.ai/api/v1/org/workloads curl -X POST https://vitna.ai/api/v1/org/workloads \ -H "authorization: Bearer vitna_push_..." \ -H "content-type: application/json" \ -d '{ "slug": "support-triage", "name": "Support triage", "classId": "model", "incumbentSkuId": "openai/gpt-5.6-luna", "unitLabel": "ticket", "unitProfile": { "inputTokensPerUnit": 1800, "outputTokensPerUnit": 400 }, "priority": "economics" }' ``` ## Scopes and plan allowance Reads (`GET`) accept either a `platform` or `push` key. Writes (`POST`) require a `platform` key (`vitna_key_...`; `router` keys are refused with `403 forbidden_scope`). Every workload carries a `status` in `active`, `paused` or `archived`: - `active` workloads count against the organisation's metered allowance (`1` on Free, `3` on Deploy, unmetered under an operator agreement). - `paused` workloads stop counting toward the active allowance while retaining all standing decisions, lane rules and measurements. Resuming (`paused` to `active`) checks the active allowance inside the write transaction and answers `402 plan_allowance_exceeded` if all slots are full. - `archived` workloads permanently retire a lane while keeping its audit trail intact (`409 workload_archived` if asked to re-activate). ## Response shape ```json { "data": [ { "id": "convexity/support-triage", "orgId": "convexity", "slug": "support-triage", "name": "Support triage", "classId": "model", "incumbentSkuId": "openai/gpt-5.6-luna", "incumbentIsPrivate": false, "unitLabel": "ticket", "unitProfile": { "inputTokensPerUnit": 1800, "outputTokensPerUnit": 400 }, "monthlyVolume": null, "constraints": {}, "delivery": {}, "priority": "economics", "notes": null, "status": "active", "pausedAt": null, "archivedAt": null, "activeRuleCount": 0, "activeFallbackCount": 0, "decisionCount": 1 } ], "meta": { "generatedAt": "2026-09-14T12:00:00.000Z", "orgId": "convexity", "count": 1, "activeCount": 1, "workloadAllowance": 3 } } ``` --- # Stateless compound pipeline analysis Evaluate a multi-stage, multi-class AI pipeline Bill of Materials, per-run trace attribution, compound reliability, and cross-layer spend optimization in memory with no API key and zero retention. ``` POST /api/v1/pipeline ``` ```bash curl -X POST https://vitna.ai/api/v1/pipeline \ -H "content-type: application/json" \ -d '{ "name": "Deep research report", "runLabel": "reports", "monthlyRuns": 10000, "discoverPeers": true, "stages": [ { "id": "search", "classId": "tooling", "incumbentSkuId": "tavily/search", "invocationsPerRun": 6, "unitProfile": { "toolCallsPerUnit": 1 } }, { "id": "parse", "classId": "extraction", "incumbentSkuId": "mistral/ocr", "invocationsPerRun": 8, "unitProfile": { "ocrPagesPerUnit": 1 } }, { "id": "synthesize", "classId": "model", "incumbentSkuId": "openai/gpt-5.4", "invocationsPerRun": 1, "unitProfile": { "inputTokensPerUnit": 18000, "outputTokensPerUnit": 2200 }, "constraints": { "qualityFloor": 93.0, "economicThresholdPct": 15, "requiresCapabilities": ["tools"] }, "candidates": ["anthropic/claude-sonnet-4-6"] } ], "evidence": [ { "skuId": "openai/gpt-5.4", "measuredAt": "2026-10-22", "metrics": { "successRatePct": 95.0 } }, { "skuId": "anthropic/claude-sonnet-4-6", "measuredAt": "2026-10-22", "metrics": { "successRatePct": 94.5 } } ] }' ``` ## What is evaluated Runs [`evaluatePipeline`](../../../routing/pipelines.md) in memory across any of the 10 Witness component classes (`model`, `compute`, `retrieval`, `speech`, `tooling`, `extraction`, `translation`, `image`, `video`, `music`), plus optional inline `privateRates` for self-hosted or custom contract SKUs, `ledgerRows` (`x-vitna-run` runtime traces), and `evidence`: 1. **Cross-layer Bill of Materials**: computes per-stage and total `costPerRunUsd`, `monthlyCostUsd`, `annualCostUsd`, `breakdown.byClass`, `breakdown.byProvider`, and `breakdown.provenance` (splitting Witness archive prices from inline `privateRates`). 2. **Compound reliability and critical-path latency**: computes `compoundSuccessRatePct` and `costPerSuccessfulRunUsd` when every critical-path stage carries `successRatePct` evidence, and sums `criticalPathLatencyP90Ms`. 3. **Two-tier spend optimization**: separates verified `decisionReadySwitches` (cleared by `decide()`) from `exploratoryTests` (eligible lower-cost list levers awaiting workload evaluation). Payloads are never persisted (`Cache-Control: no-store`). Also exposed to MCP clients as `vitna_analyze_pipeline` on [`POST /api/v1/mcp`](mcp.md). ## Response shape ```json { "pipeline": { "id": "inline/pipeline", "name": "Deep research report", "orgId": null, "runLabel": "reports", "monthlyRuns": 10000, "stageCount": 3, "criticalPathStageCount": 3 }, "totals": { "costPerRunUsd": 0.122, "monthlyCostUsd": 1220, "annualCostUsd": 14640, "pricedStageCount": 3, "unpricedStageCount": 0, "measuredQualityStageCount": 1, "compoundSuccessRatePct": null, "strictAllCallsSuccessRatePct": null, "costPerSuccessfulRunUsd": null, "criticalPathLatencyP90Ms": null, "latencyComplete": false }, "breakdown": { "byClass": [ ... ], "byProvider": [ ... ], "provenance": { "witnessedRunCostUsd": 0.122, "enteredRunCostUsd": 0, "witnessedSharePct": 100, "enteredSharePct": 0, "mixedProvenance": false, "pricePeriodIds": [ ... ], "enteredPriceIds": [] } }, "bottlenecks": { ... }, "optimization": { "currentCostPerRunUsd": 0.122, "optimizedCostPerRunUsd": 0.122, "verifiedRunSavingsUsd": 0, "verifiedMonthlySavingsUsd": 0, "verifiedAnnualSavingsUsd": 0, "verifiedSavingsPct": 0, "exploratoryAdditionalRunSavingsUsd": 0.036, "exploratoryAdditionalMonthlySavingsUsd": 360, "decisionReadySwitches": [], "exploratoryTests": [ ... ] }, "stages": [ ... ], "runAttribution": null, "caveats": [ ... ], "recommendedCommands": [ ... ], "sentence": "Pipeline Deep research report costs $0.122000 per report ($1220.00/mo across 10,000 reports) across 3 priced stages; ...", "privacy": "Evaluated in memory with zero retention; inline pipeline stages, private rates, ledger rows, and evaluation figures are never stored.", "meta": { "generatedAt": "2026-11-06T00:00:00.000Z", "source": "database", "stageCount": 3, "discoverPeers": true } } ``` --- # Proposals The decision register: what the engine proposed, what a person ruled, and the exact rows each brief was derived from. ``` GET /api/v1/proposals ``` ```bash curl "https://vitna.ai/api/v1/proposals?status=proposed" curl "https://vitna.ai/api/v1/proposals?workload=convexity/earnings-lens" ``` ## Query parameters | Parameter | Values | |---|---| | `workload` | A workload id | | `status` | `proposed`, `accepted`, `superseded` | ## Response ```json { "proposals": [ { "id": 7, "workloadId": "...", "title": "...", "reason": "...", "outcome": "...", "status": "proposed", "decidedAt": "...", "decidedBy": null, "candidateSkuId": "...", "firedByTriggerId": 3, "inputs": { "pricePeriodIds": [ ... ], "evidenceIds": [ ... ] } } ], "meta": { "generatedAt": "...", "count": 12, "awaitingReview": 2 } } ``` `meta.awaitingReview` counts proposals nobody has ruled on, which is the number in this record that is waiting for a person. ## Human decisions are here too Not only the engine's. A register showing the machine half would misreport what a workload is actually doing, because human decisions are what it is doing. `status` and `decidedBy` separate a proposal nobody has ruled on from the one that stands. ## `inputs` is published deliberately It names the exact `witness.price_period` and `aperture.evidence` ids the run read. Every one of them is append-only, so a brief can be regenerated and checked at any distance. Publishing the verdict without them would be publishing an opinion. ## Nothing here writes Accepting or refusing moves `status`, and that is a person's act: at a prompt as the owning role, or at `/console` as `vitna_console`. This is enforced below the application, and the enforcement is about the role this endpoint runs as rather than about the schema being frozen. Two roles hold a column-level `UPDATE` in `aperture`: `vitna_collector` marks a trigger fired, and `vitna_console` records a ruling on a decision. **This API runs as neither.** It reads through `vitna_site`, which holds `SELECT` and nothing else anywhere, so no write endpoint could be added here without a migration granting this role a privilege it has never had. See [the decision concept](../../../concepts/decision.md) for the two grants column by column. ## Errors | Status | Code | Cause | |---|---|---| | 400 | `bad_status` | Not `proposed`, `accepted` or `superseded` | | 503 | `record_unavailable` | Decisions are held in the record | ## Caching `no-store`. A proposal is accepted by a person, and a stale answer says somebody has not acted when they have. --- # Stateless invoice reconciliation Reconcile a usage window and optional stated vendor invoice lines against historical Witness list-price periods in memory, deriving exact list-cost bounds and implied enterprise discounts with zero retention. ``` POST /api/v1/reconcile ``` ```bash curl -X POST https://vitna.ai/api/v1/reconcile \ -H "content-type: application/json" \ -d '{ "window": "2026-10-01..2026-10-31", "unitProfile": { "inputTokensPerUnit": 2000, "outputTokensPerUnit": 500 }, "readings": [ { "sku": "openai/gpt-5.6-luna", "units": 100000 } ], "spend": [ { "amountUsd": 85.00 } ] }' ``` ## Zero retention over confidential bills `POST /api/v1/reconcile` requires no API key and stores nothing (`Cache-Control: no-store`). Historical list prices are read from `witness.price_period` across `window`; stated `spend` lines (`sku` omitted or `null` for whole-window invoice totals, or naming one SKU) are compared against the derived list bound to report `impliedDiscountPct` (negative when billed above public list). If a price moved mid-window without daily usage granularity, `listCostUsd` and `impliedDiscountPct` report `[minUsd, maxUsd]` bounds rather than inventing an intra-month traffic split. A window the record does not hold a price across, end to end, answers `200` with `"reconcilable": false` and a `reason`, never a guessed cost. The example's model was first priced on 2026-08-21, so August 2026 is not reconcilable for it and October is. ## Response shape ```json { "reconciliation": { "reconcilable": true, "window": { "start": "2026-10-01", "end": "2026-10-31", "units": 100000, "sourceRef": "stateless-api", "importedAt": null }, "rows": [ { "skuId": "openai/gpt-5.6-luna", "units": 100000, "sharePct": 100, "priced": true, "periodIds": [13, 14], "moves": 0, "listPerUnit": { "minUsd": 0.001, "maxUsd": 0.001, "exact": true }, "listCostUsd": { "minUsd": 100, "maxUsd": 100, "exact": true }, "successRate": null, "listPerSuccessfulUnit": null } ], "pricedUnits": 100000, "unpricedUnits": 0, "coveragePct": 100, "listCostUsd": { "minUsd": 100, "maxUsd": 100, "exact": true }, "moves": 0, "spend": [ { "skuId": null, "amountUsd": 85, "comparable": true, "listUsd": { "minUsd": 100, "maxUsd": 100, "exact": true }, "impliedDiscountPct": { "minUsd": 15, "maxUsd": 15, "exact": true } } ] }, "sentence": "At list, the 100,000 priced units of this window cost $100.00, covering every unit that ran. The stated spend of $85.00 implies a discount of 15 percent off list.", "caveats": [ "List prices only: what was actually paid is whatever the bill says. A supplied spend line is compared here and stored nowhere.", "Cost only, where no production success rate is recorded at or before the window's end. Cost per successful unit is derived only from a measurement somebody entered, never assumed at 100." ], "privacy": "Reconciled in memory with zero retention; supplied vendor invoice lines and usage counts are never stored.", "meta": { "generatedAt": "2026-11-16T07:17:47.025Z", "source": "database", "count": 1 } } ``` --- # Relay spreads and Anchor manifests Keyless, ETag-cached registry feeds consumed by `vitna-relay` for wholesale cloud arbitrage and by `vitna-anchor` for verified 4KB DMA local checkpoints. ## Wholesale seller spreads for Relay ``` GET /api/v1/relay/spreads ``` Optional query parameters: `author`, `quantization`, `min_sellers` (default `2`), and `limit` (`1..1000`). Joins multi-seller OpenRouter endpoint offers with Witness first-party primary list prices (`witnessPrimaryInputUsd`) and route TTFT/throughput figures. ```json { "spreads": [ { "modelId": "deepseek/deepseek-chat", "urlSlug": "deepseek--deepseek-chat", "name": "DeepSeek V3", "author": "deepseek", "bestWholesaleSeller": "DeepInfra", "bestWholesaleInputUsd": 0.25, "dearestWholesaleInputUsd": 0.8, "spreadRatio": 3.2, "sellerCount": 5, "sellers": [ ... ] } ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "mirror_joined_with_witness", "count": 80, "totalEligible": 80 } } ``` ## List Anchor DMA manifests ``` GET /api/v1/anchor/manifests ``` Optional query parameters: `family`, `moe` (`true` or `false`), and `max_ram_gb`. Enumerates verified 4096-byte sector-aligned checkpoints with cryptographic `airgapSha256` digests, MoE expert routing counts, NVMe queue tuning profiles, and Hugging Face weight licences. ```json { "manifests": [ { "manifestVersion": "1.0.0", "engine": "vitna-anchor", "id": "qwen2.5-coder-7b-dma", "aliases": [ ... ], "name": "Qwen 2.5 Coder 7B", "family": "qwen", "precision": "INT4", "denseWeightsGb": 14.8, "dmaSlabSizeGb": 4.2, "sectorCount": 1024000, "sectorSizeBytes": 4096, "totalBytes": 4194304000, "isMoe": false, "airgapSha256": "9e48a1768c707db580b0bb11bfcfb65b1d2e1bdfc08f4c08bb2542a6c8e31a01", "tuningProfile": { "minRamGb": 16, "recPrefetchDepth": 2, "recQueueDepth": 16, "recRadixKvGb": 4 }, "pullCommand": "npx vitna-anchor pull Qwen/Qwen2.5-Coder-7B-Instruct --format dma" } ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "engine": "vitna-anchor", "count": 4, "totalRegistered": 4 } } ``` ## Single Anchor DMA manifest ``` GET /api/v1/anchor/manifests/{slug} ``` Accepts either the canonical manifest `id` (`qwen2.5-coder-7b-dma`) or any registered short alias (`qwen7b`, `llama70b`, `deepseek-v3`, `mixtral`). ```json { "manifest": { "manifestVersion": "1.0.0", "engine": "vitna-anchor", "id": "qwen2.5-coder-7b-dma", "name": "Qwen 2.5 Coder 7B", "precision": "INT4", "sectorSizeBytes": 4096, "airgapSha256": "9e48a1768c707db580b0bb11bfcfb65b1d2e1bdfc08f4c08bb2542a6c8e31a01", "pullCommand": "npx vitna-anchor pull Qwen/Qwen2.5-Coder-7B-Instruct --format dma" }, "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "engine": "vitna-anchor" } } ``` --- # Router challenge Evaluate in memory whether handing model selection to a dynamic router (`openrouter/auto`, `martian/model-router`, `notdiamond/custom-router`, `routellm/mf`, or a custom router) beats naming a fixed [SKU](../../../concepts/sku.md), with zero retention. The HTTP surface for [Router Challenge (#72)](../../../routing/router-challenge.md). ``` POST /api/v1/router-challenge ``` ```bash curl -X POST https://vitna.ai/api/v1/router-challenge \ -H "Content-Type: application/json" \ -d '{ "workload": { "id": "acme/support-triage", "name": "Support triage", "unitLabel": "tickets", "constraints": { "qualityFloor": 88.0, "economicThresholdPct": 15.0, "edgeCaseVeto": ["pii_redaction"] } }, "incumbent": { "skuId": "openai/gpt-5.6-luna", "name": "GPT-5.6 Luna", "unitCostUsd": 0.03, "successRatePct": 90.0, "latencyP90Ms": 1200 }, "router": { "skuId": "martian/model-router", "surchargePerRequestUsd": 0.004 }, "observations": [ { "units": 500, "tokenSpendUsd": 7.50, "latencyP50Ms": 450 } ], "routerEvidence": { "successRatePct": 92.0, "n": 500, "latencyP90Ms": 1400, "edgeCaseFailures": [] } }' ``` ## Response ```json { "status": "evaluated", "workloadId": "acme/support-triage", "workloadName": "Support triage", "routerSkuId": "martian/model-router", "routerName": "Martian Model Router", "outcome": "switch", "reason": "Martian Model Router beats incumbent GPT-5.6 Luna by 31.1% on cost per successful unit while meeting the 88.0% quality floor.", "checks": [ { "name": "quality floor", "threshold": "88.0%", "observed": "92.0%", "passed": true } ], "knownRouters": { ... }, "meta": { "generatedAt": "2026-09-12T09:14:02.881Z", "minRequiredUnits": 20 } } ``` ## Epistemic refusal below 20 units A dynamic router publishes no static per-token list price, so its effective unit cost is derived from observed trial telemetry plus any per-request or percentage routing fee. When `observations` sum to fewer than `20` units, the endpoint returns `200` with `status: "refused"`, `outcome: "stay"`, and the exact observation shortfall rather than extrapolating from a handful of prompts. ## Zero retention Payloads are evaluated strictly in memory and are never written to the database, disk, or application logs. ## Caching `no-store`. --- # Pushing usage The one write route this API mounts, and the only route a machine authenticates to. A runtime pushes what its own ledger says served each lane on a day that has ended, and each batch lands as readings on `/console`. ``` POST /api/v1/usage ``` ``` Authorization: Bearer vitna_push_... ``` The key is org-scoped, minted by an operator with [`org:key:create`](../../../cli/keys.md), shown once, and held as a hash: a push may only name workloads of the org the key belongs to. ## The body One payload is one closed UTC day: ```json { "window": "2026-08-24", "source": "policy ecb93b2a5c50...", "rows": [ { "workload": "convexity/deep-research", "sku": "anthropic/claude-opus-4-8", "units": 3, "windowUnits": 4 }, { "workload": "convexity/deep-research", "sku": "openai/gpt-5.6-luna", "units": 1, "windowUnits": 4 } ] } ``` `source` is where the readings trace to, normally the policy digest that was routing when the requests ran; it is carried onto every reading's `source_ref` beside the key's id. `windowUnits` is the workload's own total for the day, stored on every row so a share stays a share. ## What a push is, and is not A push is a reading, `origin: runtime`, append-only. It never adopts: `confirms_decision_id` stays null and no workload's incumbent moves, because a machine that pushed enough traffic being able to move what the register says a lane runs would be the register taking dictation. Adoption stays an operator act through [`aperture:usage:import`](../../../routing/usage-import.md). A batch is atomic. Any refused row refuses the whole batch and nothing lands, for the reason a partial config is refused. A replay is one fact. The record keeps one reading per workload per window per sku, so re-pushing a day it already holds answers with `alreadyStood` rather than landing twice. That is also why only closed days are accepted: a reading of today would be a total that is wrong by midnight, standing permanently. ## The answer ```json { "landed": 2, "alreadyStood": 0, "refusals": [] } ``` `201` when anything landed, `200` when everything already stood, and on any refusal a `4xx` whose `refusals` array names every fault at once rather than one per round trip. | Refusal | Status | |---|---| | No ingest database on this instance | 503 | | No key, an unknown key, or a revoked one (named, with "mint another") | 401 | | A body over 1 MiB (1048576 bytes), counted as it arrives | 413 | | A malformed window, source or row, each fault listed | 400 | | A day that has not ended | 400 | | A workload of another org, or one the record does not hold | 403 | ## Who can read what lands Not this route, and not the role behind it: `vitna_ingest` may insert a reading and may never select one back, which `deploy/provision-usage-ingest.sh` proves against the live role. Readings reach people on [`/console`](../../../quickstart.md), scoped by org membership, on the volume line and the confirmation under each value. --- # Stateless rollout watchdog Evaluate standing-decision drift, shadow rule promotion readiness, and routed canary health in memory against Witness list prices, provider compliance terms, and three-provenance SKU capabilities, with no API key and zero retention. ``` POST /api/v1/watchdog ``` ```bash curl -X POST https://vitna.ai/api/v1/watchdog \ -H "content-type: application/json" \ -d '{ "workload": { "id": "ci/deep-research", "classId": "model", "incumbentSkuId": "openai/gpt-5.4", "unitProfile": { "inputTokensPerUnit": 4000, "outputTokensPerUnit": 800 }, "constraints": { "qualityFloor": 93.0, "latencyP90Ms": 2500, "requiresCapabilities": ["tools"] } }, "standingDecision": { "id": 12, "outcome": "stay", "decidedAt": "2026-10-20T00:00:00.000Z", "recordedUnitCostUsd": 0.018 }, "rules": [ { "id": 4, "position": 1, "mode": "route", "kind": "traffic_share", "share": 5, "serveSkuId": "anthropic/claude-sonnet-4-6" } ], "ledgerRows": [ { "at": "2026-10-24T10:00:00Z", "outcome": "served", "serve": "rule:1", "sku": "anthropic/claude-sonnet-4-6", "cohort": 2, "latencyMs": 3100 } ], "evidence": [ { "skuId": "openai/gpt-5.4", "measuredAt": "2026-10-22", "metrics": { "successRatePct": 95.0 } }, { "skuId": "anthropic/claude-sonnet-4-6", "measuredAt": "2026-10-22", "metrics": { "successRatePct": 94.5 } } ] }' ``` ## What is checked Runs [`evaluateWorkloadWatchdog`](../../../routing/watchdog.md) in memory over the inline `workload`, optional `standingDecision`, `rules`, `fallbacks`, `observations`, `ledgerRows`, `evidence`, and `thresholds`: 1. **Standing decision drift**: unit-cost movement against `standingDecision.recordedUnitCostUsd`, provider compliance terms, [three-provenance capabilities](../../../routing/capabilities.md), evaluation `qualityFloor` and `edgeCaseVeto`, fallback reliance, unserved attempt rate, and `latencyP90Ms`. 2. **Per-rule canary ladder**: evaluates each `shadow` or `route` rule for promotion readiness (`promote_ready`), rung expansion (`expand_or_adopt_ready` across `5% -> 25% -> 50% -> adopt_incumbent`), shadow retirement (`retire_shadow`), stale decision authority (`stale_authority`), or immediate canary rollback (`rollback_recommended`), returning exact operator commands (`recommendedCommands`). Payloads are never persisted (`Cache-Control: no-store`). Also exposed to MCP clients as `vitna_check_rollout` on [`POST /api/v1/mcp`](mcp.md). ## Response shape ```json { "workload": { "id": "ci/deep-research", "name": "Inline workload", "classId": "model", "servingSkuId": "openai/gpt-5.4", "incumbentSkuId": "openai/gpt-5.4", "policyAlias": null }, "verdict": "rollback_recommended", "standingDecision": { "id": 12, "outcome": "stay", "decidedAt": "2026-10-20T00:00:00.000Z", "title": null, "recordedUnitCostUsd": 0.018, "currentUnitCostUsd": 0.018 }, "decisionChecks": [ ... ], "driftReasons": [], "observations": { "readingsWindow": "2026-10-24..2026-10-24", "servedUnits": 1, "fallbackSharePct": 0, "unservedRatePct": 0 }, "rules": [ { "id": 4, "position": 1, "ruleKey": "rule:1", "kind": "traffic_share", "mode": "route", "share": 5, "serveSkuId": "anthropic/claude-sonnet-4-6", "verdict": "rollback_recommended", "reason": "Routed canary rule 4 (anthropic/claude-sonnet-4-6) breached safety bounds: observed latency 3100ms exceeded 2500ms. Remove the rule and re-emit policy for ci.", "breaches": ["observed latency 3100ms exceeded 2500ms"], "telemetry": { ... }, "recommendedCommand": "npm run aperture:rule:remove -- 4 --reason \"Canary rollback on anthropic/claude-sonnet-4-6: observed latency 3100ms exceeded 2500ms\"" } ], "thresholds": { "minShadowMatches": 50, "minCanaryUnits": 100, "maxUnservedRatePct": 5, "maxFallbackSharePct": 10, "maxPriceDriftPct": 10 }, "recommendedCommands": [ ... ], "epistemicNotice": "Unserved attempt rates reflect transport or provider availability only; answer quality (successRatePct) enters exclusively from recorded workload evaluations.", "sentence": "ci/deep-research: rollback recommended on 1 routed rule (rule 4 -> anthropic/claude-sonnet-4-6).", "ledgerSummary": { ... }, "privacy": "Evaluated in memory with zero retention; inline workload profiles, rollout rules, ledger rows, and evaluation figures are never stored.", "meta": { "generatedAt": "2026-11-05T12:00:00.000Z", "source": "snapshot", "ruleCount": 1 } } ``` --- # Watches What each workload is watching, and what has already fired. ``` GET /api/v1/watches ``` ```bash curl "https://vitna.ai/api/v1/watches?state=armed" ``` ## Query parameters | Parameter | Values | |---|---| | `workload` | A workload id | | `state` | `armed`, `spent` | ## Response ```json { "watches": [ { "id": 3, "workloadId": "...", "kind": "price_move", "state": "armed", "params": { "minChangePct": 10 }, "firedAt": null, "watching": "..." } ], "meta": { "generatedAt": "...", "count": 5, "armed": 4, "spent": 1 } } ``` ## `watching` A sentence describing the watch in words. It comes from the same `describeTrigger` function the CLI prints, so the API and the operator's register cannot describe one watch two different ways. ## Armed and spent A trigger fires once and is then spent. It will not fire again until a decision arms a new one. That is what stops one volatile cell producing the same notification every hour until somebody mutes it, and a muted watch is a watch nobody reads. ## Public today, scoped later There is one org in the record and its decisions are the published case study, so nothing here is anybody else's. That stops being true the day a second org exists. Gating this is part of what accounts means in the plan; the shape of the answer does not change, only who may ask for it. ## Errors | Status | Code | Cause | |---|---|---| | 400 | `bad_state` | Not `armed` or `spent` | | 503 | `record_unavailable` | Watches are held in the record | ## Caching `no-store`. An armed watch becomes a spent one at the end of any collector run, and a minute of staleness reports that nothing has fired when something has. --- # Events The [market log](../../../concepts/market-event.md) as data. Curated and detected rows merged, newest first. ``` GET /api/v1/events ``` ```bash curl "https://vitna.ai/api/v1/events?kind=price&since=2026-01-01" ``` ## Query parameters | Parameter | Values | Meaning | |---|---|---| | `kind` | `launch`, `price`, `policy`, `deprecation`, `availability` | One kind | | `origin` | `curated`, `detected` | Who wrote it | | `since` | `YYYY-MM-DD` | On or after this day | All optional, and combinable. ## Response ```json { "events": [ { "date": "2026-08-22", "provider": "Anthropic", "kind": "price", "origin": "detected", "title": "...", "detail": "...", "sourceUrl": "https://...", "evidence": [ { "label": "before", "sha256": "..." }, { "label": "after", "sha256": "..." } ], "supersedes": null, "supersededBy": null } ], "meta": { "generatedAt": "...", "source": "database", "count": 12, "retrieved": "2026-08-23" } } ``` ## `date` is a day, not an instant An event records the day something happened. The read that confirmed it is on the [observation](../../../concepts/observation.md), not here. ## `evidence` on detected rows A detected price move carries the hashes of both archived documents, before and after. That is what makes the move checkable rather than merely stated: fetch both, compare, and see the change yourself. ## `supersedes` and `supersededBy` The log is append-only, so a row that was wrong is corrected by a new row and never edited. The correction carries `supersedes`, naming the row it restates as ` `, and that row carries `supersededBy`, naming the correction back. Both are `null` on a row that is neither. A correction keeps the date of the row it restates and says the true date in its text where the two differ. It may restate a detected row: a move read from a figure that was misread, or right about the price and wrong about the day. Read a row with a `supersededBy` as what the record said, not what it says. ## Errors | Status | Code | Cause | |---|---|---| | 404 | `unknown_kind` | Not one of the five kinds | | 400 | `bad_since` | Not an ISO date. The message shows the expected form. | An unknown `origin` is not rejected; it simply matches nothing. ## Same source as the page This reads the same loaded snapshot the `/log` page renders, so the page and the API cannot drift. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/log/feed.md --> # Feed JSON Feed 1.1. The market log plus the triggers that fired and the decisions the engine proposed, in something a reader, a bot or a chat channel can subscribe to. ``` GET /api/v1/feed.json ``` ```bash curl https://vitna.ai/api/v1/feed.json ``` Served as `application/feed+json; charset=utf-8`. ## Shape Standard JSON Feed 1.1 (`version`, `title`, `home_page_url`, `feed_url`, `description`, `language`, `items`), so any feed reader handles it unmodified. Fifty items, newest first. A feed is the recent past, not the archive; use [`/events`](events.md) to go further back. ## Three item kinds Every item carries a `_vitna` extension object whose `kind` says which: | `_vitna.kind` | Written when | |---|---| | `market_event` | A price moved, something launched, terms changed | | `trigger_fired` | An armed watch matched the record | | `proposal` | The engine produced a decision brief | ## `_vitna` on a market event Carries `eventKind`, `origin`, `provider`, `sku` and `evidence`. Where the event was detected rather than curated, the hashes of both archived documents are appended to `content_text` as well, so a reader who wants to check a move does not have to come back for them. ## Trigger items say what did not happen A fired trigger's `content_text` states that the trigger is spent, that nothing was routed and nothing was accepted, and that a fired trigger is a question put to a person. That sentence is in the payload rather than only in these docs, because a notification implying the system took an action it did not take is the worst thing this feed could carry. ## Proposal items carry their ruling A proposal's status travels with it, so a reader returning to an item sees whether a person has ruled on it: nobody has ruled yet, a person accepted it, or a person refused it with the reason kept on the row. That third reading was "it was overtaken without being accepted" until 2026-08-26, which is what the status sounds like rather than what writes it. See [the decision concept](../../../concepts/decision.md). ## Proposal items link to the decision A proposal item carries a `url`, which is the decision's own page: ``` https://vitna.ai/decisions/{id} ``` Every proposal this feed carries belongs to an org whose feed is public, and that is the same gate the page reads, so following one of these links never lands on a 404. ## Market event items link to the event A market event item's `url` is the event's own page, at its day and a slug of its headline: ``` https://vitna.ai/log/{day}/{slug} ``` Its `external_url` is still the provider's own announcement. Until 2026-09-13 the `url` was the log anchored at the event's day, which landed on the first event of that day rather than this one whenever two shared it. The day and slug are the same ones the item's id is built from. Where two events share a day and a headline, which the record allows for detected moves, the one written first keeps the bare address and each later one takes `--2`, `--3` in the order it was written, so an address already handed out never moves. ## Item ids are stable A market event's id is its day plus a slug of its headline, which is what already identifies a curated event in the database through a partial unique index. Trigger and proposal items use their record ids. Ids do not change, so a reader will not see the same item twice. ## The same items as RSS The same fifty items are published as RSS 2.0 at `https://vitna.ai/log/rss.xml`, for a feed reader that looks for RSS rather than JSON Feed. It is a rendering of this feed and never a second one: every `guid` is the id this feed gives the same item, so a reader following both sees one item, and its `lastBuildDate` is the newest item's own date rather than the time of the request. The log page and every event page advertise both feeds as `<link rel="alternate">` in their head. ## The engine half can be absent If the engine's record cannot be read, trigger and proposal items are omitted and `_vitna.engineActivity` says so in words, rather than the feed appearing quiet. | `_vitna.source` | `_vitna.engineActivity` | |---|---| | `database` | `included` | | `snapshot` | a sentence explaining it could not be read | A feed reporting "no triggers have fired" while unable to look would be the failure this record exists to prevent. ## Only published registers are here The trigger and proposal items on this feed come only from organisations whose register is published. An organisation whose register is private has its own feed instead, at an address its owners get from the console: [your organisation's feed](../engine/org-feed.md). That feed carries its triggers and proposals and none of the market log, and this one is unchanged by it. ## Why a feed and not email Email needs SMTP credentials, a monitored inbox and a deliverability problem. A feed needs none of those, is public and cacheable, and every consumer already knows how to read one. Email remains documented in `HOSTING.md` as an add-on for whenever there is an inbox somebody reads. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/catalog.md --> # Catalog, benchmarks and capability Mirrored model metadata, Hugging Face weight licences, route speed measurements, multi-seller endpoint offers, openly licensed benchmark boards, and Vitna's IRT capability scale. Kept separate from `/api/v1/markets` so `meta.provenance` always distinguishes mirrored third-party data from Witness first-party provider observations. ## List catalog models ``` GET /api/v1/catalog/models ``` Optional query parameters: `author`, `modality`, `capability` (`tools`, `reasoning`, `structured`), `open_weights` (`true` or `false`), `q`, and `limit` (`1..2000`). ```json { "models": [ { "id": "deepseek/deepseek-chat", "urlSlug": "deepseek--deepseek-chat", "author": "deepseek", "authorName": "DeepSeek", "slug": "deepseek-chat", "name": "DeepSeek V3", "contextLength": 163840, "inputModalities": [ ... ], "outputModalities": [ ... ], "pricing": { "input": 0.3, "output": 0.88 }, "huggingFaceId": "deepseek-ai/DeepSeek-V3", "weights": { "repo": "deepseek-ai/DeepSeek-V3", "license": "mit" }, "sellerCount": 5 } ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "mirror", "count": 340, "totalCatalog": 340, "sources": [ { "id": "openrouter-models", "url": "https://openrouter.ai/api/v1/models", "retrievedAt": "2026-09-12T00:00:00.000Z", "attribution": "OpenRouter public model list" } ] } } ``` ## Single catalog model ``` GET /api/v1/catalog/models/{slug} ``` Accepts `{slug}` as `author--slug` (`deepseek--deepseek-chat`), URL-encoded `author%2Fslug`, or bare `slug`. ```json { "model": { "id": "deepseek/deepseek-chat", "urlSlug": "deepseek--deepseek-chat", "name": "DeepSeek V3", "contextLength": 163840, "pricing": { "input": 0.3, "output": 0.88 }, "speedEndpoints": [ ... ], "sellers": [ ... ] }, "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "mirror", "sources": [ ... ] } } ``` ## Model seller offers ``` GET /api/v1/catalog/models/{slug}/sellers ``` Returns every seller endpoint mirrored for one model, with list input/output rates, declared quantization, context window, and dispersion summary. ```json { "modelId": "deepseek/deepseek-chat", "urlSlug": "deepseek--deepseek-chat", "listedPricing": { "input": 0.3, "output": 0.88 }, "sellers": [ { "provider": "DeepInfra", "input": 0.28, "output": 0.88, "quantization": "fp8", "contextLength": 163840 } ], "summary": { "sellerCount": 5, "cheapestInputUsd": 0.25, "dearestInputUsd": 0.8, "spreadRatio": 3.2, "quantizations": [ ... ] }, "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "mirror", "notice": "Quantization is the seller's own declaration.", "sources": [ ... ] } } ``` ## Witness SKU capability profile ``` GET /api/v1/catalog/skus/{slug}/capabilities ``` Resolves a Witness SKU (`provider/slug` or bare `slug`, such as `claude-sonnet-4-6` or `gpt-5.4`) into its three-provenance capability profile: `declared_mirror` capabilities and context window, `measured_third_party` throughput and TTFT with sample size and retrieval timestamp, curated Witness `providerTerms` when present, and the engine's admissibility contract (`admissibility`). See [Three-provenance capability eligibility](../../../routing/capabilities.md). ```json { "sku": { "id": "anthropic/claude-sonnet-4-6", "slug": "claude-sonnet-4-6", "name": "Claude Sonnet 4.6", "classId": "model", "providerId": "anthropic", "catalogListingId": "anthropic/claude-sonnet-4.6" }, "provenanceClasses": { "declared_mirror": { "provenance": "declared_mirror", "listingId": "anthropic/claude-sonnet-4.6", "capabilities": [ ... ], "contextLength": 200000, "source": "https://openrouter.ai/api/v1/models", "retrievedAt": "2026-09-12T00:00:00.000Z" }, "measured_third_party": { "provenance": "measured_third_party", "listingId": "anthropic/claude-sonnet-4.6", "measurer": "OpenRouter speed mirror", "throughputToksSec": 46.7, "latencyTtftMs": 786, "requests": 58622, "windowMinutes": 30, "retrievedAt": "2026-09-12T00:00:00.000Z" }, "measured_customer": { "provenance": "measured_customer", "metricKey": "verifiedCapabilities", "requiredFields": [ ... ] } }, "providerTerms": { ... }, "admissibility": { "declared_mirror": { "provenance": "declared_mirror", "answersConstraints": [ ... ], "cannotAnswerConstraints": [ ... ] }, "measured_third_party": { "provenance": "measured_third_party", "answersConstraints": [ ... ], "cannotAnswerConstraints": [ ... ] }, "measured_customer": { "provenance": "measured_customer", "answersConstraints": [ ... ], "cannotAnswerConstraints": [ ... ] } }, "knownCapabilityTokens": [ ... ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "three_class_profile", "sources": [ ... ] } } ``` ## Openly licensed benchmark boards ``` GET /api/v1/catalog/benchmarks ``` Optional query parameters: `class`, `board`, and `summary=true`. Republishes only boards whose results licence was verified by `platform/boards/sources.mjs`, carrying each publisher's `licence`, `runBy`, `metric`, and mandatory `notice`. ```json { "boards": [ { "id": "aider", "label": "Aider polyglot", "classId": "model", "licence": "Apache-2.0", "runBy": "publisher", "metric": "pass rate (%)", "notice": "Source: Aider LLM Leaderboards", "url": "https://aider.chat/docs/leaderboards/", "retrievedAt": "2026-09-12T00:00:00.000Z", "columns": [ ... ], "modelCount": 60, "rowCount": 69, "rows": [ ... ] } ], "unread": [ ... ], "unreadable": [ ... ], "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "mirror", "count": 6, "sources": [ ... ] } } ``` ## Fitted capability scale ``` GET /api/v1/catalog/capability ``` Optional query parameters: `ranked_only=true` and `board`. Serves Vitna's IRT capability scale fit over the openly licensed benchmark mirrors (`mean = 100`, `standardDeviation = 15`), together with column difficulties, board ties, and excluded disconnected groups. ```json { "runs": [ { "id": "openai/gpt-5.4", "index": 118.4, "columns": 22, "boards": 4, "ranked": true } ], "columns": [ { "id": "aider:polyglot", "difficulty": 98.2, "discrimination": 1.4 } ], "ties": [ ... ], "excluded": [ ... ], "scale": { "mean": 100, "standardDeviation": 15, "minColumns": 3, "minShared": 3 }, "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "snapshot", "provenance": "derived_over_mirror", "runCount": 120, "rankedCount": 85, "columnCount": 40, "sources": [ ... ] } } ``` --- <!-- https://vitna.ai/docs/api/v1/prices/exports.md --> # Historical data exports Paginated bulk export of Witness first-party datasets (`price-periods`, `observations`, `rates`, `events`, and `provider-terms`) in JSON, JSONL (`application/x-ndjson`), or CSV (`text/csv`) for Deploy and Enterprise organisations. ## Export a dataset ``` GET /api/v1/exports/{dataset} ``` Requires `Authorization: Bearer vitna_push_...` (or `vitna_key_...`) for an organisation with Deploy or Enterprise entitlement. Supported `{dataset}` values: - `price-periods` - historical and current stretches of unchanged SKU rates - `observations` - individual parsed cells with archived `contentSha256` digests - `rates` - open (`valid_to IS NULL`) Witness periods - `events` - published curated and detected market events - `provider-terms` - append-only curated provider retention, training, region, and BYOK terms Query parameters: - `class` - filter by component class (`model`, `gpu`, `retrieval`, `speech`, `tool`, `image`, `video`, `music`) - `provider` - filter by provider id (`openai`, `anthropic`, etc.) - `from` and `to` - ISO dates (`YYYY-MM-DD`) - `after` - non-negative integer cursor from `nextCursor` or `X-Vitna-Next-Cursor` - `limit` - page size (`1..5000`, default `500`) - `format` - `json` (default), `jsonl`, or `csv` ```bash curl -H "Authorization: Bearer $VITNA_PUSH_KEY" \ "https://vitna.ai/api/v1/exports/price-periods?class=model&from=2026-01-01&format=jsonl&limit=1000" ``` ```json { "dataset": "price-periods", "rows": [ { "id": 412, "skuId": "gpt-5.4", "skuSlug": "gpt-5.4", "classId": "model", "providerId": "openai", "dimensionId": "input_per_mtok", "unit": "USD per 1M tokens", "amountUsd": 2.5, "bandFromTokens": 0, "bandToTokens": null, "validFrom": "2026-06-01T00:00:00.000Z", "validTo": null, "confirmed": true, "flap": false, "firstObservationId": 1204, "lastObservationId": 8840 } ], "nextCursor": "412", "meta": { "generatedAt": "2026-11-02T00:00:00.000Z", "source": "postgres", "orgId": "convexity", "count": 1, "limit": 500, "nextCursor": "412" } } ``` --- <!-- https://vitna.ai/docs/api/v1/prices/litellm-cost-map.md --> # LiteLLM cost map Witness's attested language-model and retrieval rates rendered in the JSON dictionary format LiteLLM's `--model_cost_map_url` (`LITELLM_MODEL_COST_MAP_URL`) loads directly, with a `vitna` provenance block on every entry. ``` GET /api/v1/gateways/litellm/model-prices.json ``` ```bash curl https://vitna.ai/api/v1/gateways/litellm/model-prices.json ``` Point a LiteLLM proxy at the live record without maintaining a hand-edited JSON fork: ```bash LITELLM_MODEL_COST_MAP_URL=https://vitna.ai/api/v1/gateways/litellm/model-prices.json litellm --config litellm.yaml ``` ## Response ```json { "openai/gpt-5.4-pro": { "litellm_provider": "openai", "mode": "chat", "input_cost_per_token": 0.00003, "output_cost_per_token": 0.00018, "input_cost_per_token_batches": 0.000015, "output_cost_per_token_batches": 0.00009, "vitna": { "sku": "gpt-5.4-pro", "providerId": "openai", "class": "model", "sourceUrl": "https://developers.openai.com/api/docs/pricing", "confirmedAt": null, "stale": false, "derived": false, "evidenceSha256": null, "citationUrl": null, "bands": [ { "fromTokens": 0, "toTokens": 272000, "label": "to 272K tokens", "inputCostPerToken": 0.00003, "outputCostPerToken": 0.00018 } ] } }, "meta": { "generatedAt": "2026-09-12T09:14:02.881Z", "source": "database", "skuCount": 260, "keyCount": 590 } } ``` ## Keying and units - **Per-token rates**: Witness stores token meters per million tokens (`input_per_mtok`, `output_per_mtok`, `embed_per_mtok`) to six decimal places. This endpoint divides each figure by `1,000,000` rounded to twelve decimal places so all six digits survive without floating-point tail noise. - **Aliases emitted**: Each SKU is indexed under its Witness `<slug>`, `<providerId>/<slug>`, and `<litellm_provider>/<apiModel>` when the SKU register records a wire model identifier. First-party rows claim bare unprefixed names ahead of hosted resellers. - **Context bands**: Base-band rates populate the top-level fields. Exact 128K and 200K thresholds also populate LiteLLM's tiered keys, while the full band schedule is always preserved in `vitna.bands` so non-200K thresholds (such as 272K) are never misstated. - **Provenance**: When served from `vitna.ai` (`meta.source: "database"`), `vitna.citationUrl` resolves directly to `/cite/{id}` and `vitna.evidenceSha256` names the archived pricing page. ## Caching `public, max-age=60, stale-while-revalidate=3600`. ## Next step Pair this cost map with local token profiling (`vitna-runtime-profile`) and CI eval normalization without changing your production gateway first: see [Three-minute gateway quickstart](../../../quickstart.md#three-minute-gateway-quickstart-litellm-and-openrouter) and [Gateway adapters](../../../routing/adapters.md). --- <!-- https://vitna.ai/docs/api/v1/prices/markets.md --> # Markets Every [component class](../../../concepts/component-class.md) Vitna prices, with counts and freshness. The index of the record. ``` GET /api/v1/markets ``` ```bash curl https://vitna.ai/api/v1/markets ``` ## Response ```json { "classes": [ { "class": "model", "label": "Language models", "unit": "One successful task", "wired": true, "priceCount": 918, "skuCount": 230, "providerCount": 14, "freshestConfirmedAt": "2026-08-23T08:00:11.402Z", "stalestConfirmedAt": "2026-08-22T19:00:04.118Z", "anyStale": false } ], "meta": { "generatedAt": "...", "source": "database" } } ``` ## Fields | Field | Meaning | |---|---| | `class` | The id to pass to `/markets/{class}/tape` | | `label` | Human name | | `unit` | What you are actually buying in this class | | `wired` | Whether Vitna has live sources yet | | `priceCount` | Priced cells, counting each dimension separately | | `skuCount` | Distinct skus | | `providerCount` | Distinct providers | | `freshestConfirmedAt` | Most recent read in the class | | `stalestConfirmedAt` | Oldest read in the class | | `anyStale` | Whether any cell is past its cadence | `priceCount` counts cells, not skus. A model publishing input, cached input and output prices contributes three. ## Watch the pair of timestamps `freshestConfirmedAt` alone is misleading: one source refreshing on schedule keeps it current while everything else silently ages. `stalestConfirmedAt` is the one that shows that, and `anyStale` is the one-glance version. ## Empty classes appear here A class with nothing priced is listed with zero counts, null timestamps and `anyStale: false`, never dropped. There are two ways to be empty, and `wired` tells them apart: | `wired` | Zero counts mean | |---|---| | `false` | The class is registered and Vitna does not read it yet | | `true` | Vitna reads the class, and no seller it reads prices anything in it today | `video` is the second kind. Google shut down Veo 3.1 Preview on 2026-10-22, and its pricing page dropped every per-second figure the class held, so all seven skus were retired and their periods closed on the last read that priced them. The source still reads the page every cycle: those reads are what closed the periods, and a video price that comes back is a read that fails loudly rather than one that passes as an empty page. The history of all seven stays at [`/skus/{slug}/history`](sku-history.md). `extraction` and `speech` were the first kind until 2026-09-10, when both were wired. A class that answers with zeroes is a gap a reader can hold the record to, and a class dropped from the answer is a gap only its authors know about. ## Parameters None. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/providers.md --> # Providers Every company whose own pages Vitna reads a price from, with the pages read, the markets each sells into, and any registered products no longer on the board. ``` GET /api/v1/providers GET /api/v1/providers/{id} ``` ```bash curl https://vitna.ai/api/v1/providers curl https://vitna.ai/api/v1/providers/openai ``` ## Index response ```json { "summary": "45 companies whose own pages Vitna reads a price from...", "providers": [ { "id": "openai", "name": "OpenAI", "homepage": "https://openai.com", "path": "/providers/openai", "pricedSkuCount": 42, "registeredSkuCount": 45, "awaitingSkuCount": 0, "withdrawnSkuCount": 3, "unattestedSkuCount": 0, "livePageCount": 2, "cadence": "every hour", "freshestConfirmedAt": "2026-09-12T08:00:11.402Z", "markets": [ { "class": "model", "label": "Language models", "rowCount": 35, "skuCount": 35 } ], "summary": "Vitna reads 2 pages from OpenAI every hour..." } ], "meta": { "generatedAt": "2026-09-12T09:14:02.881Z", "source": "database", "count": 45 } } ``` ## Provider detail response ```json { "provider": { "id": "openai", "name": "OpenAI", "homepage": "https://openai.com", "path": "/providers/openai", "termsUrl": "/api/v1/providers/openai/terms", "pricedSkuCount": 42, "registeredSkuCount": 45, "livePageCount": 2, "cadence": "every hour", "pages": [ { "id": "openai-pricing-md", "publicUrl": "https://developers.openai.com/api/docs/pricing", "cadenceMinutes": 60, "live": true, "notes": null } ], "markets": [ { "class": "model", "label": "Language models", "skuCount": 35, "skus": [ { "slug": "gpt-5.4-pro", "name": "gpt-5.4-pro", "kind": "model", "historyUrl": "/api/v1/skus/gpt-5.4-pro/history", "confirmedAt": null, "stale": false, "derived": false } ] } ], "withdrawn": [ ... ], "unattested": [ ... ], "summary": "Vitna reads 2 pages from OpenAI every hour..." }, "meta": { "generatedAt": "2026-09-12T09:14:02.881Z", "source": "database" } } ``` ## Joined through the register, never by display name A board row belongs to a provider through `sku.providerId` in the [SKU](../../../concepts/sku.md) register rather than by matching the display string printed on a board. One company spelled two ways across two markets still resolves to one row here and on `/providers/{id}`. ## `withdrawn` vs `unattested` Both arrays list registered SKUs that are absent from the current boards, and the distinction is the point: | Key | Meaning | |---|---| | `withdrawn` | A healthy source read its page on schedule and the SKU was no longer on it | | `unattested` | The provider's source could not read its page, so nobody can speak for whether the SKU is still sold | ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/quote-check.md --> # Quote Check Evaluate a vendor quote in memory against historical published list prices, peer spreads, and reference rates. ``` POST /api/v1/quote-check GET /api/v1/quote-check ``` ```bash # Evaluate in memory with POST curl -X POST https://vitna.ai/api/v1/quote-check \ -H "Content-Type: application/json" \ -d '{ "sku": "together-kimi-k3", "quotedAmountUsd": 2.50, "dimension": "input_per_mtok" }' # Or check via query parameters curl "https://vitna.ai/api/v1/quote-check?sku=lambda-h100-sxm&amount=2.49&dimension=gpu_hour_on_demand" # A long-context quote on a model priced in context bands curl "https://vitna.ai/api/v1/quote-check?sku=gpt-5.4-pro&amount=60&dimension=input_per_mtok&promptTokens=300000" ``` ## Zero retention guarantee A pasted quote is confidential enterprise procurement data. The request is evaluated purely in memory and is never written to disk, database, or server logs. Responses carry `Cache-Control: no-store`. ## Parameters | Field | Type | Description | |---|---|---| | `sku` | string | SKU slug or ID (e.g. `together-kimi-k3`, `gpt-5.6-sol`, `lambda-h100-sxm`). Required. | | `quotedAmountUsd` | number | Quoted price in USD per unit. Required. | | `dimension` | string | Meter dimension (e.g. `input_per_mtok`, `gpu_hour_on_demand`). Optional; defaults to class standard. | | `date` | string | ISO date `YYYY-MM-DD`. Optional; defaults to today. Future dates are refused. | | `promptTokens` | integer | Prompt length, in tokens, of the requests the quote covers. Optional. On a model priced in [context bands](#context-bands) it picks the band the quote is compared with; without it, the band from the first token. A whole number, 0 or more. As a query parameter it is digits only, so `300,000` is refused rather than read as 300. | ## Response The answer to the first request above, as the record stood on 17 November 2026: ```json { "quote": { "quotedAmountUsd": 2.5, "asOfDate": "2026-11-17", "unit": "USD per 1M tokens", "promptTokens": null }, "target": { "skuId": "together/kimi-k3", "skuSlug": "together-kimi-k3", "skuName": "Kimi K3", "providerId": "together", "classId": "model", "product": "Kimi K3", "productSlug": "kimi-k3", "dimensionId": "input_per_mtok", "dimensionLabel": "Input price", "unit": "USD per 1M tokens" }, "vendorList": { "found": true, "source": "database", "listPriceUsd": 3, "unit": "USD per 1M tokens", "bandFromTokens": 0, "bandToTokens": null, "bandLabel": null, "validFrom": "2026-10-12T07:08:29.430Z", "validTo": null, "confirmed": true, "confirmedAt": "2026-11-17T17:10:09.011Z", "evidenceSha256": "a0843851a01bb156386380c57a7c21a7e4a0b35845c3e1e6c4dcafa0d36702e7", "sourceUrl": "https://www.together.ai/pricing", "bands": [ { "bandFromTokens": 0, "bandToTokens": null, "bandLabel": null, "listPriceUsd": 3 } ], "discountVsListPct": 16.7, "listComparisonKind": "discount" }, "peerSpread": { "available": true, "product": "Kimi K3", "productSlug": "kimi-k3", "dimension": "input_per_mtok", "unit": "USD per 1M tokens", "providerCount": 6, "minUsd": 2.85, "maxUsd": 3.3, "spreadPct": 16, "position": "below_cheapest", "quotedVsPeerMinPct": 12.3, "quotedVsPeerMaxPct": null, "providers": [ { "provider": "DeepInfra", "skuSlug": "deepinfra-kimi-k3", "amountUsd": 2.85, "derived": false, "stale": false }, { "provider": "Parasail", "skuSlug": "parasail-kimi-k3", "amountUsd": 3, "derived": false, "stale": false }, { "provider": "Together AI", "skuSlug": "together-kimi-k3", "amountUsd": 3, "derived": false, "stale": false }, { "provider": "Novita AI", "skuSlug": "novita-kimi-k3", "amountUsd": 3, "derived": false, "stale": false }, { "provider": "Baseten", "skuSlug": "baseten-kimi-k3", "amountUsd": 3, "derived": false, "stale": false }, { "provider": "AWS", "skuSlug": "aws-kimi-k3", "amountUsd": 3.3, "derived": true, "stale": false } ], "note": null }, "referenceRate": { "available": true, "medianUsd": 3, "minUsd": 2.85, "maxUsd": 3.3, "minProviders": 3, "providerCount": 6, "method": "median-of-confirmed-list-v1", "coversOn": "2026-11-16", "discountVsReferenceRatePct": 16.7, "note": null }, "notices": { "scope": "Published list price comparison only. Enterprise quotes reflect bilateral commitments, volume tiers, support SLAs, and geographical routing terms that public list prices do not capture.", "privacy": "Quotes are evaluated in memory and never stored. Bilateral enterprise terms remain confidential.", "band": null }, "meta": { "generatedAt": "2026-11-17T17:28:04.597Z", "source": "database" } } ``` A `peerSpread` on a cache-write meter also carries `blockNote`, the same qualification [a spread](spreads.md#cache-writes) carries: sellers cache in blocks of their own size, so equal rates need not buy the same caching. ## Context bands Some models are priced by prompt length, one list price up to a threshold and another above it, and the record keeps a price per [band](tape.md#context-bands). A quote is for requests of some length, so it can say which. - **`promptTokens` picks the band.** The quote is compared with the band that holds a prompt of that many tokens. The record's lower bound is exclusive and its upper bound inclusive, so a prompt of exactly 272,000 tokens is billed in the band to 272K tokens. - **Without it, the band from the first token is compared.** That is the rate at the shortest prompt length the provider prices, and on a banded model it is not what a long prompt pays. `notices.band` says so in words. - **The answer names the band either way.** `vendorList.bandFromTokens`, `bandToTokens` and `bandLabel` describe the band compared, and `vendorList.bands` lists every band the meter is priced in on that date, with its list price, from the shortest prompt up. A meter priced once has one band, `(0, null]`, with a null label, and `notices.band` is null. - **A prompt no band holds has no list price.** Some meters are priced only up to a threshold. The list check then answers `found: false`, its band keys null, with the bands it does have listed. Peer spreads and reference rates are drawn over each seller's rate at the shortest prompt length it prices, which is the only band every seller on a peer set publishes ([spreads](spreads.md)). So a quote in a band above the first token is compared with neither: `peerSpread` answers `available: false` with `position: "not_compared"`, its counts and figures null and a `note` saying why, and `referenceRate` answers `available: false` with a `note`. A seller priced once has a single band from the first token, so its quote is compared at any prompt length, against the other sellers' shortest-band rates, which is what a spread holds. ```json { "quote": { "quotedAmountUsd": 60, "asOfDate": "2026-11-16", "unit": "USD per 1M tokens", "promptTokens": 300000 }, "vendorList": { "found": true, "listPriceUsd": 60, "bandFromTokens": 272000, "bandToTokens": null, "bandLabel": "over 272K tokens", "bands": [ { "bandFromTokens": 0, "bandToTokens": 272000, "bandLabel": "to 272K tokens", "listPriceUsd": 30 }, { "bandFromTokens": 272000, "bandToTokens": null, "bandLabel": "over 272K tokens", "listPriceUsd": 60 } ], "discountVsListPct": 0, "listComparisonKind": "at_list" }, "peerSpread": { "available": false, "providerCount": null, "position": "not_compared", "note": "Peer spreads compare each seller's rate at the shortest prompt length it prices, and this quote is in the band over 272K tokens, so none is compared." }, "notices": { "band": "The record prices this meter in 2 context bands: to 272K tokens and over 272K tokens. A prompt of 300,000 tokens is billed in the band over 272K tokens, and the quote is compared with that band's list price." } } ``` The same quote with no `promptTokens` is compared with the $30 band and reads as a premium, and its `notices.band` says that it was compared with the band to 272K tokens and that sending `promptTokens` compares it with the band the prompt is billed in. That default is what this check always did, and until 2026-11-16 it did it without saying so. ## Published list boundary Enterprise quotes carry volume commitments, dedicated capacity, latency SLAs, and custom routing terms that public list prices do not include. Quote Check compares strictly against the vendor published list on that date. --- <!-- https://vitna.ai/docs/api/v1/prices/rates.md --> # Reference rates The published [reference rates](../../../concepts/reference-rate.md): one number per product per day, the median list price among providers publishing one. ``` GET /api/v1/rates ``` ```bash curl https://vitna.ai/api/v1/rates ``` ## Response ```json { "rates": [ ... ], "method": "...", "rule": [ "A series exists for a product at least 3 distinct providers publish a confirmed price for on the day.", "The rate is the median of those prices: the middle one, or the mean of the two middle ones, with the halfway case rounded up.", "Arithmetic is exact, in whole millionths, which is the precision the record stores.", "A price read once and not yet confirmed by a second read does not count.", "A series is never restated. A day is published once, and a correction is a new row on a later day." ], "isNot": [ "a price anyone can buy at", "a volume-weighted average, since the record holds no volumes", "a forecast" ], "meta": { "generatedAt": "...", "source": "database", "count": 24 } } ``` ## The method travels with the numbers `rule` and `isNot` are in the payload, not linked from it. A benchmark whose method a reader has to go and find is one most readers will use without reading. The whole claim being made here is that the rule is fixed and checkable, so it is served where the numbers are read. `isNot` is there because all three are things a reader could reasonably assume about something called a reference rate, and none of them is true. ## Serving the snapshot An instance that cannot read the published rates answers `200` with `rates: null`, the `method` and `rule` still present, and a `note` pointing at `https://vitna.ai/api/v1/rates`. `null` rather than `[]`: an empty array would say no rates have ever been published, which is a claim about the record rather than about this instance. ## Parameters None. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/sku-history.md --> # Sku history Every [price period](../../../concepts/price-period.md) recorded for one sku. ``` GET /api/v1/skus/{slug}/history ``` ```bash curl "https://vitna.ai/api/v1/skus/gpt-5-pro/history?dimension=input_per_mtok" ``` ## Parameters | Parameter | In | Meaning | |---|---|---| | `slug` | path | The sku slug, as it appears on the tape | | `dimension` | query | Restrict to one meter. Optional. | Omitting `dimension` returns periods across every dimension the sku carries. A `dimension` the sku's class is not metered by answers `404 unknown_dimension` rather than an empty history, which would read as a price never recorded. ## Response ```json { "sku": { "slug": "gpt-5-pro", "name": "gpt-5-pro", "provider": "OpenAI", "class": "model", "attrs": {} }, "periods": [ { "dimension": "input_per_mtok", "amountUsd": 15, "unit": "USD per 1M tokens", "bandFromTokens": 0, "bandToTokens": null, "bandLabel": null, "since": "2026-08-07T00:00:00.000Z", "until": null, "confirmed": true, "flap": false, "confirmedObservationId": "412" } ], "meta": { "generatedAt": "...", "source": "database", "count": 1, "units": ["USD per 1M tokens"] } } ``` The period with `until: null` is the current price, and its `since` is the `since` the [tape](tape.md) prints for that meter. There is exactly one open period per dimension and context band, enforced by a partial unique index. ## History needs the record Price history accumulates in the database. The committed snapshot is a picture of one moment and contains no history at all. So an instance serving the snapshot answers `200` with the sku, an **empty** `periods` array, `meta.source: "snapshot"`, and a `meta.note` saying that history is held in the record. It does not invent a single period out of the current price. That would look exactly like a price that has never moved, which is a claim about the market rather than about this instance. `vitna.ai` serves the record, so this only affects local checkouts. ## Errors | Status | Code | Cause | |---|---|---| | 404 | `unknown_sku` | No registered sku with that slug | | 404 | `unknown_dimension` | The sku's class is not metered by that `dimension` | ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/snapshot.md --> # Snapshot All ten board documents in exactly the shape `app/data/` holds them: the model board, the GPU board, and the eight meter boards. ``` GET /api/v1/snapshot ``` ```bash curl https://vitna.ai/api/v1/snapshot ``` ## Response ```json { "modelMarket": { "meta": { "unit": "USD per 1M tokens" }, "data": [ ... ] }, "gpuMarket": { "meta": { "unit": "USD per GPU-hour" }, "products": [ ... ], "providers": [ ... ] }, "toolMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "retrievalMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "extractionMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "imageMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "videoMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "musicMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "translationMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "speechMarket": { "meta": { "units": [ ... ] }, "data": [ ... ] }, "meta": { "generatedAt": "...", "source": "database", "units": [ ... ] } } ``` ## What it is for The daily mirror. `scripts/export-snapshot.mjs` fetches this and overwrites the JSON files under `app/data/` verbatim. That keeps two logs at two resolutions: git is the daily one, with a commit per day and a diff you can read; the database is the hourly one. Neither is derived from the other after the fact. ```bash node scripts/export-snapshot.mjs https://vitna.ai ``` ## Board shape, not tape shape These are the documents the site renders, so they keep board structure. The GPU market is a matrix: a `products` array, and a `providers` array whose `cells` line up with it positionally, with `null` where a provider does not rent that card. If you want rows rather than boards, use [`/markets/{class}/tape`](tape.md), which flattens all of this. ## `meta.units` is an array A meter board can be metered several ways (the tool board is), so each one contributes all of its units rather than one. A single unit per board would have to be wrong for most of the tool board's rows. ## Parameters None. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/spreads.md --> # Spreads What several providers charge for the same product, cheapest first, with the percentage between the low and the high. ``` GET /api/v1/spreads/{class}/{product} ``` ```bash curl https://vitna.ai/api/v1/spreads/compute/h100-sxm curl "https://vitna.ai/api/v1/spreads/model/kimi-k3?dimension=output_per_mtok" ``` ## Parameters | Parameter | In | Meaning | |---|---|---| | `class` | path | `model`, `tooling`, `retrieval` or `compute` | | `product` | path | Product slug, lowercased and dashed | | `dimension` | query | Which meter. Optional. | Product slugs come from the sku's `attrs`, not from a separate table. `H100 SXM` is `h100-sxm`. ## Default dimension Omitting `dimension` gives the one the boards draw: | Class | Default | |---|---| | `model` | `input_per_mtok` | | `tooling` | `tool_call_per_1k` | | `retrieval` | `embed_per_mtok` | | `extraction` | `page_ocr` | | `compute` | `gpu_hour_on_demand` | ## Response ```json { "class": "compute", "product": "H100 SXM", "productSlug": "h100-sxm", "dimension": "gpu_hour_on_demand", "unit": "USD per GPU-hour", "providerCount": 6, "minUsd": 1.99, "maxUsd": 3.29, "spreadPct": 65, "freshestAt": "2026-08-23T08:00:11.402Z", "stalestAt": "2026-08-22T19:00:04.118Z", "anyStale": false, "providers": [ { "provider": "...", "skuSlug": "...", "amountUsd": 1.99, "derived": false, "stale": false, "fetchedAt": "...", "evidenceSha256": "..." } ], "dimensions": ["gpu_hour_on_demand"], "meta": { "generatedAt": "...", "source": "database" } } ``` `spreadPct` is `(max - min) / min`, as a rounded whole percent. It is expressed against the cheapest seller, so 65 means the dearest costs 65 percent more than the cheapest, not that it is 65 percent of it. `dimensions` lists every meter this product has a spread for, so a caller can discover what else to ask for without a second request. ## Two providers minimum A dimension with fewer than two priced providers has no spread and is omitted. A single-seller meter is a price, not a spread, and reporting a zero percent spread for it would suggest agreement between sellers that do not exist. ## Cache writes A spread on `cache_write_5m_per_mtok` or `cache_write_1h_per_mtok` carries a `blockNote`. Sellers cache in blocks of a size each one chooses, so two equal rates per million tokens at one retention window need not buy the same caching. No other meter carries the field. See [what a cache write can and cannot be set against](../../../concepts/dimension.md#what-a-cache-write-can-and-cannot-be-set-against). ## Errors | Status | Code | Cause | |---|---|---| | 404 | `unknown_class` | Not a registered class | | 404 | `unknown_product` | No product by that slug in the class | | 404 | `unknown_dimension` | Product exists, is not metered that way. The message lists the ones it has. | ## Two paths, one shape With a database the answer comes from the SQL view; without one it is computed from the committed snapshot. The shapes are identical by design, so the site and the API cannot disagree about a figure. `meta.source` is the only way to tell which path answered. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/tape.md --> # Tape Every current price in one class, flattened to one row per priced cell. ``` GET /api/v1/markets/{class}/tape ``` ```bash curl https://vitna.ai/api/v1/markets/model/tape ``` ## Path parameters | Parameter | Values | |---|---| | `class` | `model`, `tooling`, `retrieval`, `compute`, `extraction`, `speech`, `translation`, `image`, `video`, `music` | Anything else answers `404 unknown_class`. ## Response ```json { "class": "model", "tape": [ { "sku": "gpt-5-pro", "name": "gpt-5-pro", "provider": "OpenAI", "market": "First-party", "featured": true, "dimension": "input_per_mtok", "dimensionLabel": "Input price", "amountUsd": 15, "unit": "USD per 1M tokens", "since": "2026-08-07T00:00:00.000Z", "confirmedAt": "2026-08-23T08:00:11.402Z", "fetchedAt": "2026-08-23T08:00:11.402Z", "stale": false, "derived": false, "sourceUrl": "https://openai.com/api/pricing/", "evidenceSha256": "9f2c..." } ], "meta": { "generatedAt": "...", "source": "database", "count": 918, "units": ["USD per 1M tokens"] } } ``` ## Fields | Field | Meaning | |---|---| | `sku` | Slug, for `/skus/{slug}/history` | | `provider` | Who sells it | | `dimension` | What it is metered by | | `amountUsd` | The price, in `unit` | | `since` | When this meter's price started, in this band: the `since` of its open period in `/skus/{slug}/history`, not the oldest date across the sku | | `confirmedAt` | When it was last confirmed | | `fetchedAt` | When the document was read | | `stale` | Past its source's cadence | | `derived` | Computed by Vitna rather than read from the page | | `derivation` | How, when `derived` is true | | `sourceUrl` | The page a human should read | | `evidenceSha256` | The archived document | | `observationId` | The leaf this figure is, for [its proof](../verification/proof.md) | Model rows additionally carry `market` and `featured`. A derived row carries `derivation` in every class. Compute rows carry the key on every row, as `null` where the figure was read as printed; every other class leaves it out. A model row served from the committed snapshot (`meta.source: "snapshot"`) holds no read per figure, so a derived figure there says that it was derived and not how. A row priced in a [context band](#context-bands) also carries `bandFromTokens`, `bandToTokens` and `bandLabel`. ## The leaf behind a figure `observationId` names the [observation](../../../concepts/observation.md) this figure is, which is what turns a number on the board into something you can prove is in the signed record: ```bash curl "https://vitna.ai/api/v1/observations/<observationId>/proof" ``` It is the observation that **last confirmed** the price, which is the same read `confirmedAt` and `evidenceSha256` describe. All three on one row therefore refer to one read, so checking the document and checking the leaf are checks of the same thing. `fetchedAt`, `stale`, `sourceUrl` and `derived` are that read's too. That holds per row, not per sku. One model's figures can come from different reads: OpenAI's and Anthropic's flex and priority rates come from a second source, the tiers page, read in a fetch of its own, and a table a page stops printing keeps its periods open on the last read that saw it while the rest of the row is confirmed every day. Each figure carries its own read, so a fresh input price is never called stale because its batch rate is, and a batch rate never cites a document it is not in. > [!NOTE] > Until 2026-11-16 the model tape copied one read onto every figure of a model: > the oldest of its reads for `confirmedAt` and `fetchedAt`, any of them for > `stale`, and the input price's document for `evidenceSha256`. On that day 100 > of 1,578 model figures carried a read that was not their own. Anything that > describes a whole model in one line, a board row on the site or an entry in > the [LiteLLM cost map](litellm-cost-map.md), still gives the oldest of its > reads and is stale if any of its figures is, which is the right answer for > one line and the wrong one for a figure. One id per row of the tape, not per sku. A single fetch writes one observation per cell it reads, so a model priced on four meters has four leaves even though one document covers them all. That is why the tape carries the id and the board does not. > [!NOTE] > `observationId` is `null` when an instance is serving a committed snapshot > written before the ids were carried. Null rather than a borrowed id: inventing > one from the row's other identifiers would produce a proof for a figure nobody > asked about. The live record answers with the id. Ids are strings. They are `bigserial`, and a JSON number stops being exact above 2^53. ## Context bands Some providers price one model differently by how much context a request carries. A price therefore applies in a band rather than universally, and a sku priced that way publishes **one entry per band per meter**. | Field | Meaning | |---|---| | `bandFromTokens` | Exclusive lower bound | | `bandToTokens` | Upper bound, `null` when unbounded | | `bandLabel` | What the provider calls it | So a model split at 272,000 tokens publishes an input price for `(0, 272000]` and another for `(272000, null]`, each naming its own band. > [!NOTE] > A sku priced once carries **no band keys at all**, rather than a band of > `(0, null]`. That is deliberate: a consumer written before bands existed reads > exactly the same rows for exactly the same skus, so the addition breaks > nothing already parsing this tape. A band is not a [dimension](../../../concepts/dimension.md) and not a [sku](../../../concepts/sku.md). It is a property of the price, which is why it travels on the row rather than multiplying either registry. ## One row per cell A model publishing input, cached input and output prices produces three rows. This is the flat shape on purpose: it is joinable and countable, where the board shape the site renders is not. > [!WARNING] > A cell a provider does not publish is absent, not zero. Do not read a missing > `batch_input_per_mtok` as free batching. See > [dimension](../../../concepts/dimension.md). ## Empty classes answer 200 with an empty tape A registered class with nothing priced returns `"tape": []`, which is the honest answer and the same one the site gives. A `404` would say the class does not exist, which is different and untrue. An empty tape means one of two things, and [`/markets`](markets.md) says which: a class Vitna does not read yet (`wired: false`), or a class it reads whose sellers price nothing today (`wired: true`). `video` is the second kind since Google shut down Veo 3.1 Preview on 2026-10-22, which took every figure the class held. ## `meta.units` The distinct units present, which is more than one for most classes. The speech tape carries two units across three meters, since transcription is billed by the audio-minute pre-recorded and again streaming, and synthesis by the million characters. The retrieval and extraction tapes are single-unit. The tooling tape is metered four ways. The model tape carries two, because a model that emits images is billed by the image rather than by the token. An extraction row carries one field no other tape has, `page`, which says which kind of page its figure is for. Two per-page rates are not comparable without it: the cheapest is a page read for its characters and the dearest is a page read against a schema. > [!WARNING] > Read `unit` off the row, never off the class. Within one model tape a row may > be quoted per million tokens and the next per image, and treating the first > unit you see as the tape's unit is how a per-image price gets compared against > a per-token one. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/prices/terms.md --> # Provider terms What providers have published on their own domains about how they handle customer work: retention, training use, processing regions, and bring-your-own-key support. The read surface for [provider terms](../../../concepts/provider-term.md). ``` GET /api/v1/terms GET /api/v1/providers/{id}/terms ``` ```bash curl https://vitna.ai/api/v1/terms curl "https://vitna.ai/api/v1/terms?provider=anthropic&kind=zero_retention" curl https://vitna.ai/api/v1/providers/openai/terms ``` ## Response ```json { "standing": [ { "id": 5, "providerId": "openai", "provider": "OpenAI", "kind": "zero_retention", "value": "no", "observedOn": "2026-08-01", "sourceUrl": "https://openai.com/enterprise-privacy", "sourceVerifiedOn": "2026-08-01", "note": null, "summary": "retains prompts or responses, as read on 2026-08-01" } ], "terms": [ { "id": 5, "providerId": "openai", "provider": "OpenAI", "kind": "zero_retention", "value": "no", "observedOn": "2026-08-01", "sourceUrl": "https://openai.com/enterprise-privacy", "sourceVerifiedOn": "2026-08-01", "note": null, "summary": "retains prompts or responses, as read on 2026-08-01" } ], "kinds": { "zero_retention": "whether the provider retains prompts or responses after answering", "no_training": "whether the provider trains on customer data", "regions": "which regions the work is processed in", "byok": "whether the provider accepts a key the customer holds" }, "meta": { "generatedAt": "2026-09-12T09:14:02.881Z", "source": "database", "count": 4, "standingCount": 3 } } ``` ## Standing vs history `witness.provider_term` is append-only: when a provider changes its retention or training policy, a new row is recorded and the earlier row remains so older decisions still resolve against what stood on the day they were made. - `standing` holds the newest reading (`observedOn`) for each `(providerId, kind)` pair, which is what the engine's eligibility check reads today. - `terms` holds every recorded reading matching the filter, newest first. When served from the committed snapshot with no database connection, `standing` and `terms` are `null` with an explanatory `note` rather than `[]`, so an offline checkout never implies that the record holds zero terms. ## Parameters | Parameter | Values | Default | |---|---|---| | `provider` | Any registered provider id (for `/api/v1/terms`) | All providers | | `kind` | `zero_retention`, `no_training`, `regions`, `byok` | All four kinds | An unknown provider answers `404 unknown_provider`; an unknown kind answers `400 bad_term_kind`. Both are validated before touching the database. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/status/health.md --> # Health Whether this instance is up, and what it found when it reached for the database. ``` GET /api/v1/health ``` ```bash curl https://vitna.ai/api/v1/health ``` ```json { "ok": true, "db": "ok", "commit": "8312c96eefc323220963a6098440d4f57a9ceb0f", "latencyMs": 3, "at": "2026-08-23T09:14:02.881Z" } ``` ## Fields | Field | Meaning | |---|---| | `ok` | `true` whenever the process answered, including with a `503` | | `db` | `ok`, `unset` or `error` | | `commit` | The revision this process was built from, or `null` where nothing recorded one | | `latencyMs` | Round trip of `select 1`, or `null` when no pool is configured | | `at` | RFC 3339 timestamp | | `detail` | Present only when `db` is `error`. A short reason. | ## `db` states | State | Meaning | |---|---| | `ok` | A connection opened and `select 1` returned | | `unset` | No `DATABASE_URL`. Serving the committed snapshot. | | `error` | Configured but unreachable. `detail` says what happened. | ## `commit` is what the deploy checks The deploy hook records the revision it restarted before it restarts, so this is the commit the running process was built from and not the one the box's bare repository points at. Those two disagree exactly when a deploy did not take: the hook aborts before the restart, so the previous build keeps serving, and it serves healthily. `null` means nothing recorded a revision, which is a laptop or a box whose hook predates the field. It is deliberately not a guess. A caller that cannot be told which revision is answering is told that, rather than a plausible-looking sha it might act on. ## Liveness and readiness Use `ok` for liveness and the status for readiness, and point a health check or an uptime monitor at the status. | Status | When | |---|---| | `200` | `db` is `ok` or `unset` | | `503` | `db` is `error`: a database is configured and `select 1` did not return | `ok` is `true` whenever the process answered, a `503` included. `db` names the state behind the status. ## A 503 still names the state The body is the same either way, so a `503` still says what it found and, in `detail`, what the connection reported. That is why this endpoint once answered `200` whatever the database was doing: a health check that fails with a stack trace tells you less than one that names the state. The cost fell on everything that reads only the status, uptime monitors among them, and to those a database that had stopped answering looked like a healthy site. A `503` is also the rule the rest of this API keeps: an endpoint that cannot reach the record answers `503` ([overview](../overview.md), "503 rather than an empty list"). `unset` answers `200`. With no `DATABASE_URL` the instance serves the committed snapshot, which is a mode rather than an outage. An instance that should have a database and has lost its `DATABASE_URL` reads the same, so a check that needs the database to have answered reads `db` and wants `ok`. ## Credentials are stripped from `detail` Connection errors quote the target they failed to reach, and that target can carry a password. The handler removes anything between `//` and `@` and truncates to 120 characters before returning. ## Caching `no-store`. This endpoint answers for the moment it is asked or not at all. Unlike every other v1 endpoint, this one does not use the shared envelope: it has no `meta` and carries no `ETag`. --- <!-- https://vitna.ai/docs/api/v1/status/sources.md --> # Source health Every [source](../../../concepts/source.md) Vitna reads, and whether it is still answering. ``` GET /api/v1/sources/health ``` ```bash curl https://vitna.ai/api/v1/sources/health ``` ## Response ```json { "sources": [ { "source": "...", "provider": "...", "enabled": true, "stale": false, "consecutiveFailures": 0, "lastFetchAt": "...", "lastOkAt": "...", "cadenceMinutes": 60 } ], "meta": { "generatedAt": "...", "source": "database", "count": 21, "staleCount": 0, "failingCount": 0, "ok": true } } ``` `meta.ok` is the one-glance answer: true when nothing enabled is stale and nothing enabled has failed three times running. | `meta` field | Meaning | |---|---| | `staleCount` | Enabled sources past their cadence | | `failingCount` | Enabled sources at three or more consecutive failures | | `ok` | Both counts are zero | A source that has been switched off is still listed, with `enabled: false` and the failure count it stopped at, and counts toward neither. It is no longer read, so that count can never move again, and [/witness](https://vitna.ai/witness) shows the source as held rather than failing. ## Public on purpose A red source is part of the record. A tape that silently stopped updating because a pricing page moved looks exactly like a market that stopped moving. Publishing source health is what lets a reader tell those apart without asking anyone. ## 503 rather than a green light If the record cannot be read, this answers `503 record_unavailable`. It never falls back to the committed snapshot. A health endpoint reporting green from a file on disk would be the one lie this whole system exists to prevent, and it would be reporting on sources it had not looked at. ## Errors | Status | Code | Cause | |---|---|---| | 503 | `record_unavailable` | The record is not reachable from this instance | ## Caching `no-store`. A cached green is a claim about the past. --- <!-- https://vitna.ai/docs/api/v1/verification/attestations.md --> # Attestations The signed daily Merkle roots, newest first, with everything a check needs. ``` GET /api/v1/attestations ``` ```bash curl https://vitna.ai/api/v1/attestations ``` ## Response ```json { "attestations": [ { "coversOn": "2026-08-22", "root": "...", "statement": "...", "signature": "...", "publicKey": "...", "leafCount": 918, "signedAt": "..." } ], "algorithm": "...", "statementPrefix": "...", "howToVerify": [ "Rebuild each observation's leaf as sha256(0x00 || canonical JSON) with the field order in platform/witness/attest.mjs.", "Hash pairs as sha256(0x01 || left || right), promoting an odd node unchanged, until one root remains.", "Check that root against this row, then verify the signature over `statement` with `publicKey`, Ed25519." ], "proves": "The record for that day has not changed since it was signed.", "doesNotProve": "That the provider published the price. ...", "meta": { "generatedAt": "...", "source": "database", "count": 3 } } ``` ## `proves` and `doesNotProve` ship in the payload Both sentences are in the response rather than only in documentation, because this is the claim most easily overstated by someone quoting the record. A signature proves the record for that day has not changed since it was signed. It does not prove the provider published the price: no provider signs its own pricing page. What ties a figure to a provider is the archived document, which records what Vitna fetched. ## Getting from one figure to a root This endpoint publishes roots. [`/observations/{id}/proof`](proof.md) publishes the path from a single observation up to one, which is what turns a root from a claim into something checkable. ## Serving the snapshot Answers `200` with `attestations: null`, the algorithm and statement prefix still present, and a note pointing at the live record. `null` rather than `[]`, for the same reason as everywhere else: an empty list would read as "nothing was ever signed". ## Parameters None. ## Caching `public, max-age=60, stale-while-revalidate=3600`. The endpoint is published in full because a root nobody can read attests to nothing. The mechanism is that the statement was in the open on the day, so an edit made later has to contradict something already published. --- <!-- https://vitna.ai/docs/api/v1/verification/citations.md --> # Citations One [observation](../../../concepts/observation.md) in the three paste-ready citation forms `/cite/{id}` prints (`reference`, `footnote`, `schedule`), alongside the figure's provenance fields and signed-day status. See [citing a figure](../../../citing-a-figure.md) for the permanence rules that govern these URLs. ``` GET /api/v1/citations/{id} ``` ```bash curl https://vitna.ai/api/v1/citations/445935 ``` Take `{id}` from `observationId` on any [tape](../prices/tape.md) row or period in [SKU history](../prices/sku-history.md). ## Response ```json { "observationId": "445935", "url": "https://vitna.ai/cite/445935", "proofUrl": "https://vitna.ai/api/v1/observations/445935/proof", "promiseUrl": "https://vitna.ai/docs/citing-a-figure", "verifyUrl": "https://vitna.ai/docs/verify-a-figure", "headline": "OpenAI GPT-5.5, input price, $1.25 per 1M tokens", "figure": { "observationId": "445935", "provider": "OpenAI", "sku": "GPT-5.5", "skuSlug": "gpt-5.5", "skuStatus": "active", "classId": "model", "dimensionId": "input_per_mtok", "dimension": "Input price", "unit": "USD per 1M tokens", "amountUsd": 1.25, "band": null, "readAt": "2026-09-12T00:00:00.000Z", "fetchedAt": "2026-09-12T08:00:11.402Z", "source": "openai-pricing-md", "adapter": "openai-pricing", "sourceUrl": "https://developers.openai.com/api/docs/pricing", "evidenceSha256": "4f2a...", "derived": false, "derivation": null, "locator": "table:row 0", "parserVersion": 3 }, "signedDay": { "state": "signed", "day": "2026-09-12", "label": "2026-09-12", "attestation": { "coversOn": "2026-09-12", "root": "a9c1...", "signature": "c2ln", "publicKey": "a2V5", "leafCount": 5, "algorithm": "sha256-merkle-rfc6962", "signedAt": "2026-09-13T04:00:00.000Z" } }, "forms": { "reference": "OpenAI GPT-5.5, input price, $1.25 per 1M tokens...", "footnote": "OpenAI GPT-5.5, input price: $1.25 per 1M tokens...", "schedule": "Rate: $1.25 per 1M tokens..." }, "formDescriptions": [ { "id": "reference", "label": "Short reference", "note": "For a slide, a chart footer or anywhere the line has to fit under something else." } ], "proves": "Vitna recorded these observations on this UTC day and has not altered them since signing.", "boundary": "No provider signs its own pricing page: the signature covers Vitna's record of the fetch and not the provider's page.", "meta": { "generatedAt": "2026-09-13T09:14:02.881Z", "source": "database" } } ``` ## Snapshot behaviour Citations resolve strictly from the database because the committed JSON snapshot carries no observation IDs. On a snapshot checkout without `DATABASE_URL`, this route answers `503 no_record` pointing to `https://vitna.ai/api/v1/citations/{id}`, matching [observation proof](proof.md). An `{id}` that is not a positive integer or does not exist in `witness.observation` answers `404 unknown_observation`. ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/api/v1/verification/evidence.md --> # Evidence The record of one archived document, and behind a token, the document itself. ``` GET /api/v1/evidence/{sha256} ``` ```bash curl https://vitna.ai/api/v1/evidence/9f2c... curl -H 'Authorization: Bearer <token>' "https://vitna.ai/api/v1/evidence/9f2c...?body=1" ``` ## Path parameters | Parameter | Meaning | |---|---| | `sha256` | 64 lowercase hex characters | Anything else answers `404 unknown_evidence` without touching the archive. ## The public answer ```json { "sha256": "9f2c...", "archived": { "url": "https://...", "fetchedAt": "...", "httpStatus": 200, "headers": { ... }, "userAgent": "...", "adapter": "...", "parserVersion": 3, "bytes": 48211, "via": null, "robots": null }, "reads": [ ... ], "body": { "available": true, "url": "/api/v1/evidence/9f2c...?body=1", "access": "Bearer token. ..." }, "meta": { "generatedAt": "...", "source": "archive" } } ``` `archived` is the read that first stored this document. `reads` is every fetch that has since found the same bytes, which is how you see that a page has been unchanged for weeks. ## Why the body is gated The public half is what a verification needs. Given the URL and the fetch time, you can fetch the page yourself and compare hashes; Vitna does not need to be trusted for that to work. The stored copy is a third party's page, and republishing it raises a copyright question this record does not need to answer in order to be checkable. ## The body response Not JSON. The stored bytes, with the archived `content-type`, plus: | Header | Meaning | |---|---| | `x-vitna-fetched-at` | When it was read | | `x-vitna-source-url` | What was read | | `x-vitna-sha256` | The key | It is sent with `no-transform`, so intermediaries such as Cloudflare pass the bytes through unaltered and the body hashes to the key. ## Errors | Status | Code | Cause | |---|---|---| | 404 | `unknown_evidence` | Malformed hash, or nothing archived under it | | 401 | `unauthorized` | `?body=1` without a valid bearer token | The `401` message restates that the hash, URL and fetch time are public and are what a verification needs, so a caller without a token knows they are not blocked from checking anything. ## Caching `no-store`. Both the record and the body; the body adds `no-transform`. --- <!-- https://vitna.ai/docs/api/v1/verification/keys.md --> # Signing keys Every key the record acknowledges, what each one is for, whether it has been retired, and how much it provably signed. ``` GET /api/v1/keys GET /api/v1/keys/policy-key.pub ``` ```bash curl https://vitna.ai/api/v1/keys curl -s https://vitna.ai/api/v1/keys/policy-key.pub ``` ## Response ```json { "keys": [ { "purpose": "record", "algorithm": "ed25519", "publicKey": "MCowBQYDK2VwAyEA...", "fingerprint": "5d0c3f...", "enteredServiceAt": "2026-08-01T04:00:00.000Z", "retiredAt": null, "retiredReason": null, "signed": { "attestations": 43, "artifacts": 0 } }, { "purpose": "policy", "algorithm": "ed25519", "publicKey": "MCowBQYDK2VwAyEA...", "fingerprint": "a81e07...", "enteredServiceAt": "2026-09-02T12:00:00.000Z", "retiredAt": null, "retiredReason": null, "signed": { "attestations": 0, "artifacts": 7 } } ], "policyKey": "MCowBQYDK2VwAyEA...", "policyKeyFile": "/api/v1/keys/policy-key.pub", "note": "policyKey is the value a runtime pins as VITNA_PUBLIC_KEY. A retired key stays listed with its date and reason, because a file it signed is still checkable against it.", "meta": { "generatedAt": "...", "source": "database", "count": 2 } } ``` Oldest first. Retired keys are listed, with the date and the reason, because a file signed last year by a key since retired is still a file Vitna signed. ## Fields | Field | Meaning | |---|---| | `purpose` | `record` signs each day's Merkle root; `policy` signs the configs and policies emitted for customers | | `algorithm` | Read off the key itself. `null` for a row that does not parse as a key | | `publicKey` | Base64 SPKI DER, the shape `VITNA_PUBLIC_KEY` pins | | `fingerprint` | Lowercase hex sha256 of the DER bytes: decode `publicKey` from base64 and hash what comes out | | `enteredServiceAt` | When the key was registered, or when it first signed a root for a `record` key nobody registered | | `retiredAt`, `retiredReason` | Both `null` while the key is in service. The reason is the operator's text, published as written | | `signed.attestations` | Attestation rows in the record this key signed, all of them. [`/attestations`](attestations.md) serves the newest 30, so it checks a recent root's key rather than the whole count | | `signed.artifacts` | Configs and policies this key signed for customers. Absent, and the note says so, where the instance cannot count them | | `policyKey` | The value a runtime pins, or `null` when no key is registered to sign policies | ## `policyKey` is what a runtime pins, and it can be two keys Ordinarily one key. During a changeover two policy keys are current at once, and `policyKey` lists both, comma separated in the order they entered service, which `VITNA_PUBLIC_KEY` accepts as a set. The runbook for that is [rotating a signing key](../../../operations/key-rotation.md). A `record` key is never offered here. It signs the public record's roots, and a runtime that pinned it would be trusting the wrong signer. ## No policy key is an answer A register with no key registered to sign policies answers `200` with `policyKey: null` and a `note` saying so. It is not a `404`: "there is no policy key yet" is the fact somebody about to pin one needs, and a `404` reads as a wrong address. ## `policy-key.pub` The same `policyKey`, as one line of `text/plain`, so it can be read straight into the variable: ```bash VITNA_PUBLIC_KEY="$(curl -s https://vitna.ai/api/v1/keys/policy-key.pub)" ``` When no key is registered to sign policies it answers `404` with the code `no_policy_key` in the error envelope, because a file has nothing to be when there is no key. Fetching the key you will verify against from the signer's own site is a weaker out-of-band channel than reading it off `/console/deployment`, where each signed policy prints the key that signed it. Compare the two, or compare the `fingerprint` with one somebody at Vitna read you, before you trust either. ## Serving the snapshot Answers `200` with `keys: null`, `policyKey: null` and a note pointing at the live record, the way [`/attestations`](attestations.md) does. `null` rather than `[]`, because an empty list would read as "the record has no keys". `policy-key.pub` answers `503` with `no_record`. ## Caching `public, max-age=60`, for both, with no `stale-while-revalidate`. Most of v1 may serve an hour-old answer while it revalidates, and a key set cannot: a planned rotation gives days of notice, but a compromise retires a key at once, and a copy an hour old would still offer the retired key to pin. A retirement reaches every reader within the minute. --- <!-- https://vitna.ai/docs/api/v1/verification/proof.md --> # Observation proof The path from one observation up to a signed root. The other half of the [attestation](attestations.md). ``` GET /api/v1/observations/{id}/proof ``` ```bash curl https://vitna.ai/api/v1/observations/1/proof ``` ## Path parameters | Parameter | Meaning | |---|---| | `id` | An [observation](../../../concepts/observation.md) id. Digits only. | ## Response ```json { "leaf": { "id": "445937", "fetchId": "20411", "sourceId": "...", "contentSha256": "...", "skuId": "...", "dimensionId": "input_per_mtok", "amountUsd": "15.000000", "observedAt": "...", "derived": false, "derivation": null, "locator": "...", "parserVersion": 3, "bandFromTokens": 0, "bandToTokens": null }, "canonicalLeaf": "{\"id\":\"445937\",\"fetchId\":\"20411\",...}", "leafHash": "...", "leafIndex": 417, "coversOn": "2026-08-22", "leafCount": 918, "algorithm": "...", "statementPrefix": "...", "proof": [ { "side": "left", "hash": "..." } ], "computedRoot": "...", "attestation": { "root": "...", "statement": "...", "signature": "...", "publicKey": "..." }, "signedDay": { "state": "signed", "newestSignedDay": "2026-09-12" }, "unattested": null, "howToVerify": [ ... ], "proves": "This observation was among the ones the signed root covers for that day.", "doesNotProve": "That the provider published the price. ...", "meta": { "generatedAt": "...", "source": "database", "algorithm": "..." } } ``` `canonicalLeaf` is the exact string the leaf hash is computed over, and `leaf` is the same fields as an object, in the same order. The one difference is the band. `leaf` carries `bandFromTokens` and `bandToTokens` on every figure, as `0` and `null` on a figure priced once. `canonicalLeaf` carries them only on a figure priced in a context band, in that order, after `parserVersion`. ## The check Walked end to end, with the document check beside it, in [Verify a figure yourself](../../../verify-a-figure.md). 1. `leafHash = sha256(0x00 || canonicalLeaf)`, over the UTF-8 bytes of `canonicalLeaf` exactly as served. It must equal the `leafHash` here. 2. Check that `canonicalLeaf` says what `leaf` says: `leaf` written as JSON with the keys in the order shown, no whitespace, amounts as the strings given, and `bandFromTokens` and `bandToTokens` left out where they are `0` and `null`, is `canonicalLeaf` byte for byte. 3. Fold the path. For each step: `sha256(0x01 || hash || running)` when `side` is `left`, and `sha256(0x01 || running || hash)` when it is `right`, where `hash` is that step's own `hash`. 4. The result must equal `attestation.root`, and `attestation.statement` must verify against `attestation.publicKey` with Ed25519. Deliberately short. A verification nobody can reimplement in an afternoon is one nobody will. ## Nothing here has to be trusted A verifier rebuilds the leaf from the fields, hashes it, walks the path, and compares against a root inside a statement that somebody else's copy of the signature also covers. If this endpoint lied about any of it, the arithmetic would stop agreeing. ## `computedRoot` against `attestation.root` `computedRoot` is what the endpoint calculates from the record as it stands now. `attestation.root` is what was signed. They should be equal. Publishing both is what lets a reader see a disagreement rather than take Vitna's word that there is none. ## A day with no root Roots are signed the morning after a day closes. When no root covers `coversOn`, `attestation` is `null` and the endpoint still answers `200`, because a missing root is a true statement about the record and not an error. It is one of two different statements, though, and `signedDay.state` says which. `unattested` says the same thing as a sentence. | `signedDay.state` | When | `unattested` | |---|---|---| | `signed` | A root covers `coversOn` | `null` | | `pending` | `coversOn` is `newestSignedDay` or later, or no day has been signed | Not signed yet, provable once its root is cut | | `unsigned` | `coversOn` is before `newestSignedDay` and has no root | A gap, naming the newest signed day | `newestSignedDay` is the newest day the record has signed, or `null` if it has signed none. The state is decided against it and never against a clock, so the same record gives the same answer whenever it is asked. **Pending** is the ordinary absence. An observation from today sits in a day whose root has not been cut, and a figure read today is provable tomorrow. **Unsigned** is a gap. A later day has been signed and this one has not, and the nightly run signs only the day before, so the day is not waiting its turn. The sentence names the newest signed day and makes no promise about this one. The document the figure was read from, named by `contentSha256`, is a separate check and does not need a root. ## Errors | Status | Code | Cause | |---|---|---| | 404 | `unknown_observation` | Not digits, or no such observation | | 503 | `no_record` | Serving the snapshot. A proof cannot be built from it. | ## Caching `public, max-age=60, stale-while-revalidate=3600`. --- <!-- https://vitna.ai/docs/routing/overview.md --> # Routing How an accepted decision becomes a value a machine loads, and how the record finds out whether the change actually took. ## The gap this closes A [decision](../concepts/decision.md) produces a brief. A brief is a document, so somebody has to read it and then go and change a value somewhere else, and that second step is where a decision quietly fails to happen. Until now `aperture:accept` moved exactly one column, the proposal's status, and said plainly why it moved no others: whether traffic actually moved is not something Vitna can see. Two pieces close it from both ends: | Piece | Direction | |---|---| | [`aperture:config`](config.md) | Out. The decision rendered as a loadable value, signed. | | [`aperture:policy`](policy.md) | Out. The same decisions as a document a runtime executes, signed. | | [`aperture:sign`](signing.md) | Out. The same artifacts, built and signed from a queue rather than at a prompt. | | [`aperture:usage:import`](usage-import.md) | Back. What the consumer's own export says actually served the work. | The register those decisions are made from can be composed at `/console` as well as at a prompt: see [Composing the register](console-composing.md). ## Two ways out, one record A config is for a consumer that already reads its model ids from somewhere: a gateway, a variable, an existing platform. A policy is for one that does not, and would otherwise have nowhere to put a decision at all. They are emitted from the same accepted decisions and a test asserts they agree, so which one an organisation consumes is a question about its infrastructure rather than about what Vitna decided. ## What the router is not Worth being exact, because "router" invites the opposite assumption. **Nothing here reaches a provider or moves a token.** Vitna emits a file. A person reads the diff and applies it. **One key exists, and it is not a provider key.** An org can connect its OpenRouter account, which grants Vitna a key OpenRouter minted and that org can revoke at source. No provider credential is involved, and nothing Vitna runs calls a provider with it: [the runtime](../runtime/overview.md) a customer operates reaches providers with the operator's own keys, from the operator's own environment, and never sees this one. See [Connected accounts](connected-accounts.md), which is exact about what sealing that key buys and what it does not. **No runtime fetches its policy from Vitna.** That is deliberate and it is a boundary, not an omission: a consumer fetching its routing from Vitna at request time would have made Vitna a dependency of its own uptime. A runtime loads its policy from local disk at boot, which is what keeps that true even when Vitna's software is in the request path. An organisation does [collect its signed policy](../api/v1/engine/artifacts.md) between deployments, with its own key or as a download from the console, and no endpoint serves a config. Corrected 2026-09-13: this said no endpoint serves either file. **Nothing fetches usage either.** A person exports it from the console or database they already have, and the export arrives as a file with a `source_ref` on it, exactly as a measurement does. A runtime's operator can also schedule a push of its closed days to [`POST /api/v1/usage`](../api/v1/engine/usage.md) with an org key, which Vitna receives and never requests. So the loop is closed by a person at both ends, on purpose. What changes is that the record can now state what it told a workload to run and what the workload was observed running, rather than holding an intent and waiting. **And a hosted one is frozen, not operated.** Every sentence above describes what runs. [The hosted router](router.md) works through what it would take for Vitna to operate one without ever holding a provider key. Its software was built on 2026-08-31 on the owner's instruction and frozen on 2026-09-09; it is not installed on Vitna's box, and no phase is scheduled to operate it. Corrected 2026-09-13: this said none of it was built. It is published because a custody design that nobody outside can read is a custody design nobody outside can refuse. ## The pieces | Page | What it covers | |---|---| | [Delivery targets](delivery.md) | Where a workload's decision gets applied | | [Emitting a config](config.md) | `aperture:config`, and the eight ways it refuses | | [Emitting a policy](policy.md) | `aperture:policy`, the document a runtime loads | | [Signing without an operator](signing.md) | `aperture:sign`, the queue that answers an ask for an artifact | | [Collecting a signed artifact](../api/v1/engine/artifacts.md) | How an organisation takes the bytes away, with its own key | | [Fallback chains](fallbacks.md) | What may serve a lane when the decided model cannot | | [Rules within a lane](rules.md) | Declared per-request predicates, shadow first | | [Rollout and decision watchdog](watchdog.md) | Standing-decision drift and shadow-to-canary ladder readiness | | [Multi-class unit profiles and compound pipelines](pipelines.md) | Cross-layer Bill of Materials, `x-vitna-run` attribution, and two-tier spend optimization | | [Audit and procurement attestation packs](audit-packs.md) | Six-layer offline-verifiable `vitna.audit-pack.v1` governance, pricing, and FinOps bundles | | [Rotating the seller](sellers.md) | Several hosts of the same weights, under the decision that already stands | | [The runtime](../runtime/overview.md) | The process that applies a policy per request | | [Per-request routing](../runtime/route.md) | Deterministic `vitna-route` per-request model and effort selection under a signed policy | | [The hosted router](router.md) | The design, the credential ladder it turns on, and what of it now runs | | [Running the hosted router](hosted.md) | `router:start` and `router:trail`, multi-tenant serving under C1 | | [Model ids](model-ids.md) | Why a registered sku can still be unnameable | | [Importing usage](usage-import.md) | `aperture:usage:import`, windows and shares | | [Did the decision do what it said](outcome.md) | What was predicted, against what the export observed | | [Connected accounts](connected-accounts.md) | OAuth PKCE, and why no provider key is collected | | [Ruling from the console](console-ruling.md) | Accepting or refusing a proposal at /console, and the fifth role that may | | [Verifying a config](verifying.md) | Checking a file you were handed | --- <!-- https://vitna.ai/docs/routing/adapters.md --> # Rendering a policy for a gateway you already run A signed policy is a document a Vitna runtime executes. An adapter renders that same document as the configuration of a gateway you run instead, so one accepted decision configures whatever is already in your request path rather than asking you to adopt something new. ```bash VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target litellm --policy policy.json --out litellm.yaml ``` Add `--via openrouter` to write the same policy for a gateway holding one OpenRouter key rather than a key per provider. Written targets: `litellm`, `cloudflare`, `portkey` and `openrouter`. Vercel, the sixth in [#111](https://github.com/convexityos/vitna/issues/111), is refused rather than unbuilt, and the section at the end says why. ## An adapter reads the policy, never the register This is the property that makes six targets one product rather than six products. Every adapter takes the signed policy as its only input, so no adapter can disagree with what was decided: it never re-reads what the decision read. It follows that an adapter needs no database, no credentials and no network. It reads a file and writes a file. ## The key is required, and for the runtime's reason `VITNA_PUBLIC_KEY` pins the key the policy must carry. Without it the command refuses to run. A policy that verifies against its own key proves only that the file agrees with itself, and anybody can produce a file that agrees with itself. What this command writes is a configuration that will route production traffic, so it is pinned exactly as [a runtime pins it](../runtime/serving.md): take the key the record publishes on [/witness](https://vitna.ai/witness) once, out of band. ## Nothing is applied The command writes a file. Getting it to the gateway is yours, whether that is a commit somebody reviews or a deployment you already run. That is the architecture rather than a missing feature. A runtime that fetched its routing from vitna.ai at request time would have made Vitna's availability a term in its own, and the same is true of a gateway. What Vitna emits is a document; when it takes effect is your decision, made in your repository. ## What no adapter carries: rules Each target below refuses a routing rule, and each names its own nearest miss. The reason is the same one four times, so it is worth reading once here rather than as four separate limitations. **A rule is measured, not just executed.** A rule is registered in shadow, which means it logs what it *would* have served as two fields on the ledger row of what actually ran. It is promoted to routing only under an accepted decision whose brief cites those shadow readings by window. The ladder is ``` watching -> shadow -> 5% canary -> 25% -> accepted incumbent ``` and every step after the first depends on readings from the step before. **A gateway has no ledger**, so it produces no readings. That is why a routing rule is refused rather than approximated, and why a shadow rule is dropped rather than rendered: not because the gateways lack a routing feature, but because they are not instruments. Rendering a Vitna share onto a `weight`, a `percentage` or a `loadbalance` would produce a canary that routes and can never graduate, because the evidence its promotion requires would never exist. **So what an adapter carries is decisions, and what it does not carry is the apparatus for earning the next one.** Every accepted decision reaches your gateway with its chain, in order, naming the decision that put it there. If you want the ladder, the path is not to adopt a Vitna runtime permanently. It is to run one for the measurement period, let the shadow rows accumulate, accept the decision those readings support, and hand that decision to the gateway you already run. The runtime is how a decision is earned; an adapter is how one is run. ## What OpenRouter presets can carry, and what they cannot ```bash VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target openrouter --policy policy.json --out presets.json ``` A [preset](https://openrouter.ai/docs/features/presets) holds the config-shaped fields of a request, and a lane's chain is [an array in priority order](https://openrouter.ai/docs/guides/routing/model-fallbacks): `{"models": ["anthropic/claude-opus-4.8", "openai/gpt-5.6-luna"]}`, tried in order, the next one used when the previous returns an error. There is no separate primary field: the primary is the first element. Formats read 2026-08-29. | Policy | OpenRouter | Carried | |---|---|---| | Lane alias | the preset slug, called as `@preset/<slug>` | Yes | | The decided model | first element of `models` | Yes | | Fallback chain | the rest of `models`, in order | Yes, in the policy's order | | A shadow rule | nothing | Dropped, and named in the file | | A routing rule | nothing | **Refused** | | Which decision put it there | nothing that persists | Beside the body, not inside it | **This target needs no arguments at all**, and that is the policy's doing. Portkey needs `--provider` because a target names an integration only you have. OpenRouter names models in its own public namespace, and the policy already carries that name per lane and per chain entry. That field exists precisely because Vitna registers `claude-opus-4-8` and OpenRouter lists `claude-opus-4.8`: an adapter deriving one from the other would fail on the dot. **There is no direct route, and asking for one is refused.** Every other adapter picks between two names because the gateway could call a provider itself or go through OpenRouter. This gateway is OpenRouter, so the choice does not exist. **A sku OpenRouter does not list makes the lane impossible rather than merely unrouted.** Elsewhere an empty name costs the substrate file and the direct one still renders. Here there is no other file. **Applying one is the strangest of the four.** A preset is created and versioned by POSTing to an inference route, `POST /api/v1/presets/{slug}/chat/completions`, with a body shaped like a request. OpenRouter persists the fields that overlap with the preset config and silently ignores the transient ones (`messages`, `input`, `prompt`, `stream`). An existing slug gains a new active version. So the file this emits is not a document stored as written: it is a request body whose persistent half becomes the preset. Nothing transient is included, because a `messages` array in a generated file is one somebody would reasonably expect to be sent. **Applying it is a one-way door, and this is the only target where that is true.** A preset cannot be removed or disabled through the API. Checked 2026-08-30 against a real account: `DELETE` by slug, `DELETE` by id, `PATCH` and `PUT` all answer `404`, and after the first of those the preset was still listed. OpenRouter's published spec documents no such route, so unlike the two places this file records a vendor disagreeing with its own schema, here the schema's silence was accurate. The lifecycle exists and is simply not public: a preset carries a `status` whose documented values are `active`, `disabled` and `archived`. Removing one is a dashboard action. Two consequences for a target this file otherwise presents as CI-appliable. Versions accumulate, with no documented way to prune them. And a lane that is later decided away leaves its preset behind, still callable as `@preset/<slug>`, holding the last models it was given. So a policy that drops a lane does not retract the preset that lane created, and nothing in the emitted file can. Compare Cloudflare, where a generated route and the gateway holding it were both created and deleted over the API in the same session. **And deleting one burns its name, invisibly.** Removing a preset from the dashboard archives it rather than destroying it, and an archived preset keeps its slug permanently. What makes that worse than it sounds is that nothing shows you: `GET /presets` omits it and `GET /presets/{slug}` answers `404`, exactly as though the name were free. The only thing that reveals it is a create, which fails with `Cannot create a new version of a archived preset`. Measured 2026-08-30, on slugs deleted by hand earlier the same day. For this adapter that is a constraint on lane aliases rather than a curiosity. A preset slug is the lane alias, so a lane whose preset has ever been deleted cannot be rendered to this target again under the same name: the emitted file will be correct, the POST will fail, and no read beforehand would have warned you. Renaming the lane is the only way out, and a lane alias is not a free variable, it is the string a caller has in their code. **A routing rule refuses with no near miss to decline.** LiteLLM had `weight`, Cloudflare `percentage`, Portkey `loadbalance`, each a probabilistic thing worth naming. A preset has no field for a share of traffic and none that reads the size of a request. There is simply nowhere to put one, which is a cleaner refusal and no less total. Provenance is beside the body for a sharper reason than elsewhere too: an unknown key would be accepted and dropped rather than rejected. ## What Portkey can carry, and what it cannot ```bash VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target portkey --policy policy.json --out configs.json \ --provider anthropic=@anthropic-prod --provider openai=@openai-prod ``` A [Portkey config](https://portkey.ai/docs/product/ai-gateway/configs) is `{ strategy, targets }`, and targets under `strategy.mode: "fallback"` are tried in the order written. That is the closest fit to a lane any target has offered: LiteLLM needed a map beside its deployment list and Cloudflare needed a graph, and this is an ordered array that means what it looks like. Format read 2026-08-29. | Policy | Portkey | Carried | |---|---|---| | Lane alias | the config `name`, referenced by its saved id | Yes | | The decided model | `targets[].override_params.model` | Yes, by the route `--via` names | | Fallback chain | `targets` under `strategy.mode: "fallback"` | Yes, in the policy's order | | A shadow rule | nothing | Dropped, and named in the file | | A routing rule | nothing | **Refused** | | Which decision put it there | nothing in the schema | Beside the config, not inside it | **`--provider` is required, and it is the first thing an adapter has needed besides the policy.** A target names a provider integration saved in your Portkey account: `@openai-prod`, or whatever the person who set it up typed. A signed policy carries `openai`, and cannot know the rest. So the mapping is an argument, a provider with no mapping refuses the whole file and names every one that is missing, and a slug written without its leading `@` is refused rather than corrected. That widening is worth being precise about. It does not let the adapter disagree with the decision, which is what the contract protects: the map says how to address a provider the decision already named, in a namespace only you have. It is the same category as `--via`, which picks between two names the policy itself carries. The alternative was emitting `@anthropic` and hoping, and a config naming an integration that does not exist fails at the first request. **Applying it is one call under your own key**, unlike Cloudflare: `POST /v1/configs` creates one and `PUT /v1/configs/{slug}` versions it, and `GET /v1/configs/{slug}/versions` lists what came before. Post `name` and `config` as the request body; `vitna` is not part of it. The verb was corrected on 2026-08-29, from `PATCH /v1/configs/{id}`, against Portkey's published OpenAPI and its prose docs, which agree with each other and define no PATCH. **A routing rule refuses the whole file**, against a third nearest thing. Portkey's `loadbalance` mode splits across targets by `weight`, which is LiteLLM's mismatch again: a share is decided by cohort, not by a draw. No documented mode reads the size of a request, so `context_chars_over` has nothing to become either. ## What Cloudflare can carry, and what it cannot ```bash VITNA_PUBLIC_KEY=$(cat policy-key.pub) \ npm run aperture:adapter -- --target cloudflare --policy policy.json --out routes.json \ --timeout 60000 --retries 2 ``` A Cloudflare AI Gateway [dynamic route](https://developers.cloudflare.com/ai-gateway/features/dynamic-routing/) is a graph rather than a list: `{ id, name, elements }`, where each element is one of `start`, `conditional`, `percentage`, `rate`, `model`, `end` and they point at each other by `outputs.<name>.elementId`. Every route must start with a Start element, and a route is called by putting `dynamic/<name>` where the model name goes. Schema read from [the JSON configuration page](https://developers.cloudflare.com/ai-gateway/features/dynamic-routing/json-configuration/), retrieved 2026-08-29. | Policy | Cloudflare | Carried | |---|---|---| | Lane alias | the route `name`, called as `dynamic/<name>` | Yes | | The decided model | a `model` element's `properties.provider` and `.model` | Yes, direct route only | | Fallback chain | each model's `outputs.fallback`, pointing at the next | Yes, in the policy's order | | A shadow rule | nothing | Dropped, and named in the file | | A routing rule | nothing | **Refused** | | Which decision put it there | nothing in the schema | **Beside the route, not inside it** | **One route per lane, and the chain is the structure.** A model element carries `outputs.success` and `outputs.fallback` natively, so the primary points its fallback at the first entry, that entry points at the second, and each points success at the End. This is a better fit than LiteLLM's, where the chain lives in a map beside the deployment list: here there is no second place to write the order down, so it cannot drift. The last model falls back to the same End its success points at, because Cloudflare requires a `fallback` output on every model element and arriving at the End is what having nothing left to try looks like here. **`--timeout` and `--retries` are required, and this target is the second to need something beside the policy.** Cloudflare's model element requires both and documents no default for either: the JSON configuration page calls them "Request timeout in milliseconds" and "Number of retry attempts" and shows 60000 and 4 in an example, which is an example and not a default. A signed policy carries neither, and neither is harmless to invent. Retries interact with the chain: a non-zero value has the primary attempted several times over before the first fallback is reached at all, which is failover behaviour no decision decided. Cloudflare bounds it as "up to 5 attempts" without saying whether the initial call counts as one, so this file does not say either. Both are inputs, refused rather than defaulted, in the same category as Portkey's `--provider`: they say how to operate a route the decision already determined. **A routing rule refuses the whole file, and `percentage` is the trap.** That element routes requests *probabilistically*, which is the same mismatch as LiteLLM's `weight` wearing different clothes: a Vitna share is a deterministic cohort bucket. `conditional` reads request body, headers and metadata, and the expression language it reads them with is not documented on that page, so `context_chars_over` would be a guess at syntax. **A decision id cannot ride inside the route.** Cloudflare's model element documents `provider`, `model`, `timeout` and `retries` and says nothing about custom fields, so each route carries its decision under a `vitna` key beside `elements`, and the file says that key must be removed before posting. What that costs a Cloudflare user is a lookup rather than an answer: the decision is in the register either way. **Applying it is automated after all, and this file said otherwise until 2026-08-29.** Cloudflare's prose pages still say dynamic routing "is not currently available on the REST API", and two other things Cloudflare publishes contradict that sentence. Its API schema repository carries ``` POST /accounts/{account_id}/ai-gateway/gateways/{gateway_id}/routes ``` under the operation id `aig-config-post-gateway-dynamic-route`, summarised "Create a new AI Gateway Dynamic Route", with `PATCH`, `GET`, `DELETE` and `/versions` and `/deployments` beside it; and Cloudflare's official TypeScript SDK exposes exactly those paths as `client.aiGateway.dynamicRouting.*`. Both retrieved 2026-08-29 ([api-schemas](https://github.com/cloudflare/api-schemas), [SDK](https://github.com/cloudflare/cloudflare-typescript)). And the endpoint answers. A route generated by this adapter has been posted to a real gateway, read back and deployed, which settles it against the prose page rather than leaving three sources disagreeing on paper. The [section below](#cloudflare-which-rejected-what-this-repo-used-to-emit) records what that established and what it did not. So this is a target CI can apply, and the emitted file names the endpoint rather than sending the reader to the dashboard. ## What LiteLLM can carry, and what it cannot | Policy | LiteLLM | Carried | |---|---|---| | Lane alias | `model_list[].model_name` | Yes | | The decided model | `litellm_params.model` | Yes, by the route `--via` names | | Fallback chain | `router_settings.fallbacks`, which is attempted in order | Yes, in the policy's order | | A shadow rule | nothing | Dropped, and named in the file | | A routing rule | nothing | **Refused** | | Which decision put it there | `model_info`, returned by `GET /model/info` | Yes, per lane | Format read from [the config reference](https://docs.litellm.ai/docs/proxy/configs) and [the reliability page](https://docs.litellm.ai/docs/proxy/reliability), both retrieved 2026-08-29. ## LiteLLM exercised, not only read **This section is about LiteLLM alone.** It was the first target given to the software it configures, and for a while the only one. Cloudflare and OpenRouter have since been exercised too, separately and with different results, in [their own section below](#cloudflare-and-openrouter-exercised-and-portkey-not). Nothing here is evidence about them, or about Portkey, which has still met no account. Every LiteLLM row began as documentation too. On 2026-08-29 a generated config was fed to LiteLLM 1.98.0, and what that establishes is listed rather than summarised, because "it works" is the kind of claim this record does not make. | Checked | Result | |---|---| | The proxy starts on the config | `/health/readiness` answers `{"status":"healthy"}` | | The lane alias is the callable name | `/v1/models` lists `deep-research`, `classification` | | Provenance reaches the running gateway | `/model/info` returns every `vitna_` field | | A decision id stays a string | `"vitna_decision":"154"`, not `154` | | The chain is accepted as emitted | `Router.fallbacks` is `[{'deep-research': ['gpt-4o-mini']}]` | | The substrate form resolves | LiteLLM infers provider `openrouter` from `openrouter/anthropic/claude-3.5-sonnet` | **What it does not establish, which is the more important half.** No request was routed, no provider was called, and no failover was observed. Every check above is about a config the gateway accepts and reports back, not about a config that serves. Whether fallbacks fire in the written order under a real failure is still LiteLLM's documentation talking. It has since been answered for OpenRouter alone, in the section below, and answering it there says nothing about here: LiteLLM's chain lives in a different field of a different document and has never been made to fall through. One practical note for anybody repeating this on Windows: LiteLLM's startup banner cannot encode into cp1252 and the proxy exits before it binds. Set `PYTHONUTF8=1`. The failure is nothing to do with the config, and it looks like everything to do with it. ### A routing rule refuses the whole file Neither rule kind survives the translation, and reaching for the nearest LiteLLM feature would ship something that looks right and answers a different question. A `traffic_share` is not a `weight`. A weight is a draw per request; a Vitna share is `FNV-1a(cohort key) % 100 < share`, so the same caller lands the same way every time, promoting five percent to twenty five grows that arm instead of reshuffling it, and every serve can be recomputed from the ledger. A weight has none of those three properties. A `context_chars_over` is not a `context_window_fallbacks`. That fires when a provider returns a context-window error. A Vitna threshold fires on a number you chose, before anybody is called. One is a recovery and the other is a routing decision. The refusal costs the whole file rather than the one lane, on the rule [emitting a policy](policy.md) already holds: a partial configuration leaves whatever the gateway already had for the lanes it omits, which is a routing state nobody decided. ### A shadow rule is dropped, and the file says so A shadow rule routes nothing, so a configuration without it behaves identically and refusing would be false strictness. What is lost is not behaviour but evidence. The would-haves a promotion has to cite are recorded by a Vitna runtime and by nothing else, so a lane whose canary is being measured should not be run through an adapter yet. The generated file names each dropped rule where whoever is reading the configuration will see it. ## Cloudflare and OpenRouter exercised, and Portkey not Three of the four targets have now been given to the gateway they configure. Portkey has not, and says so below rather than going quiet. Two of these found real defects, and they found opposite ones, which is the reason each was posted rather than reasoned about. **A published schema is not the service.** Cloudflare's understates nothing and its API rejected what this repo was emitting; OpenRouter's overstates a requirement its API does not enforce. Reading either one alone would have produced a confident wrong answer, in a different direction each time. All of the below was done 2026-08-29. ### Cloudflare, which rejected what this repo used to emit The prose page says dynamic routing "is not currently available on the REST API". A scratch gateway was created, routes were posted into it, and it was deleted afterwards. | Checked | Result | |---|---| | Does the REST surface answer | Yes. `GET .../gateways/<any>/routes` returns a `200` with a paginated envelope, where a path that genuinely does not exist returns `404 "Route not found"` | | **The shape this adapter emitted before today** | **Rejected, `400`.** Cloudflare named five paths: `properties.timeout` and `properties.retries` on both model elements, and `outputs.fallback` on the last one | | The shape it emits now | Accepted, `200` | | Does the chain survive | Yes. Read back with the primary's `fallback` pointing at the chain entry, that entry's at the End, and both `success` outputs at the End | | Do the operator numbers survive | Yes, `"timeout":60000,"retries":2` on every model element | | A lane with no chain | Accepted | | Versions and deployments | Both answer. The route was created with an `active` version and an auto-created deployment naming that `version_id` | So this adapter had been emitting routes Cloudflare refuses, since it was written, and nothing would have caught it: the file validated against nothing, and the one page it was written from says the endpoint does not exist. The three required fields are real rather than an artifact of a generated schema, which is exactly what the OpenRouter row below shows cannot be assumed. **What it does not establish.** No request was routed through a dynamic route. The gateway accepted the configuration, reported it back and deployed it, and whether traffic falls through the chain in the written order under a real provider failure is still Cloudflare's documentation talking. The `vitna` key was stripped before posting, as the file says to; whether Cloudflare would have rejected it is untested, because the emitted instruction is to remove it. ### OpenRouter, which accepted a body its own schema calls invalid | Checked | Result | |---|---| | The emitted `body`, posted unmodified | Accepted, `200`, preset created and active | | What its published schema says about that | That it is invalid. `ChatRequest` marks `messages` required with `minItems: 1`, and the body carries none. The service does not enforce it, so the emitted instruction is correct and the schema is not | | Does the chain survive | Yes. The stored config is `{"models":["anthropic/claude-haiku-4.5","openai/gpt-4.1-nano"]}`, the emitted array exactly and in order | | "An existing slug gains a new active version" | Confirmed. A second POST produced `v2` and moved `designated_version` to it, so re-running a policy updates a preset in place | | Is an unknown key really accepted and dropped | Yes, and this is why provenance sits beside the body. A `vitna` key posted alongside `models` returned `200`, and the stored config came back holding `models` alone | | **Is the lane alias callable, and does it serve** | **Yes. A request was routed.** `@preset/<slug>` was answered by the decided model, `finish_reason: stop`, at a cost of $0.0000088 | That last row is the only place in this document where a real request reached a real provider through a generated configuration. #### The chain, made to fall through Every ordering claim in this file was the vendor's documentation talking. On this one target it is now a reading, and getting there took three attempts because the first two were not what they looked like. | Checked | Result | |---|---| | A first entry that cannot serve the request | **Skipped, and the next entry in the written order served.** An image sent to a text-only first entry was answered by the second, `200` | | The control, without which that proves nothing | That same entry alone returned `404 "No endpoints found that support image input"`, so the chain rescued a lane that would otherwise have failed | | **An entry naming a model OpenRouter does not list** | **Kills the whole request, `400`, fallback or not.** `"... is not a valid model ID"`, identical with a healthy second entry present and without it | | A prompt larger than the first entry's context window | Not a failure at all. Alone, that entry truncated the prompt and answered `200`, so context overflow is an input to selection rather than a trigger for failover | **The third row is the important one, and it settles a refusal this adapter already made.** An unlisted sku does not cost you that entry, it costs you every request through the lane, including the ones the surviving entries could have served. That is why this target refuses the whole file when a lane or any chain entry has no OpenRouter name, and the refusal now rests on a `400` rather than on reasoning. **What it still does not establish.** Whether OpenRouter discovers an entry cannot serve before calling the provider or by calling it and getting an error. The response names the model that answered and carries no attempt trail, so the two are indistinguishable from the outside, and the distinction matters: a provider that is up but failing is the case a chain exists for, and it is not the case demonstrated here. What is demonstrated is narrower and still worth having: the order is honoured, and an entry that cannot serve does not take the lane down with it unless it is unlisted. The fourth row is a caution about method rather than about OpenRouter. The oversized prompt was the first thing tried, the chained call did fall through to the second entry, and it looked like a clean demonstration until the control showed the first entry would have answered anyway. A fallback that fires and a model that was never going to be chosen produce the same reading, and only the control separates them. ### Portkey, which has met no account No Portkey credentials were available, so nothing was posted. What was done is weaker and is not the same claim: | Checked | Result | |---|---| | The emitted `{ name, config }` against `POST /configs` | Validates against Portkey's published OpenAPI | | Which verb updates an existing config | `PUT /configs/{slug}`. Both Portkey's spec and its prose docs say so and neither defines `PATCH`. This file and the emitted `apply` line said `PATCH /v1/configs/{id}`, and now say `PUT` | | How far that validation reaches | Not far. The spec types `config` as a bare `object` and mentions `override_params` nowhere, so nothing checked the `strategy` and `targets` shape that carries the lane | | Whether a missed `vitna` strip is caught | No. Extra keys are not forbidden at the root | **What it does not establish, and the Cloudflare row above is the warning.** Schema validation passing means less than it sounds, because the part of a Portkey config that encodes the decision is the part the spec declines to describe. Cloudflare's adapter also looked correct by inspection and was rejected by the first real POST. The verb correction rests on two Portkey-published sources agreeing with each other, not on a request that returned 405. #### It is gated twice, and not for want of a key An earlier version of this section said this target was where the next credential should go. That was wrong, and the correction is worth stating because it changes what "unexercised" costs here. Measured 2026-08-30 against a real account holding a valid Portkey key: | Route | Result | |---|---| | `GET /v1/configs`, `/virtual-keys`, `/integrations`, `/api-keys` | `403` `AB03`, "You do not have enough permissions" | | `POST /v1/chat/completions` carrying the config inline in `x-portkey-config` | `400` `inline_config_blocked` | The `403` is not a `401`, so the key is valid and simply not an admin one. Portkey files admin API keys under its enterprise offering, so saving a config through the API is an Enterprise action rather than a key you can go and make. And the other door is shut by `block_inline_config`, a governance flag that refuses an inline config object and demands a saved `pc-...` slug, which is the thing the `403` prevents creating. `x-portkey-virtual-key` stays allowed when inline configs are blocked, and carries no `strategy` or `targets`, so it cannot exercise what this adapter emits. So the honest statement is not that nobody has got round to it. On a non-Enterprise account there is currently no way to hand Portkey a generated config at all. #### The one thing that could be settled, and was That left a specific and unpleasant possibility. Portkey's open source gateway validates a config with a Zod schema ([`src/middlewares/requestValidator/schema/config.ts`](https://github.com/Portkey-AI/gateway), read 2026-08-29) which enumerates the permitted keys, and `override_params` is not among them, while Zod strips unknown keys by default. This adapter puts the decided model in `targets[].override_params.model`. If that validator were what runs, the emitted config would be accepted and would route to the target's default model instead of the decided one: valid, plausible, and wrong in the one way this whole document exists to prevent. The hosted product could not answer it, per the two gates above. The open source gateway could, since it is the codebase that validator lives in. Run 2026-08-30 against `@portkey-ai/gateway@1.15.2` with a mock provider recording what the gateway sent upstream, because a gateway's response names the target it used and not the model it asked for: | Cell | Upstream received | |---|---| | Control, a target with no `override_params` | the request body's model, so the path works and the mock sees what is sent | | The shape this adapter emits, `override_params.model` set | **the override**, not the body's model | So `override_params` is applied, and the OSS validator's omission of the key is a stale schema rather than a live behaviour. The decided model goes somewhere the gateway honours. **What that does not cover.** It is the open source gateway, not the hosted product. And the OSS gateway has no concept of an account integration, so the target was addressed as `provider` plus `api_key` rather than as the `@slug` form the adapter emits: what was tested is the `strategy`, `targets` and `override_params` structure, not the slug addressing. A hosted account that can save a config is still the only thing that would close this target. ### The part of this that CI now keeps Everything above was done by hand once. A finding that is true on the day it is written and unwatched afterwards is how the Cloudflare defect survived from the day that adapter was written, so the schema half of it is now a test. `tests/adapter-vendor-schemas.test.mjs` holds each adapter's output against the request schema its vendor publishes, archived under `tests/fixtures/` by `scripts/export-vendor-schemas.mjs` and kept verbatim. It runs offline, like every other test here. **What it checks is required-field presence, not full schema validation.** That narrowing is the finding rather than a shortcut: every one of the five errors Cloudflare returned was `7001 Required`, so missing required fields is the failure mode that actually shipped, and checking it needs no validator in a dependency tree that has none. A config could still be refused for something this does not model. Three things it pins that are easy to lose: - The Cloudflare route carries every field Cloudflare marks required, and a **regression cell** rebuilds the pre-2026-08-29 shape and asserts it is still caught. Without that, a passing suite is also consistent with the check having quietly stopped checking. - Portkey's spec still types `config` as a bare object. The scoping above depends on that, so if Portkey starts describing the config object the test fails and this section is stale. - OpenRouter still requires `messages` on a route that accepts a body without one. That disagreement is the example this file uses for a schema overstating its requirements, and if OpenRouter drops the requirement the example is gone. Re-running the exporter and reading the diff is how a vendor changing its mind becomes visible, which is the event nothing used to notice. ### And the part that only a real gateway can answer The test above inherits a blind spot it cannot fix: a published schema is not the service. Cloudflare refused a route its schema also refuses, which an offline check catches; OpenRouter accepted a body its own schema calls invalid, which no offline check ever could. Both were found by posting, and both directions are recorded here because neither is predictable from the other. ```bash npm run aperture:adapter:verify # every target it has keys for npm run aperture:adapter:verify -- --target cloudflare ``` `platform/engine/adapter-verify.mjs` generates a config from a signed policy, gives it to the gateway, reads it back, and checks the chain survived in order. Three properties are deliberate. **A target with no credentials is skipped, not failed.** A verification that goes red because nobody exported a key is one people learn to ignore, and this file already records what a permanently red check costs. **It is not in CI, and that is a decision rather than a gap.** The Cloudflare token cannot be narrowed below account scope, so a repository secret would hand every workflow edit rights over every AI Gateway on the account. An unattended job that creates objects in a real account also leaves orphans when it fails. So it runs when a person runs it. **Every target posts something that must be refused.** A harness that only ever sends valid configs cannot tell you the gateway validates anything. Cloudflare is sent the pre-2026-08-29 shape and OpenRouter a chain naming an unlisted model; both must be rejected, and a PASS elsewhere means less than it looks if either is accepted. That discipline is not theoretical here: the first failover result in this document was a false positive until a control cell killed it. Cleanup runs in a `finally` and is confirmed by reading the account back rather than by trusting a status code. Cloudflare's routes and gateway are removed. OpenRouter's preset cannot be, so the command uses one stable slug, `vitna-verify-roundtrip`, and versions it in place rather than littering a new name on every run. Portkey is written and has never been executed, because no account existed to run it against. It says so when it runs, and it will fail loudly rather than pass quietly if any of it is wrong. Among other things it is the request that would settle the `PUT` correction above, which currently rests on two published sources agreeing with each other rather than on a response. Cloudflare's and Portkey's specs are OpenAPI 3.0 and OpenRouter's is 3.1, so anyone reaching for a full validator instead wants a 2020-12 one for the third, which is the only mechanical trap in it. ## A decision id in the config, which saves a lookup **What answers "why does production run this" is the decision in the register, not the config**, and that is true whichever target you use. A decision keeps the exact price-period and evidence ids it read, its prediction is regenerated from those append-only rows rather than stored, and every mode carries every accepted decision. None of that depends on the emitted file carrying anything. So what follows is a convenience rather than a property. Where a target lets a config hold a decision id, this writes one, because during an incident somebody is looking at a gateway and not at Vitna, and a lookup is expensive at exactly that moment. Where a target does not, the answer is still in the register and costs a lookup. LiteLLM is the one target of four that lets a config hold it as data: ```yaml - model_name: deep-research litellm_params: model: anthropic/claude-opus-4-8 model_info: vitna_sku: "anthropic/claude-opus-4-8" vitna_workload: "acme/deep-research" vitna_decision: "154" vitna_decided: "2026-08-10T00:00:00.000Z" ``` LiteLLM stores arbitrary `model_info` fields and returns them from `GET /model/info`, which its documentation describes as the mechanism for annotating a model with your own metadata (retrieved 2026-08-29). So on that target the id comes back from the running gateway. That is not a thing gateways cannot do. LiteLLM is a general-purpose gateway and does it fine. What a gateway has no way to put there is a decision, because it makes settings rather than decisions, and that difference belongs to the record rather than to this file. **A chain entry carries its sku and no decision.** Two lanes can fall back to one sku under chains decided separately, and they share one deployment, so a decision id there would be one lane's answer presented as the deployment's. Which lane reaches it, and in what order, is `router_settings.fallbacks` beside that lane's own entry. ## Two routes, and the file commits to one The policy carries two names for every model: the provider's own, and OpenRouter's. A runtime picks between them per lane, by which key its operator holds. A config file cannot pick per request, so `--via` picks once for the whole document. | `--via` | What `litellm_params.model` says | The key the gateway needs | |---|---|---| | `direct` (default) | `anthropic/claude-opus-4-8` | One per provider | | `openrouter` | `openrouter/anthropic/claude-opus-4-8` | `OPENROUTER_API_KEY` | The substrate form has three segments because OpenRouter's own ids carry a slash, so `openrouter/` prefixes a name that already looks like a path. That is LiteLLM's own convention, from [its OpenRouter page](https://docs.litellm.ai/docs/providers/openrouter) retrieved 2026-08-29, where `api_base` is optional and defaults to OpenRouter's. **A name does not change with the route, only the string it resolves to.** A lane is its alias and a chain entry is its direct model id under either route, so switching `--via` changes the `model` lines of a diff and nothing else. Somebody moving onto or off the substrate should be able to read what changed. **A sku OpenRouter does not list refuses the substrate file.** The policy leaves that name empty, which is a fact about the market rather than a gap, and a config naming `openrouter/` and nothing would fail every request. The same policy still renders for `--via direct`, so the refusal is about the route rather than about the policy. Per-provider mixing is not offered. A runtime can be told to reach one provider directly and another through the substrate; this writes one route per file, which is how a gateway operator actually holds keys. ### Which route to write The two executors reach differently, and that is not a disagreement about the decision. The Vitna runtime speaks OpenAI and Anthropic directly and reaches everything else through OpenRouter; LiteLLM speaks many providers directly. So a Google lane written `--via direct` is `google/gemini-3.7-flash`, which LiteLLM can call and the runtime cannot, and written `--via openrouter` both take the same route to the same place. Which is right is a fact about your gateway, not about the decision. Under either route it is the same sku. ## One name is one destination A lane alias and a model id are drawn from the same character set, so a lane called `claude-opus-4-8` can sit beside a chain falling back to that model. LiteLLM keys a deployment by `model_name`, so the two would claim one name and one of them would silently serve the other's model, in a file that is valid and a gateway that starts. That is refused, naming both destinations. Rename the lane, or take one of them out of the policy. ## Multi-gateway policy delivery When an infrastructure stack runs more than one gateway concurrently, delivering configurations individually risks configuration drift between targets. The delivery runner renders and writes verified gateway configurations across Portkey, OpenRouter, and LiteLLM in one command, asserting signatures and cryptographic digests before any destination file is written: ```bash npm run aperture:policy:deliver -- --policy policy.json --out-dir ./gateway-configs ``` Optional flags include `--targets` to restrict destinations, `--provider` to supply Portkey provider slug mappings (for example `--provider openai=@openai-prod`), and `--expected-key` to enforce public key pinning. ## Why there is no Vercel target Vercel AI Gateway is the sixth target [#111](https://github.com/convexityos/vitna/issues/111) names, and it is refused rather than pending. The reason is not effort. A lane has nowhere to live in what Vercel's gateway configures. Its team-wide surface is [routing rules](https://vercel.com/docs/ai-gateway/models-and-providers/routing-rules), read 2026-08-29, and a rule is a rewrite or a deny that **matches requests by model**: ```bash vercel ai-gateway rules add --type rewrite \ --source anthropic/claude-opus-5 --destination anthropic/claude-haiku-4.5 ``` Three things follow, and each on its own is enough. **`--source` is a model, not a name you can define.** The documentation puts it plainly: "your application keeps requesting the source model and AI Gateway substitutes the destination". A rewrite redirects a model somebody already asks for. A lane alias like `deep-research` is not a model, nobody asks for it, and there is no documented way to make the gateway answer to one. **A rewrite is one source to one destination.** There is no ordered list, so a chain has nowhere to go. Vercel does support fallbacks, and they are part of "everything else you send": per request, from application code, which is the one place an adapter must not write. **Vercel says not to depend on it.** The page carries their own caution that routing rules are in beta and "avoid relying on them in production". None of that is a gap in Vercel. Its gateway rules are a governance primitive, for standardising a team on a model or blocking one nobody approved, and they do that job. It is simply a different job from naming an intent and giving it an ordered set of models to try. An adapter here would translate a routing decision into a governance rule and lose the alias and the chain on the way, which is the kind of near-miss every other target's refusals exist to prevent. **What would change this.** A named route or a model alias, something a request can address that is not already a model. If Vercel adds one, the lane gets somewhere to live and this is worth revisiting. The rewrite rule is not it. --- <!-- https://vitna.ai/docs/routing/audit-packs.md --> # Audit and procurement attestation packs Bundle an organisation's declarative Stackfile, standing routing decisions, signed runtime policy envelope, CI policy lock, RFC 6962 Witness price observation Merkle proofs, provider compliance terms, three-provenance SKU capabilities, rollout watchdog status, and compound pipeline Bill of Materials into one self-contained, offline-verifiable `vitna.audit-pack.v1` file. ## Why an organisation pack exists beside a decision pack [`GET /api/v1/decisions/{id}/pack`](../api/v1/engine/decision-pack.md) seals a single routing decision and the observations cited when that one proposal was taken. Procurement reviews, FinOps sign-offs, and model-risk governance audits ask a wider question: what is the organisation running across every component class today, what list and private rates is it paying, which compliance terms and capabilities make each lane eligible, is any shadow or canary rollout drifting, and can an external reviewer verify every figure offline without database or network access? `buildAuditPack` (`platform/engine/audit-pack.mjs`) assembles that full point-in-time record into one deterministic JSON document (`vitna.audit-pack.v1`) stamped with a canonical SHA-256 `digest` and an optional outer Ed25519 signature. ## Six-layer offline verification `verifyAuditPack` runs with zero database queries and zero outbound sockets across six independent layers: 1. **Envelope and claim integrity** (`envelope`): checks `schema` (`vitna-audit-pack-v1`), `format` (`vitna.audit-pack.v1`), recomputes the canonical SHA-256 `digest` over all pack fields (with `digest` and outer signature fields stripped and keys recursively sorted), and checks that `claims.citationBoundary` matches Witness's published claim boundary verbatim. 2. **Outer pack signature** (`packSignature`): when signed (or when `requirePackSignature` is set), verifies the Ed25519 signature over `vitna-audit-pack-v1\norg=<orgId>\ngeneratedAt=<generatedAt>\nstackfile=<stackfileDigest|none>\ndigest=<digest>\n` against a pinned `trustedPackPublicKey` or the embedded SPKI key. 3. **Stackfile, signed policy, and CI lock** (`stackfileAndPolicy`): validates `governance.stackfile` via [`parseStackfile`](stackfile.md) and recomputes `stackfileDigest`; verifies any embedded `policy.json` envelope signature (`vitna-routing-policy-v1`) via [`verifyPolicy`](verifying.md); and runs `verifyPolicyAgainstLock` when `governance.policyLock` (`vitna.lock.json`) is present so lane drift is caught offline. 4. **Witness RFC 6962 price attestations** (`witnessPriceAttestations`): for every entry in `pricingAndProvenance.observationProofs`, recomputes the canonical observation string and `0x00`-prefixed SHA-256 leaf hash, walks the `0x01`-prefixed Merkle inclusion path to the UTC day's Merkle root, and verifies the Ed25519 signature on `vitna-attestation-v1\n<coversOn>\n<leafCount>\n<root>`. Customer-entered private infrastructure rates stay separated under `enteredPrivatePrices` and never carry Witness Merkle proofs. 5. **Provider compliance terms and customer evaluations** (`evidenceAndTerms`): recomputes deterministic SHA-256 digests over every `providerTerms` row (`vitna-provider-term-v1`) and every `customerEvidence` row (`vitna-evidence-v1`). 6. **Operational and FinOps gates** (`governanceGates`): evaluates CI and procurement gates (`requirePackSignature`, `requireAttestedPrices`, `failOnRollback` when `rolloutWatchdog.verdict` is `rollback_recommended`, and `maxMonthlyCostUsd` over `summary.totalEstimatedMonthlySpendUsd`). When every cryptographic check passes and all present layers are verified against caller-pinned public keys (`trustedPackPublicKey`, `trustedRecordPublicKey`, `trustedPolicyPublicKey`), `verifyAuditPack` returns `verdict: "verified"`. When signatures verify solely against keys embedded inside the pack (unpinned) or cite observations from an open UTC day before daily root signing, `verifyAuditPack` returns `verdict: "verified_with_warnings"` and enumerates each warning. Any tampered byte, forged proof, broken lock invariant, or breached gate returns `ok: false` and `verdict: "failed"`. ## Cryptographic key separation and claim boundaries Three distinct Ed25519 key roles participate in an audit pack and are never conflated: - **Witness record key** (`record`): signs daily observation Merkle roots (`vitna-attestation-v1`). Published at [`GET /api/v1/keys`](../api/v1/verification/keys.md). - **Routing policy key** (`policy`): signs loadable `policy.json` envelopes (`vitna-routing-policy-v1`). Published at `GET /api/v1/keys/policy-key.pub`. - **Operator or customer pack key** (`pack`): signs the outer `vitna.audit-pack.v1` envelope (`vitna-audit-pack-v1`) at export time. Both `signAuditPack` and `verifyAuditPack` refuse key conflation if an outer `pack` signer public key equals any Witness `record` attestation key, preserving the boundary between first-party market observations and customer stack configurations. Likewise, `claims` embeds `PROVES`, `DOES_NOT_PROVE`, and `CITATION_BOUNDARY` verbatim from `platform/witness/claims.mjs`: Witness proves what a vendor pricing page published when fetched, not what a vendor billed on a private invoice or whether customer-supplied benchmark scores are representative. Stripping or editing `claims.citationBoundary` fails verification with `altered_citation_boundary`. Export, sign, and verify audit packs from the CLI with [`aperture:audit:pack`](../cli/aperture/audit-pack.md), export an authenticated organisation's pack via [`GET /api/v1/org/audit-pack`](../api/v1/engine/org-audit-pack.md), or run stateless zero-retention verification and assembly via [`POST /api/v1/audit-pack`](../api/v1/engine/audit-pack.md) and MCP `vitna_verify_audit_pack`. --- <!-- https://vitna.ai/docs/routing/capabilities.md --> # Three-provenance capability eligibility Capabilities and runtime speed constraints narrow which candidates may serve a workload before cost per successful unit is weighed, and each constraint is bound to the provenance class that can honestly answer it. ## Why provenance classes exist Not every fact about a model has the same standing. A provider or public catalog listing may declare that a model accepts `tools` or `structured_outputs`, while a customer's own schema still fails in 14% of calls. Conversely, a third-party speed monitor measures real tokens per second across tens of thousands of routed requests, which is stronger than a brochure and still different from the customer's own workload traffic. Treating all three as interchangeable either admits vendor claims as if they were verified evaluations or forces operators to omit real eligibility requirements from the decision record. Vitna records all three and holds each to its own admissibility boundary: | Provenance class | Source | Constraints answered | Admissibility rule | |---|---|---|---| | `declared_mirror` | `app/data/catalog.json` paired via `platform/catalog/sku-map.mjs` | `requiresCapabilities`, `minContextTokens` | Narrows candidate eligibility only. Never crosses a quality or economic threshold and never satisfies `requiresVerifiedCapabilities`. | | `measured_third_party` | `app/data/speed.json` paired via `sku-map.mjs` + `baseId` | `minThroughputToksSec`, `maxTtftMs` | Evaluates throughput and time-to-first-token bounds while naming the measurer, request sample count, and `retrievedAt`. Refuses when unmeasured. | | `measured_customer` | Workload evaluation evidence (`verifiedCapabilities`, `n > 0`, `sourceRef`) | `requiresVerifiedCapabilities` | Requires a customer evaluation measurement with positive sample count `n` and non-empty `sourceRef`. Only provenance class allowed to satisfy `requiresVerifiedCapabilities`. | ## Canonical capability tokens Both `requiresCapabilities` (`declared_mirror`) and `requiresVerifiedCapabilities` / `verifiedCapabilities` (`measured_customer`) take comma-separated values on the CLI or string arrays in JSON (`vitna.stackfile.v1` and `POST /api/v1/decide`) from the canonical token set: - `tools` - `structured_outputs` - `reasoning` - `vision` - `audio_input` - `video_input` - `file_input` - `image_output` ## Registering capability constraints and verified evidence ```bash # Narrow eligibility using declared catalog capabilities, context window, and third-party speed npm run aperture:workload:edit -- convexity/invoice-extraction \ --constraint requiresCapabilities=tools,structured_outputs,vision \ --constraint minContextTokens=128000 \ --constraint minThroughputToksSec=40 \ --constraint maxTtftMs=1200 # Require customer-verified structured_outputs and record the evaluation that proved it npm run aperture:workload:edit -- convexity/invoice-extraction \ --constraint requiresVerifiedCapabilities=structured_outputs npm run aperture:evidence:add -- \ --workload convexity/invoice-extraction \ --sku anthropic/claude-sonnet-4-6 \ --kind eval \ --label invoice-eval-2026-09 \ --measured-at 2026-09-14 \ --n 500 \ --source-ref https://github.com/org/evals/commit/4a9c210 \ --metric successRatePct=95.4 \ --metric verifiedCapabilities=tools,structured_outputs ``` The same constraints travel cleanly in a declarative [Stackfile](stackfile.md) under `workloads[].constraints` and in stateless checks via [POST /api/v1/decide](../api/v1/engine/decide.md) and MCP `vitna_decide_stack`. ## Refusal and stored-brief reproducibility Every capability check follows the engine's general eligibility rule: **a requirement the record cannot check is not satisfied**. - If a workload sets `requiresCapabilities` or `minContextTokens` and a candidate has no paired catalog mirror entry (for example, an unmapped SKU or private infrastructure), the candidate is refused with the missing source named. - If a workload sets `requiresVerifiedCapabilities` and a candidate has `declared_mirror` support in the catalog mirror with no customer `verifiedCapabilities` evidence row, the engine refuses eligibility and states explicitly that a vendor declaration cannot satisfy a customer-verified requirement. - Whenever a decision consults `declared_mirror` or `measured_third_party` records, the exact readings (`listingId`, `retrievedAt`, snapshot `sha256`, `measurer`, and `requests`) are frozen into `decision.inputs.capabilityReadings` and printed under `## What it read` in the decision brief so stored briefs regenerate byte for byte even after daily mirror updates. --- <!-- https://vitna.ai/docs/routing/config.md --> # Emitting a routing config `aperture:config` renders an organisation's accepted decisions as a signed environment fragment. ```bash DATABASE_MIGRATE_URL=... VITNA_SIGNING_KEY=... npm run aperture:config -- --org convexity npm run aperture:config -- --org convexity --out /etc/convexity/models.env ``` ## Flags | Flag | Meaning | |---|---| | `--org <id>` | Whose config to emit. Required unless verifying. | | `--out <path>` | Write to a file instead of standard output | | `--verify <path>` | Check a file instead of emitting one. Takes neither of the above. | ## Which sku a decision emits | Outcome | Emits | |---|---| | `switch` | The candidate | | `stay` | The incumbent | | `test` | The incumbent | A test is an instruction to measure, not a change. Emitting its candidate would move production traffic on the strength of a proposal that said do not. ## The file One line per delivery target, plus headers: `org`, `canonical`, `generated`, `entries`, `digest`, `signature`, `public-key`. Provenance rides on comment lines marked `#:`, naming the workload and the decision id behind each value. Those lines are part of the canonical body, so "this value came from decision 13" is a claim the signature covers rather than a note anyone can retype. The statement prefix is `vitna-routing-config-v1`, distinct from the attestation prefix so a routing signature cannot be read as an attestation. The canonical form is versioned as `config-lines-v1` and is bumped whenever the bytes that get hashed change. ## It reads no clock The instant is passed in, exactly as the brief builder takes one. The same record and the same instant produce the same bytes, which is what makes a diff meaningful. ## The eight refusals A workload with no delivery target is skipped silently, because that is the ordinary case. A workload that names one and cannot be emitted is named, with the reason: | Refusal | |---| | Names a delivery kind nothing can write | | Names a key that is not an environment name | | Claims a key another workload already claims | | Has no accepted decision | | Names a decision that is proposed or superseded rather than accepted | | Has an accepted switch naming no candidate sku | | Names no incumbent, so the key has no value behind it | | Is decided on a sku the registry does not hold, or one that [cannot be named in a request](model-ids.md) | Every one is a refusal rather than a silent omission, because a config missing a line leaves whatever the consumer already had in place. ## Nothing is applied The command prints the file and says so. Diff it against what the consumer holds, then load it there. --- <!-- https://vitna.ai/docs/routing/connected-accounts.md --> # Connected accounts An org connects its OpenRouter account to Vitna from `/console`, by OAuth PKCE. Vitna is never given a provider API key, and there is no field anywhere on this site that would accept one. ```txt /console -> Connect OpenRouter -> openrouter.ai/auth -> /auth/openrouter/callback ``` ## What is granted, and what is not What OpenRouter hands back is a key it minted for Vitna, scoped to the account that approved it. An org's OpenAI, Anthropic and Google credentials are not part of this and never travel: if they exist at all they sit in that org's own OpenRouter workspace under BYOK, which is a relationship between that org and OpenRouter that Vitna is not a party to. The difference is worth stating precisely, because "connect your AI account" usually means "paste your provider keys here". | | Where it lives | Who can revoke it | |---|---|---| | An org's provider keys | The org's own OpenRouter workspace | The org, at OpenRouter | | The key Vitna holds | Sealed in Vitna's database | The org, at OpenRouter | | Anything typed into vitna.ai | Nothing is | n/a | The third row is the point. Connecting is a redirect and a consent screen, so there is no moment at which a credential is typed into this site, which means there is no moment at which this site could leak one it was never given. ## Who may connect An owner of the org. A membership admits somebody to read an org's register; attaching a billable credential to that org is a different act, and it is the first thing in the record to read the `owner` and `member` distinction that `org_member` has carried since sign-in shipped. A member who presses the button is refused with a sentence naming who can. The check runs twice, at the start and again at the callback, because a membership can be revoked while somebody is away at a consent screen and the act that matters is the one that writes the row. ## The verifier is the whole security of the flow OpenRouter's `/auth` takes a callback URL and a code challenge. It takes no `state` parameter, so this flow cannot carry the correlation between the browser that started a connection and the browser that came back the way the GitHub sign-in carries it. It carries it in the session instead. The PKCE verifier is generated on the server, stored against the session that started the connection, and never sent anywhere; OpenRouter receives only its sha256. Two consequences: - A callback arriving in a browser that started nothing finds no pending row and is refused before any network call happens. - A callback that does find one still cannot be completed with somebody else's authorization code, because the exchange presents this session's verifier against a challenge that code was never minted under. The row is single use and lives ten minutes, which is OpenRouter's own lifetime for an authorization code. ## What is refused - A response that is not the shape of a key, however OpenRouter answered. A 200 carrying an error is a refusal, not a key. - A **management or provisioning key**. Those administer the account and mint other keys, a PKCE exchange should never produce one, and if the flow ever changes so that it can, Vitna declines rather than storing an account-administration credential it asked nobody for. - A key that has already expired. - An instance with no sealing key configured refuses before sending anybody to a consent screen, rather than after, so nobody grants a key that then has to be thrown away. ## How the key is held Sealed with AES-256-GCM under a key in `/etc/vitna/env.site`, bound by additional authenticated data to the provider, the org and the key's fingerprint, so a row copied into another org's name fails to open rather than quietly billing the wrong account. Two properties are worth reading as claims somebody can check rather than as reassurance: - **The nightly dump carries ciphertext.** `vitna-backup.timer` copies this database to object storage, and the sealing key is not in it. This is the same reasoning `site.session` stores only a hash from. - **The process that writes the key cannot read it back.** `vitna_auth` holds `INSERT` on `site.provider_connection` and a column-list `SELECT` that excludes `sealed_key`. Postgres refuses `SELECT sealed_key` for that role however it is spelled, by a future feature, a bug or an injection alike. It is the same shape of property `vitna_forms` has over the waitlist. What sealing does **not** buy is worth saying too. A process holding the sealing key that can read the column can open these values. So this is not a system in which nobody can see the key. It is one in which the key is not in the database, not in the backup, not on a screen, not in a log line, and not readable by the process that wrote it. ## Disconnecting is two halves, and Vitna can only do one **Forget this key** deletes the row. Vitna no longer holds the key and cannot use it again. It does not revoke the key at OpenRouter, and it cannot: a key minted by PKCE belongs to the account that granted it. The console renders the owner's revoke link beside the button rather than instead of it, and repeats it on the confirmation, because a key Vitna has forgotten is still live until its owner ends it. Both links are built from the key's sha256, which is the digest OpenRouter's own deep links are keyed on: | | Where | |---|---| | What this key has done | `openrouter.ai/logs?api_key_hash=<sha256>` | | Revoke it | `openrouter.ai/keys/<sha256>` | They resolve only for the signed-in owner and 404 for anybody else. ## Nothing Vitna runs calls a provider with it This is a control-plane connection and it does not put Vitna in the request path. Nothing Vitna runs calls `/chat/completions` with it, usage arrives by [`aperture:usage:import`](usage-import.md) from a file somebody exported or by a runtime operator's own push, and no runtime a customer operates reads a connection. Corrected 2026-09-13: this said nothing in this repository calls `/chat/completions`. The customer-run runtime calls providers with the operator's own keys, and the frozen [hosted router](router.md) would call OpenRouter with a connected key; it is not installed on Vitna's box. This section used to say what the connection was for: a next phase reading an org's own usage from OpenRouter's analytics endpoints, so that the import stops being an afternoon of exporting. A correction, dated 2026-08-26: that phase is not possible under this grant, and [issue #87](https://github.com/convexityos/vitna/issues/87) closed saying so, with the findings and their sources in its comments. Every documented route to day-granularity usage, `GET /api/v1/activity` and the Analytics API alike, requires a management key; what an ordinary key may read is itself, and the key Vitna holds serves no traffic by design, so its own counters would read zero, permanently, about the wrong thing. The key a consent screen mints is an ordinary key, and the refusal above to store anything more is not an obstacle to remove: it is what makes the consent screen safe to approve, so the grant is not widened to fit the feature. The sealed key stands as this page describes it, held and unread, and the console's sentence stays true as written. What would change the answer is OpenRouter scoping a day-granularity read to the key class a consent screen can mint. ## What the grant may serve, established rather than implied The sentence above, that the key Vitna holds serves no traffic by design, says what Vitna does with the key and was read twice as though it also said what the key is able to do. Those are different claims and only the first was ever checked. `docs/routing/router.md` names the gap: a credential class that cannot serve inference is not a routing option at all, so the whole of that design's C1 rung rests on an assumption nobody had sourced. Established 2026-08-30, against OpenRouter's own documentation. A key minted by the PKCE consent flow may call the inference endpoint. The [OAuth PKCE page](https://openrouter.ai/docs/use-cases/oauth-pkce), retrieved 2026-08-30, carries a section headed "Step 3: Use the API key" whose sample posts to `https://openrouter.ai/api/v1/chat/completions` with the exchanged key as a bearer token, and tells the integrator to store it and use it to make OpenRouter requests. The capability is the documented purpose of the flow rather than a side effect of it. **Issue #87 found the boundary in the other direction, and the two findings agree.** OpenRouter separates the classes both ways: the [management keys page](https://openrouter.ai/docs/features/provisioning-api-keys), retrieved 2026-08-30, says management keys are exclusively for administrative operations and cannot call the completion endpoints. So an ordinary key serves traffic and cannot read the account, a management key reads the account and cannot serve traffic, and the consent screen mints the first kind. #87 is not evidence that this grant is narrow in general. It is the same partition seen from the side that happened to be asked about first. What this does not change is the section above it. Nothing Vitna runs calls `/chat/completions` with a connection, and the console's sentence stays true of what runs, because it describes a choice rather than a limit. What would change it is operating the [hosted router](router.md), whose software exists, frozen and not installed. What it does not establish, and neither does anything else on this page: - **Who is billed.** The key is minted under the granting account and [the limits page](https://openrouter.ai/docs/api-reference/limits), retrieved 2026-08-30, describes spending as drawn from that account's balance under an optional per-key cap. Read as a documented mechanism, not as a settled commercial fact for a product that has no customers yet. - **What it costs in rate limit.** The same page ties free-variant request limits to credits purchased on the account, which is the granting org's account and not Vitna's, so a shared ceiling is not the failure mode to design against. Nothing was measured. - **That any of this should be built.** The capability question is settled and it was the smallest of the three preconditions on `docs/routing/router.md`. ## Configuring an instance | Variable | What it is | |---|---| | `VITNA_CONNECTION_KEY` | 32 bytes of base64. Seals connected keys. | ```bash openssl rand -base64 32 ``` Without it, `/console` shows the connect affordance as unavailable and the start route refuses. There is no OAuth client secret and nothing else to configure: PKCE has no client credential, which is why there is nothing here to rotate if it leaked. --- <!-- https://vitna.ai/docs/routing/console-composing.md --> # Composing the register from the console An owner of an org can enter a workload, put a sku in play, record a measurement and arm a watch at `/console`, instead of running the [commands](../cli/aperture/workloads.md) at a prompt on the box. [Ruling](console-ruling.md) was the first thing the console could write and it was half a product: you could answer a proposal and you could not enter the workload a proposal would be about. This is the other half. ## The console does not validate This is the rule the whole surface is built on, and it is worth being exact about because it looks like an implementation detail and is not. A form here is turned into the **argument vector the command would have taken** and handed to the same functions `aperture:workload:add` calls. Every sentence you see is produced by `platform/engine/register.mjs`, the module that refuses: - an id that is not `org/slug`, or an org that does not exist - an incumbent the registry does not carry, or one from another market - a profile key the class does not spend - a profile left empty on a workload that names an incumbent - a metric the engine never reads - a flag on a trigger kind that does not take it None of those is checked in the browser. If it were, the register would have two doors with two rules, and the refusals are the product. It is why the fields are named after the flags (`unit-label`, `threshold-pct`, `measured-at`) and why the profile, the constraints and the metrics are textareas of `key=value` lines: that is what the flags already take, so nothing had to be taught which keys exist. ## What it can write, and what it cannot | Act | Written | Why | |---|---|---| | Enter a workload | Yes | A workload is a registry row | | Correct one | Yes | Same, and only the fields you fill in are written | | Put a sku in play | Yes | A candidate is a registry row | | Record a measurement | Insert only | Evidence is the record. A correction is a better measurement, never an edit of the one somebody decided from | | Arm a watch, call one off | Yes | Arming is a control on each sku of a register row; a disarmament keeps the reason on the row, dated | | Start a rollout | Insert only | A [rule](rules.md) is registered in shadow, so it logs would-haves and routes nothing | | Promote a rollout, or take one out | No | Letting a rule route needs an accepted decision citing its shadow readings, which is a ruling. The grant withholds UPDATE, so no page here can reach it | | Register your own infrastructure | Yes | A [private sku](../cli/aperture/private.md) is a registry row, scoped to your org and published nowhere | | Enter what it costs | Insert only | An entered rate is a record row. A correction is the better rate from the date it became true | | Rule on a proposal | No | That is [ruling](console-ruling.md), a different role and a different page | | Anything at all in `witness` | No | The market record is the collector's, and no console role writes it | | Delete anything, anywhere | No | Append-only is enforced by grants, not by convention | Fallback chains and the delivery target are not here yet. Each is a refusal surface of its own and neither was on the path to a first signed policy, which is what that release was for. Lane rules were on that list until 2026-08-29 and are the exception now, for a reason that is about the product rather than the schema: policies are signed and runtimes execute them, so the ladder is the thing somebody actually wants on a Tuesday, and a canary that needs a shell on the box is a canary most teams will never start. Only the registering half moved. The two acts that change what a rule does to live traffic stayed at a prompt. ## Arming a watch from the row the sku is on Every other act here is reached from a form somebody opens having already decided to perform it. A watch is not: what makes one worth arming is looking at what a workload runs and what is in play against it, which is the register's own board at `/console/workloads`. So each sku on a row carries **Watch its price**, and it opens one form with one question on it. The workload, the kind and the sku are already answered by the row it was pressed on and travel as hidden fields; the six kinds of trigger and the dimension a move is measured on stay on the workload's own page, for somebody who came to arm something specific. **There is no default threshold, and there must never be one.** A watch that fires on a move you would not have acted on is one you learn to ignore, which is worse than no watch at all, so how far a price has to move before it is worth an interruption is your number. The field is empty when the page renders and it carries no suggested figure, and leaving it blank is refused by `--threshold-pct is required on a price move`, which is the command's own sentence rather than one this page wrote. The bounds are the command's too: a tenth of a percent fires on a rounding difference in a provider's own table, and a hundred waits for a price to more than double. A firing writes a **proposed** decision and stops. Nothing on this page, and nothing a watch can reach, accepts anything. ## The eighth role Composing writes as `vitna_register`, which is not `vitna_console` and not `vitna_auth`. The reasoning is the one that made each of those separate: a bug in the composing path must not be able to accept a proposal, and a bug in the ruling path must not be able to invent a workload for one to be about. `drizzle/0025` is the grant, widened once by `0030` for lane rules, and `tests/console-register-scenario.test.mjs` asserts it against a real Postgres, because "this role may compose the register and may rule on nothing" is a statement about the database and only the database can be asked whether it is true. ## It is an owner's act A membership admits somebody to read an org's register. Entering what the organisation runs, and what evidence stands behind it, is the act the register exists for, so it is limited to owners on the same reasoning ruling is. A workload of an org this account does not own is refused by the same sentence as one that does not exist, because which of the two it was is information about somebody else's register. ## What it still does not do It routes no traffic and it runs no evaluation. Entering a workload records what an organisation runs; running the engine against it and accepting what the engine proposes are separate acts with separate grants. Vitna measures nothing: a measurement here is one your own tooling produced, entered with the sample size and the source reference the command already demands. --- <!-- https://vitna.ai/docs/routing/console-ruling.md --> # Ruling from the console An owner of an org can accept or refuse an open proposal at `/console`, instead of running [`aperture:accept`](../cli/aperture/decide.md) at a prompt on the box. This is the act the register exists for. Until it existed, `/console` could show you a proposal and not let you answer it, which made it a window onto the product rather than the place the product happens. ## It is a smaller change than it sounds A decision is append-only **but for two facts a person writes**, and it was designed that way from the first migration that created it. From the table's own note: > `status` records what happened to the proposal, not which decision is current. > Append-only but for two facts a person writes through the CLI as the owning > role, exactly as a candidate's status is written: `status` and the `note` > giving the reason. The outcome, the thresholds, the inputs, the reason and the > brief are fixed at insert, so a correction is a new decision that supersedes > the old one and never an edit. So nothing became writable here that was not already meant to be written by a person. What changed is where that person may be standing. Two columns were added beside those two, `ruled_by` and `ruled_at`, which say who ruled and when. `decided_at` still means when the engine ran. ## Who may An **owner**. A membership admits somebody to read an org's register; ruling is a different kind of act, and it is the first thing in the record to read the `owner` and `member` distinction that `org_member` has carried since sign-in shipped. A member sees the proposals and no buttons. A proposal belonging to an org this account does not own is refused by the same sentence as a proposal that does not exist. Which of the two it was is information about somebody else's register. ## What the two buttons do **Accept** marks the proposal accepted and arms the next review in the same transaction. That is one transaction on purpose: a decision recorded with nothing watching for what would overturn it is how a stale strategy survives, so an acceptance that could not arm must not be able to half happen. **Refuse** needs a reason, which is kept on the row. No acceptance is written and the decision the workload is actually running is untouched, because a refusal is a fact about the proposal rather than a new decision about the workload. What a refusal does write is the watch back. A firing spends the watch that produced the proposal, and until that was added a refusal armed nothing, so a workload watched for a price move that refused the resulting proposal stopped being watched for price moves. It puts back the watch that fired and only that one: the watches armed beside it never fired and are still armed, and arming them again would fire twice off one move. A review date is the exception and comes back on a new date, because a date fires whenever it has passed and putting the same one back would propose again on the next collector pass. The reason is required for the same cause it is required at the prompt: the engine will propose the same thing again the next time the market looks like this, and a refusal nobody wrote a reason for is one the next reader has to make again from nothing. The re-arm is what makes that sentence true, and the reason travels with it: the re-armed watch records the refusal in `armed_by_decision_id`, so the next proposal it writes carries what was said about the last one. Neither routes anything. Accepting a switch leaves the workload's incumbent alone, because whether the traffic moved is not something Vitna can see. ## What it refuses The refusals are `platform/engine/ruling.mjs`, shared word for word with the CLI. A person at a prompt and a person pressing a button rule on the same record, so two implementations would be two answers the day one of them was corrected. | Refusal | Why | |---|---| | The proposal is already accepted or refused | Two people pressed the button. The second is told, rather than ruling twice. | | The record moved under it | The brief's figures were derived against a tape that is no longer what the workload is doing. Re-run the engine. | | A refusal with no reason | The next reader would have to make the judgement again from nothing. | | An acceptance carrying a reason | The brief is the reasoning and is already in the record; that column is where a refusal's reason is read from. | All of them except the last two are checked **inside the writing transaction**, on the connection that writes, with the decision row locked. A proposal that was open when the page rendered can have been ruled on by somebody else by the time a button is pressed, so the only answer that means anything is the one taken at the moment of writing. ## No arming flags `--review-on` and `--price-move-pct` stay at the prompt. Accepting here arms the defaults, which is what the CLI does when nobody passes a flag. A threshold typed into a web form without the tape in front of you is a number chosen to fill a field, and this record is not a place for those. ## The fifth role Ruling writes with `vitna_console`, which exists only for this and holds: | | | |---|---| | `SELECT` | `aperture.decision`, `aperture.workload`, `aperture.trigger` | | `UPDATE` | four columns of `aperture.decision`: `status`, `note`, `ruled_by`, `ruled_at` | | `INSERT` | `aperture.trigger`, so accepting can arm | And nothing else anywhere. It cannot read a session, a membership, the waitlist or the market record, and no grant lets it touch an outcome, a title, a reason, a threshold, an input or a brief. A correction is still a new decision that supersedes the old one, and that is a privilege the database enforces rather than a convention the code follows. A fifth role rather than widening `vitna_auth`, on the reasoning that made `vitna_auth` a fourth: the value of it being separate is that a bug behind sign-in reaches the session tables and nothing else, and letting it accept decisions would spend exactly that. So a bug in the read path cannot rule on anything, and a bug in the ruling path cannot reach a session. `tests/console-ruling-scenario.test.mjs` asserts every line of that table against a real Postgres, by being refused when it reaches past it. ## Configuring an instance | Variable | What it is | |---|---| | `DATABASE_CONSOLE_URL` | The `vitna_console` role. Generated by `deploy/provision-console.sh`. | Unset, `/console` renders the proposals and no buttons, and ruling stays a command at a prompt. That is the deliberate degradation: a page offering a control that would answer 503 is a page that lies about what it does. **The provisioning script must run before migration 0021 deploys**, the way `provision-github-auth.sh` had to run before 0013. A password can never live in a migration, so 0021 only distributes privileges over a role it expects to exist. A deploy that reaches the migration first fails it, which is safe rather than silent: the hook aborts before the restart and the previous build keeps serving. --- <!-- https://vitna.ai/docs/routing/delivery.md --> # Delivery targets Where a workload's accepted decision gets applied. Without one, a decision is only readable; with one, it is emittable. ## The shape `aperture.workload` carries a `delivery` column, a JSON object: ```json { "kind": "env", "key": "MODEL_SONNET" } ``` ```json { "kind": "policy", "alias": "deep-research" } ``` | Field | Meaning | |---|---| | `kind` | How the value is delivered: `env` for a variable a consumer loads, `policy` for a lane a runtime resolves | | `key` | The environment name to set. `env` only, and required there. | | `alias` | The lane a caller names in a request. Required on `policy`, optional on `env`. | An empty `{}` means nothing emits this workload, and that is the ordinary case. Most workloads are decided and never deployed by anything Vitna can write. ## A workload can be both An `env` target may carry an alias as well: ```json { "kind": "env", "key": "MODEL_OPUS", "alias": "deep-research" } ``` That row is written by [`aperture:config`](config.md) as a variable and carried by [`aperture:policy`](policy.md) as a lane, from one decision. It is the state a team holds while it moves from one mode to the other, and a schema that could not express it would force a flag day. ## Setting one ```bash npm run aperture:workload:edit -- <id> --env-key MODEL_SOMETHING npm run aperture:workload:edit -- <id> --policy-alias deep-research ``` Both flags merge rather than replace, so naming a lane on a workload that already writes a variable does not stop it writing one. An empty value removes that half: ```bash npm run aperture:workload:edit -- <id> --env-key "" ``` Removing both leaves `{}`, which is the register saying nothing applies this workload, rather than a kind with nothing behind it. ## The database settles the shape, not the command A check constraint requires `delivery` to be empty, an `env` kind with a key matching `^[A-Z][A-Z0-9_]*$` and optionally an alias, or a `policy` kind with an alias matching `^[a-z][a-z0-9-]*$` and no key. That is in the database rather than only in the command that writes it, because an env key that is not an env key produces a configuration line the consumer silently ignores, and an alias that is not a lane name is a lane no caller can address. A silent ignore is the exact failure this whole phase exists to prevent. The two patterns are also the ones the emitters read from, in one place, and a test compares the constraint's text against them: a constraint that admitted an alias the emitter refuses would let a row into the register that no policy could ever carry. ## One key, one workload. One alias, one workload Two workloads claiming the same key is a refusal, not a merge. The second would otherwise win silently, and which one won would depend on iteration order. The same holds for an alias, and it matters more: two workloads claiming one lane means a request has two answers. ## Why an environment fragment, and why also a document The first consumer already reads its model ids from the environment, so applying a decision costs no code at the other end. A consumer with no such variable has nothing to load a fragment into, which is what the policy document is for. A third consumer that loads neither would add its own arm to the `kind` check and its own writer. The constraint is written to make that an addition rather than a loosening, and `policy` was the first time that was collected on. --- <!-- https://vitna.ai/docs/routing/fallbacks.md --> # Fallback chains What may serve a lane when the model a decision named cannot. ## The question a chain has to answer A fallback is the first thing this register holds that serves production traffic without a decision having chosen it for that request. So the question is not which model, but who said this model may ever answer for this lane. The answer is a column. Every entry names the accepted decision it was registered under, and every request a fallback serves carries that decision on its [ledger row](../runtime/ledger.md). A fallback serve is still an attributable serve. ## Registering one ```bash npm run aperture:fallback:add -- --workload convexity/deep-research \ --sku anthropic/claude-sonnet-5 --note "cross-vendor cover for the frontier tier" ``` The decision is not a flag. It is the workload's standing accepted decision, read at the moment of registration, because that is what the chain exists to back up. `--position` puts an entry at a place in the chain; without it the entry goes on the end. The runtime walks the chain in order and stops at the first model that answers. ## Re-deciding a lane retires its chain When a lane is re-decided, every fallback registered under the old decision stops being emittable, and [`aperture:policy`](policy.md) refuses rather than shipping them: ``` convexity/deep-research falls back to anthropic/claude-sonnet-5 under decision 154, and decision 161 is what stands. Re-register the chain under 161, or take it out ``` That is deliberate, and it is the part worth arguing with. A chain assembled to back up one model is not self-evidently the right chain to back up whatever replaced it, and quietly carrying it over would be the register deciding something nobody did. Re-registering is one command per entry. ## The refusals | Refusal | |---| | The lane has no accepted decision, so there is nothing to back up | | The sku is what the lane already serves, so the chain would repeat itself | | The sku is from another market than the workload | | The sku [cannot be named in a request](model-ids.md) | | The sku is already in this chain | | That place in the chain is held by another sku | ## Taking one out ```bash npm run aperture:fallback:remove -- --workload convexity/deep-research \ --sku anthropic/claude-sonnet-5 --reason "no longer cleared for this work" ``` The row is kept and dated rather than deleted. A fallback that served traffic during an incident is part of what happened, and a row that vanished when somebody tidied the chain would take that with it. A reason is required by the database as well as by the command. The remaining entries keep the positions they had; the emitter renumbers from one when it writes the policy, so a gap here is not a gap there. ## What the runtime does with a chain See [serving a lane](../runtime/serving.md). In short: it walks the chain only when a provider cannot serve the request now, which means a transport failure, a rate limit or the provider's own 5xx. A 4xx is the request being wrong, and asking a second model the same wrong question spends money to be told the same thing twice. Every attempt is its own ledger row, so one request a fallback answered is two: the primary's failure, worth no units, and the fallback's serve, worth one. ## What a chain costs, and why the engine does not say Nothing. Not because a chain is free, but because its price is not knowable before it runs. A route's cost depends on how often the primary fails over, and that rate is the one number no tape carries: it is a property of an incident, not of a listing. So the engine prices the primary and reports the chain as a constraint the decision was made under, and the brief says as much where a reader would otherwise take the figure for the cost of the route. That is a choice with a cost, so both sides are written down. What it buys is that a lane carrying a chain is decidable at all, with every existing trigger and comparison working unchanged. What it costs is a figure narrower than the thing it sits beside, which is why the disclosure is part of the brief rather than a footnote somebody may skip. The alternative stays available and is the truer answer once there is enough history: require an observed split from a prior window before a route is decidable, which makes a route economically knowable only in its second generation. Since [importing usage](usage-import.md) now attributes units to a declared fallback rather than merely tolerating them, the reading that alternative needs is being collected. The rule stays as written until somebody wants the second generation more than the first. --- <!-- https://vitna.ai/docs/routing/hosted.md --> # Running the hosted router `router:start` serves an OpenAI-compatible endpoint for several organisations at once, resolving each request against that organisation's own signed policy and paying for it with that organisation's own delegated OpenRouter key. ```bash VITNA_ROUTER_POLICIES=/etc/vitna/policies \ VITNA_PUBLIC_KEY=$(cat /etc/vitna/policy-key.pub) \ VITNA_ROUTER_LEDGERS=/var/lib/vitna/router \ VITNA_CONNECTION_KEY=... DATABASE_ROUTER_URL=... \ npm run router:start ``` This is Vitna in a request path, which every other page here says Vitna is not. [The design](router.md) is where that is argued and where its costs are named, including the one that does not go away: a hosted router's outage is an outage, for the customers it exists to serve. ## What a hosted policy is, and why it is a different file ```bash npm run aperture:policy -- --org convexity --hosted --out /etc/vitna/policies/convexity.json ``` `--hosted` emits only the lanes that organisation's own connections can fund. A lane it cannot pay for is **absent from the document**, not refused when somebody calls it, which is the same move the emitter already makes for a workload's constraints: a lane carrying a data requirement is written with no OpenRouter name at all, so the traffic cannot go there rather than being asked not to. Three things follow, and none of them needed designing: - The router asks nothing about permission while a request is waiting. That is a property of where the check runs. - Disconnecting an account is a policy re-emission, signed like everything else, rather than a runtime state change nobody can audit. - Every dropped lane is named when the policy is emitted, so a policy quietly smaller than the register is not a thing that can happen silently. A hosted policy and a customer-run policy are different documents with different digests, deliberately. What a customer runs on their own box with their own keys and what Vitna may serve on their behalf are not the same set of lanes, and one file pretending to be both would overstate one of them. ## Calling it ```bash curl https://router.example/v1/chat/completions \ -H 'authorization: Bearer vitna_push_...' \ -H 'content-type: application/json' \ -d '{"model":"vitna/deep-research","messages":[{"role":"user","content":"hello"}]}' ``` The key decides which organisation the request is for. A model name without the `vitna/` prefix is forwarded untouched and marked unrouted, exactly as the runtime does. **A router key is not a push key.** They are the same shape and they are two capabilities: a push key reports what already happened, and a router key spends money at a provider. Mint one deliberately: ```bash npm run org:key:create -- --org convexity --scope router --label "router, prod" ``` A push key presented to the router is refused with a sentence saying so, and a router key presented to `/api/v1/usage` is refused the same way. A capability that is a superset of another has not been separated from it. ## What it holds, and what it cannot | | | |---|---| | The credential in front of a request | The organisation's own OpenRouter key, minted by a consent screen | | Who bills the customer | OpenRouter, directly, on the account that granted the key | | Their kill switch | `openrouter.ai/keys/<sha256>`, without telling Vitna | | A stored provider key | **Refused.** There is no configuration that turns it on | The router is the only thing anywhere permitted to read a sealed connection, and it holds no write on anything. The console can store a credential and cannot read one back; the router can read one and cannot write one. Neither can do both. ## Configuring an instance | Variable | Meaning | |---|---| | `VITNA_ROUTER_POLICIES` | Directory of signed hosted policies, one per organisation, as `<org>.json`. Required. | | `VITNA_PUBLIC_KEY` | The key to verify them against, pinned once out of band. Several, comma separated, are accepted. Required. | | `DATABASE_ROUTER_URL` | The `vitna_router` role. Reads one key row and one connection, and writes nothing anywhere. Required. | | `VITNA_CONNECTION_KEY` | The sealing key the console's connect flow uses. Required, or no delegated key opens. | | `VITNA_ROUTER_LEDGERS` | Directory for one ledger per organisation. Optional, and its absence is said out loud. | | `VITNA_ROUTER_PORT` | Default 8788. | `GET /status` answers for the organisation whose key is presented and for no other. `GET /healthz` is unauthenticated and says only whether the process is up, because that is not a fact about a tenant. `SIGHUP` reloads every policy. A reload that cannot be verified leaves that organisation's running policy in force and takes nobody else's lane down. ## Checking the router against a record it does not keep This is the part worth reading even if you never run the router yourself. Under the file and GitOps modes, a mistaken or forged policy still has to reach a runtime, which means a diff somebody merges. A hosted router has no diff and no reviewer: Vitna signs the policy and Vitna executes it. The merge control is not thin there, it is absent. What replaces it is a check you run at a party Vitna is not. Every request the router makes carries your own delegated key, so OpenRouter's record of what that key did is a second account of the same traffic, kept by somebody with no stake in Vitna's ledger being right. 1. Read what your key did at `openrouter.ai/logs?api_key_hash=<sha256>`. The digest is on your `/status` response and on the console beside the connection. The page resolves only for the signed-in owner. 2. Export it, as JSON or CSV, with a timestamp column and a model column. 3. Compare it against the router's ledger for your organisation: ```bash npm run router:trail -- --ledger convexity.jsonl --provider openrouter-activity.json ``` It exits non-zero when the two accounts disagree, so it can sit in your own CI and mean something without being read. **What it catches**: a serve the router made and did not record, a serve the ledger claims that never happened, and a model substituted for the one your decision named. **What it does not**: a forged policy routing within the same model you already funded, which both accounts would agree on. What bounds that is the reachable universe above, which is why the two mechanisms are described together. The command holds no key and reaches no network. It reads two files you already have, and it runs equally well against a ledger somebody else produced, which is the other half of what makes it worth having. **A clean comparison is not proof the router behaved.** It is one account agreeing with another over the window compared, which is weaker than a merge because it catches after rather than before, and stronger than a promise because the checker is not the party being checked. --- <!-- https://vitna.ai/docs/routing/model-ids.md --> # Model ids Three names exist for one model, and only two of them are the registry's. | Name | Example | Whose | |---|---|---| | Sku id | `anthropic/claude-sonnet-5` | Vitna's | | Slug | `claude-sonnet-5` | The market's | | Call name | whatever the API expects in a request body | The provider's | For the first-party models those three currently coincide. That is a fact about how these providers happen to name things, not a rule anybody promised: OpenRouter already spells Opus 4.5 as `claude-opus-4.5`, and a dated snapshot id or a `-fast` variant would break the coincidence tomorrow. So the call name is written down by hand, the same way the catalog's listing id is, and a test keeps it honest. ## A registered sku can still be unnameable Three reasons, each a property of the sku's market rather than a gap in a file: | Reason | Why | |---|---| | `hosted` | The same model is served under a deployment name the customer chose. There is no id Vitna could know, and guessing one produces a config that 404s. | | `open_weight_serving` | Each host spells somebody else's weights its own way in its own catalogue, which Witness does not read. | | `first_party` | A back-catalogue model no workload names. Adding it is one line, and it belongs to the change that points a workload at it. | ## Unnameable is a refusal, not an omission A sku that cannot be named produces a named refusal from [`aperture:config`](config.md), carrying the reason above. The alternative would be a config quietly missing a line, which leaves the consumer running whatever it already had while the record believes it emitted a decision. --- <!-- https://vitna.ai/docs/routing/outcome.md --> # Did the decision do what it said A decision predicts. An [imported usage export](usage-import.md) observes. This is the two of them side by side, which the record could always answer and never had to. The comparison is read at `/console`, which counts the checks and the shortfalls, and in full on a workload's own page. ## Where the prediction comes from Not from a stored copy. A decision keeps the exact `witness.price_period` and `aperture.evidence` ids it read, in `inputs`, and those rows are append-only, so the figures it was taken on are recovered by running the same derivation over the same rows. That is the property [a decision](../concepts/decision.md) is built to have, and this is the first thing to spend it: a stored figure could drift from the rows it came from, and a regenerated one cannot. Two consequences follow, and both are reported rather than hidden. A decision somebody **entered by hand** names no prices or measurements. Most of the entered history is like this. Its own figures cannot be recovered, so it has no comparison, and the console says that rather than showing an empty panel. A decision whose derivation **no longer runs**, because the profile it recorded spends a dimension the pinned tape does not carry, is a refusal with the reason on it rather than a figure. ## The three things it is careful about **It compares raw cost per unit, not the stored headline.** A decision's headline saving is quality-adjusted: it divides each cost by a success rate a measurement supplied, so it is a saving per SUCCESSFUL unit. An export carries no success rate. Comparing the stored headline against an observed cost per unit would be comparing two different quantities, and the difference would read as a shortfall that nothing about the deployment caused. So both sides are raw cost per unit, derived from the same two figures the decision stored, and the quality-adjusted headline is reported beside them as what it was. **Both sides are priced at what the decision read.** The pinned periods do not move, so the mix is the only thing that can change the observed figure, which is the question: whether the traffic went where it was told. It follows that this says nothing about what the work cost at the market's later prices. That is a different question and it needs the window's own periods. **A figure says how much of the window it covers.** Rows need not sum to `window_units`: a model an export names that the registry does not carry gets no row and its units still count in the denominator, and a sku outside the two the decision priced has no cost here either. Both are counted as unpriced, excluded from the blend rather than counted at zero, and disclosed. A blended cost over 60 percent of a window presented as the cost of the window is the kind of figure this record exists not to print. ## What it does not answer **Quality.** Whether it held after the change is a measurement somebody enters with `aperture:evidence:add`, not something an export can observe. Vitna runs no evaluation and that boundary does not move. **Not cost only, since the evidence loop.** A reading pushed by a runtime now also carries how the units were reached, what was refused and what failed, the tokens the provider reported, and latency at the middle and the tail. Those come back per workload and are printed beside the confirmation, so "the decision was applied" and "the work went well" stopped being the same sentence. A reading an operator typed from an export carries none of them, and the page says so rather than printing a zero. And it is a reading of one window, the newest one imported. A workload with no export has no comparison, which is not the same as a decision that failed. ## Reading a shortfall A shortfall is usually the mix. A window where seven units in ten went to the decided sku realises seven tenths of the predicted saving, and the sentence under the figures says so with the share that caused it. That is a deployment that is partly done rather than a decision that was wrong, and the two want different responses. A window that comes in **better** than predicted has one explanation with two priced skus: the decision moved to the dearer of them, which a workload run for quality does, and the cheaper incumbent is still serving part of the window. The saving is not a win, it is the change not having finished. --- <!-- https://vitna.ai/docs/routing/pipelines.md --> # Multi-class unit profiles and compound pipelines Evaluate end-to-end AI workflows that fan out across multiple component classes, attribute runtime `x-vitna-run` trace volume back to each stage, and separate decision-ready migrations from exploratory class-peer levers. ## Multi-class unit profiles across all 10 Witness classes Production AI workloads consume more than language model tokens. `UNIT_TERMS` in `platform/engine/decide.mjs` maps every one of the 33 registered Witness price dimensions across all 10 component classes into `unitProfile` keys: | Class | Profile keys | Witness dimensions | |---|---|---| | `model` | `inputTokensPerUnit`, `cachedInputTokensPerUnit`, `cacheWrite5mTokensPerUnit`, `cacheWrite1hTokensPerUnit`, `outputTokensPerUnit`, `batchInputTokensPerUnit`, `batchOutputTokensPerUnit`, `priorityInputTokensPerUnit`, `priorityOutputTokensPerUnit`, `flexInputTokensPerUnit`, `flexOutputTokensPerUnit`, `outputImagesPerUnit`, `batchOutputImagesPerUnit` | `input_per_mtok`, `cached_input_per_mtok`, `cache_write_5m_per_mtok`, `cache_write_1h_per_mtok`, `output_per_mtok`, `batch_input_per_mtok`, `batch_output_per_mtok`, `priority_input_per_mtok`, `priority_output_per_mtok`, `flex_input_per_mtok`, `flex_output_per_mtok`, `output_per_image`, `batch_output_per_image` | | `compute` | `gpuHoursPerUnit` | `gpu_hour_on_demand` | | `retrieval` | `embedTokensPerUnit`, `rerankTokensPerUnit` | `embed_per_mtok`, `rerank_per_mtok` | | `speech` | `audioMinutesBatchPerUnit`, `audioMinutesStreamingPerUnit`, `ttsCharsPerUnit` | `audio_minute_batch`, `audio_minute_streaming`, `tts_per_mchar` | | `tooling` | `toolCallsPerUnit`, `pageFetchesPerUnit`, `containerHoursPerUnit`, `sessionHoursPerUnit`, `storageGbDaysPerUnit` | `tool_call_per_1k`, `page_fetch_per_1k`, `container_hour`, `session_hour`, `storage_per_gb_day` | | `extraction` | `ocrPagesPerUnit`, `layoutPagesPerUnit` | `page_ocr`, `page_layout` | | `translation` | `translateInputCharsPerUnit`, `translateOutputCharsPerUnit` | `translate_input_per_mchar`, `translate_output_per_mchar` | | `image` | `imagesPerUnit`, `batchImagesPerUnit`, `imageMegapixelsPerUnit` | `image_per_image`, `batch_image_per_image`, `image_per_megapixel` | | `video` | `videoSecondsPerUnit` | `video_per_second` | | `music` | `musicTracksPerUnit` | `music_per_track` | Because `register.mjs`, `reconcile.mjs`, `stackfile.mjs`, `decide()`, and `evaluatePipeline()` all read from that single `UNIT_TERMS` table, non-model workloads participate in the exact same decision briefs, invoice reconciliations, counterfactuals, and signed policy workflows as language models. ## Compound pipeline Bill of Materials Evaluating stages one lane at a time hides two structural properties of real agentic and multimodal systems: - **Stage fan-out dominates unit price**: a web-search stage called 8 times and a layout-extraction stage parsing 15 pages per report frequently cost more per end-user run than the final synthesis model called once. - **Sequential reliability compounds**: across critical-path stages with recorded `successRatePct` evaluations, compound pass rate is the product of stage pass rates, raising `costPerSuccessfulRunUsd` above raw list `costPerRunUsd`. `evaluatePipeline` (`platform/engine/pipeline.mjs`) computes: - Per-stage `unitCostUsd`, `invocationsPerRun`, `runCostUsd`, `monthlyCostUsd`, `costSharePct`, compliance/capability eligibility, quality, and p90 latency. - Spend rollups `breakdown.byClass` and `breakdown.byProvider`, plus explicit `breakdown.provenance` splitting Witness archive observations (`witnessedRunCostUsd`) from customer-entered private infrastructure rates (`enteredRunCostUsd`). - Pipeline bottlenecks (`dominantCostStage`, `dominantCostClass`, `dominantLatencyStage`, and `weakestQualityStage`). ## Runtime trace attribution (`x-vitna-run`) When callers pass `x-vitna-run` (and optional `x-vitna-agent`) headers through `vitna-runtime`, every JSONL ledger line carries `runId` and `agentId`. `summarizeLedgerForPipeline` groups ledger rows by `runId` to measure empirical upstream `invocationsPerRun` per lane, local response cache hits (`cacheHits`), clean vs fallback-degraded vs failed runs (`runCompletionRatePct`), per-run call distributions, and per-lane p50/p90/p95 latencies. When paired with `evaluatePipeline`, empirical `runtime_ledger` fan-out replaces static stage guesses and reports `invocationDriftPct` against declared fan-out. ## Verified switches vs exploratory peer levers `evaluatePipeline` keeps two strictly separated optimization buckets: - **`decisionReadySwitches`**: stages where a candidate clears compliance terms, [three-provenance capabilities](capabilities.md), `qualityFloor`, `edgeCaseVeto`, and `economicThresholdPct` inside `decide()`, returning `outcome: "switch"`. Only `decisionReadySwitches` lower `optimization.optimizedCostPerRunUsd`. - **`exploratoryTests`**: eligible candidates (including active Witness class peers discovered when `discoverPeers: true`) that cost less at list price while still awaiting workload `successRatePct` evaluations or critical-failure clearance. Exploratory levers carry zero quality claim and emit exact `aperture:candidate:add` and `aperture:evidence:add` commands. Evaluate compound pipelines from the CLI with [`aperture:pipeline`](../cli/aperture/pipeline.md), for an authenticated organisation via [`GET /api/v1/org/pipeline`](../api/v1/engine/org-pipeline.md), or statelesly via [`POST /api/v1/pipeline`](../api/v1/engine/pipeline.md) and MCP `vitna_analyze_pipeline`. --- <!-- https://vitna.ai/docs/routing/policy.md --> # Emitting a routing policy `aperture:policy` renders an organisation's accepted decisions as a signed policy document, which is what a Vitna runtime loads and resolves requests against. ```bash DATABASE_MIGRATE_URL=... VITNA_SIGNING_KEY=... npm run aperture:policy -- --org convexity npm run aperture:policy -- --org convexity --out /etc/convexity/vitna-policy.json ``` ## Why there are two artifacts [A config](config.md) is a value a person loads into a variable something already reads. That is the whole answer for a consumer with a gateway, and it stays the mode with no runtime footprint at all. A consumer with no such variable has nothing to load a config into. A policy is the same decisions in a shape a process can execute: lanes a caller names, models that answer them, and the decision behind each. Both come out of one record and one accepted decision. They cannot disagree about what was decided, and a test emits both from one set of workloads and asserts they name the same sku from the same decision. ## What is still not true **Nothing serves this file to a runtime.** A runtime loads its policy from local disk at boot; a runtime that fetched it from Vitna per request would put Vitna's availability inside its own, which is the dependency this whole arrangement exists to refuse. Corrected 2026-09-13: this said there is no endpoint and there will not be one. Since 2026-08-31 an organisation can [collect a signed policy](../api/v1/engine/artifacts.md) between deployments, which is a collection before the runtime starts, not a fetch while a request waits. **Nothing here holds a key or calls a provider.** This command reads the record and writes a file. **Nothing is applied.** A person puts the file where the runtime reads it. ## Which sku a lane serves The same rule as a config, for the same reason: | Outcome | Serves | |---|---| | `switch` | The candidate | | `stay` | The incumbent | | `test` | The incumbent | A test is an instruction to measure. Routing its candidate would move production traffic on the strength of a proposal that said do not. ## The document JSON, because a runtime loads it and because later phases add a fallback chain and rules within a lane, which an environment fragment has no shape for. ```json { "statement": "vitna-routing-policy-v1", "canonical": "policy-json-v4", "org": "convexity", "generated": "2026-08-24T09:00:00.000Z", "lanes": [ { "alias": "deep-research", "model": "claude-opus-4-8", "provider": "anthropic", "openrouter": "anthropic/claude-opus-4.8", "sku": "anthropic/claude-opus-4-8", "workload": "convexity/deep-research", "decision": "154", "decided": "2026-08-10T00:00:00.000Z", "outcome": "stay", "fallbacks": [ { "position": "1", "model": "claude-sonnet-5", "provider": "anthropic", "openrouter": "anthropic/claude-sonnet-5", "sku": "anthropic/claude-sonnet-5", "under": "154" } ], "rules": [ { "position": "1", "mode": "shadow", "kind": "context_chars_over", "params": "threshold=800000", "model": "claude-sonnet-5", "provider": "anthropic", "openrouter": "anthropic/claude-sonnet-5", "sku": "anthropic/claude-sonnet-5", "under": "154" } ] } ], "digest": "...", "signature": "...", "publicKey": "..." } ``` `openrouter` is the substrate's own name for the same model, which is a third name after the sku id and the provider's call name and cannot be derived from either: Vitna registers `claude-opus-4-8` and OpenRouter lists `anthropic/claude-opus-4.8`. It is empty when OpenRouter does not list the sku, which costs that lane the substrate route and nothing else. It is empty on purpose, at the same cost, when the workload states a data requirement or an approved-provider list that does not name `openrouter`: the substrate is one more party in front of the provider, and neither the requirement nor the list was checked against it. A fallback or a rule of that lane carries no OpenRouter name either. The canonical form is versioned and has moved three times: `v2` added `openrouter`, `v3` added `fallbacks`, and `v4` added [`rules`](rules.md), each carrying its mode and the decision that authorises that mode. The statement prefix is unchanged, because it names the kind of claim rather than the byte format, and the statement itself carries the canonical version so the two cannot be confused. **Only the fields inside `lanes` are signed.** The digest is recomputed from them in a declared field order, so it does not depend on how a serialiser felt about whitespace. Anything else in the file, including the `warning` line the emitter writes for whoever opens it, is prose and is covered by nothing. **Provenance is data rather than a comment.** A config carries "this value came from decision 13" on a comment line inside the signed body. A lane carries it as a field, because a runtime cannot read a comment and because "why did this request run this model" has to resolve to a decision id per request. The statement prefix is `vitna-routing-policy-v1`, distinct from the config and attestation prefixes, so no signature can be read as another kind of claim. For deterministic per-request model and effort selection (`vitna-route`), the runtime verifies a sibling signed format (`vitna-auto-route-policy-v1` / `auto-route-json-v1`); see [Per-request auto-routing](../runtime/route.md). ## It reads no clock The instant is passed in, exactly as [the config](config.md) and the brief builder take one. The same record and the same instant produce the same bytes, which is what makes a diff between two policies meaningful. ## The refusals A workload with no [lane alias](delivery.md) is skipped in silence, whichever delivery kind it names: an `env` target without one is a config-mode workload and is complete as it stands. A workload that names a lane and cannot be routed is named, with the reason: | Refusal | |---| | Names a delivery kind nothing can write | | Names the `policy` kind and no alias, so no caller could ask for it | | Names an alias that is not a lane name | | Claims a lane another workload already claims | | Has no accepted decision | | Names a decision that is proposed or superseded rather than accepted | | Has an accepted switch naming no candidate sku | | Names no incumbent, so the lane has no model behind it | | Is decided on a sku the registry does not hold, or one that [cannot be named in a request](model-ids.md) | | Is decided on a sku that names no provider, so no route could reach it | | Carries a [chain](fallbacks.md) registered under a decision that no longer stands | | Carries a chain naming what the lane already serves, or a sku no request can name | | Carries a [rule](rules.md) whose present-mode authority is not the standing decision | | Carries a rule with unreadable terms, serving what the lane serves, or a sku no request can name | Every one is a refusal rather than a silent omission, and nothing is written when any lane fails. A policy missing a lane leaves a runtime resolving that alias against whatever it loaded last. ## Checking one ```bash npm run aperture:policy -- --verify /etc/convexity/vitna-policy.json ``` Needs no key and no database: it recomputes the digest from the lanes, rebuilds the statement and checks the signature against the public half the file carries. With a database reachable it also compares that key against the one the record publishes, which is the half a signature alone cannot answer. Same three questions, in the same order, as [verifying a config](verifying.md). ## What a policy may not say A decision that respected a constraint, and a policy that then permitted the opposite, would be a signed document permitting what the register refuses. The signature would make that worse rather than better, so the emitter refuses instead, and it refuses where the document is written rather than where it is executed: **a policy that cannot contain a violating lane needs no runtime to check for one**, including every runtime already installed. **An unapproved provider refuses the whole policy.** If a workload names `allowsProviders`, a lane, a fallback entry or a rule target from anybody else is refused, and nothing is written, on the rule that already forbids a partial policy. The chain is where it matters most: a chain is walked when something has already gone wrong, which is the worst moment to reach a provider nobody approved. **A lane with a data requirement is emitted with no substrate route.** A requirement is checked against the terms the *decided provider* published. Riding a substrate puts a second party in front of that provider, and the record holds no term of that party about this work, so the requirement would be satisfied on paper by a route nobody checked. Leaving the OpenRouter name out is how the artifact says so, and it costs the lane exactly the substrate: the field was already allowed to be empty, and the runtime already refuses that one route while serving the lane directly. `VITNA_ALLOW_PROVIDERS` stays and stops being the only place a boundary is written down. It is the operator's second belt: theirs, about their box, over whatever policy it is handed. --- <!-- https://vitna.ai/docs/routing/router-challenge.md --> # Router Challenge (#72) A router executes a routing policy per request. Vitna determines whether handing the choice to a router beats naming a model. ## The Epistemic Problem of Pricing a Router Every fixed model or provider infrastructure class has a published list price on Witness (`witness.price_period`). Vitna archives the raw bytes, timestamps the read, and tracks the price period. A dynamic router (such as Not Diamond, Martian, RouteLLM, or OpenRouter Auto) has no single published list price: 1. It routes dynamically across an arbitrary set of underlying models. 2. It charges in one of several ways: nothing on top of the selected model's own rate, which [OpenRouter's Auto Router documentation](https://openrouter.ai/docs/guides/routing/routers/auto-router) states for that router (retrieved 2026-08-26), a fee per request, or a percentage markup. 3. Its effective cost per unit is knowable only retrospectively from observed deployments or controlled test runs. A decision engine that assumes a static list price for a router would be inventing a figure the market does not publish. Aperture solves this through Router Challenge (#72): evaluating a router candidate retrospectively from confirmed telemetry recorded in `aperture.deployment_observation` and evals recorded in `aperture.evidence`. The [/challenge](https://vitna.ai/challenge) page runs the same function over three illustrative scenarios. Nothing on it was observed, and it names no router and no model. ## The Economics of a Router Candidate For a window of observed requests, the total router cost is: ``` Total Spend = Underlying Token Spend + Router Surcharges Router Unit Cost = Total Spend / Observed Units ``` When a router charges a fee per request, the platform fee can consume a substantial fraction of the token savings gained by picking cheaper models. Router Challenge computes the fee drag: ``` Fee Drag = Router Surcharge per Unit / (Incumbent Token Cost - Router Underlying Token Cost) ``` If underlying model selection saves \$0.006 per unit, but the router fee is \$0.004 per unit, the fee drag is 66.7%, leaving only \$0.002 in net savings. Like every decision in Aperture, candidates are compared on cost per successful workload unit: ``` Successful Unit Cost = Unit Cost / Task Success Rate ``` A router that cuts token cost in half but drops task accuracy from 95% to 85% may produce a higher cost per successful unit once failed retries are factored in. ## Running a Router Challenge From the command line: ```bash npm run aperture:challenge -- <workload-id> --router <router-sku> [--fixed <sku>] [--surcharge <usd>] [--format markdown|json] ``` ## Router ids the engine knows `KNOWN_ROUTERS` in `platform/engine/router-candidate.mjs` carries a default fee for four router ids, used when `--surcharge` is not given. One of those defaults is a sourced fact. The other three are the engine's own assumptions, so a challenge run against one of them should pass `--surcharge` with the fee your own contract states. | Router identifier | Default fee | Source | |---|---|---| | `openrouter/auto` | \$0.000 / req | [OpenRouter's Auto Router documentation](https://openrouter.ai/docs/guides/routing/routers/auto-router), retrieved 2026-08-26: you pay the selected model's standard rate, and nothing more for the routing | | `notdiamond/custom-router` | \$0.001 / req | Not sourced. An engine default, never read from Not Diamond | | `martian/model-router` | \$0.004 / req | Not sourced. An engine default, never read from Martian | | `routellm/mf` | \$0.000 / req | Not sourced. An engine default | ## Minimum Evidence Guardrail A router candidate evaluation requires at least 20 observed units on the workload (`MIN_ROUTER_OBSERVATIONS = 20`). Aperture refuses to evaluate a router on thinner data. ## Verdicts * **SWITCH:** The router satisfies the quality floor, passes all edge case vetoes, and beats the incumbent by at least the workload's `economicThresholdPct` on cost per successful unit. * **STAY:** The fixed incumbent model remains more economical, or the router fails the quality floor. * **TEST:** The router achieved favorable economics, but triggered edge case vetoes or has marginal sample size requiring targeted validation before production rollout. --- <!-- https://vitna.ai/docs/routing/router.md --> # Vitna router **Built on 2026-08-31, ahead of the preconditions this page sets out, on the owner's instruction.** The software exists: `npm run router:start` runs a multi-tenant router serving C1, and [Running the hosted router](hosted.md) is how. What does not exist is the product. Vitna operates no traffic for anybody, the three preconditions in `RUNTIME.md` phase R5 are still unmet, and this page remains the design to hand a reviewer rather than a description of a service somebody can buy. **Frozen at the design on 2026-09-09, and no further code goes into it.** What Vitna sells is routing in the customer's own infrastructure, where independence needs no argument, the customer's own merge is still a control, and an outage here is not an outage there. Hosted routing serves the one buyer that position does not reach, the buyer who cannot run a process at all, and nobody has counted that buyer yet. The page stays published for the reason it was written: a design a reviewer can refuse is worth more unshipped than a service nobody asked for is worth running. Drafted 2026-08-30. It existed because the blocking precondition on that mode is "a credential custody design that has been reviewed by someone whose job is to refuse it", and a design cannot be refused until somebody writes one down. That is still what it is for, and the reviewer's job is now easier and more urgent: there is code to read beside the argument. **The marker this page opened with is gone, and its going is the mechanism working rather than a rule being dropped.** It read "Not built", named `npm run router:start` as the thing whose existence would falsify it, and `tests/docs-drift.test.mjs` failed on the commit that added that script and sent somebody here. `docs/README.md` records why the check runs in that direction: a page that goes on calling itself a plan after the thing ships reads as modesty and is a false statement about what the reader is running. So the sentence changed on the day the claim did. **What each section is now**, because a design page and a description of running software want different reading: | Sections | Status | |---|---| | 1 to 3 | The argument and the custody ladder. Unchanged, and still the thing to refuse | | 4 | Built. The reachable universe is compiled at emit time by `platform/engine/reachable.mjs` | | 5 | Built, with C1 as the only route. C2 and C3 are designed and are not implemented | | 6 | The costs. Unchanged, and none of them is paid off by the code existing | | 7 | Half built. The key split shipped; the independent trail is `npm run router:trail` | | 8 | The sequencing, with steps 0, 1, 3 and 5 struck through or amended below | | 9 and 10 | What a router may never do, and what is still not verified | Authority sits elsewhere and this document defers to all three. `RUNTIME.md` is authoritative for the execution modes, the application axis, the phases and the open decisions. `PRICING.md` is authoritative for what may be charged and what no plan includes, and is itself a proposal dated 2026-08-30 rather than a settled schedule. `COMPETITION.md` is authoritative for the landscape and for where this position is weak. Those three are internal and are cited here by name so a claim can be traced to where it was decided, not so a reader outside can open them. Where this page and any of them disagree, they are right and this is a draft that has aged. ## 1. The word, and the boundary it looks like it crosses `/method` states three boundaries: never a dependency, not an eval platform, not an autopilot. The first read "not a router" until 2026-08-24, when R1 shipped: it became a statement about the control plane rather than about all Vitna software, and `RUNTIME.md` records the exact sentences that changed and why. The page now carries the narrowed form, so this paragraph names what a reader will actually find there. This document does not narrow it a second time, and the distinction is worth being exact about because the word invites the opposite assumption. | | In the request path | Holds provider keys | Bills for tokens | |---|---|---|---| | The record and the emitters | No | No | No | | `vitna-runtime`, customer-run | The customer's software is | The customer does | No | | The router described here | **Yes** | **No, and section 3 is how** | **No, and `PRICING.md` forbids it** | | A router as the market builds one | Yes | Yes | Usually | The third row is the whole proposal. The fourth row is what `COMPETITION.md` calls a scorekeeper marking its own exam, and it is the thing this design has to avoid becoming rather than the thing it has to compete with. ## 2. What "rotate" means, and each kind has its own gate Five distinct rotations get bundled under the word. They are not one feature and they must not share an authorisation gate. | Rotation | Trigger | Gate | Ledger shape | Blast radius | |---|---|---|---|---| | Lane to a different sku | Price move, new evidence, a trigger firing | An accepted decision, and a re-emitted policy | `serve: "primary"`, decision id | All of a lane | | A rule within a lane | A declared predicate on what the request can be counted for | Born shadow. Promoted only under a decision whose brief cites its shadow readings | `serve: "rule:<n>"`, `cohort` where a share looked | A declared slice | | A canary share | `traffic_share`, nested buckets, shipped 2026-08-27 | The ladder: watching, shadow, five percent, twenty five, incumbent | `cohort`, 0 to 99, on every row | Five to twenty five percent of a lane | | A fallback | Transport failure, 429, the provider's own 5xx. Never a 4xx, and deliberately never a 401 | The chain an accepted decision registered | `serve: "fallback:<n>"`, never blended into the primary's numbers | One request | | **The seller of the same model** | A cheaper or faster host for weights nothing decided to change | **Does not exist** | Would need the route on the row | One request | The first four are built, in `platform/runtime/` and `platform/engine/`, and run today on the customer's own box. The fifth is the one that most wants a router, and it is the one this document recommends building first, in section 8, for a reason that has nothing to do with hosting. **Why the seller is different from the model.** Vitna already mirrors what every seller of every model charges, key-free, into `app/data/endpoints.json`. Those prices move on a much shorter half-life than a model choice does, and a customer applying decisions by hand cannot chase them. Rotating the seller while holding the model constant is also a change with no quality question attached, which is what would let it move under a standing decision naming the model rather than under a new one. It is not free of a gate, and the record already says why. Those are list prices per seller and not what anybody paid, and quantization is the seller's own declaration and is verified by nobody. So a seller whose declaration cannot be checked is reachable only where a decision named it, and the gate is the declaration plus measured throughput rather than nothing. ## 3. The custody ladder One test orders it, and the test is the design in a sentence. > **The router never holds a credential the customer cannot end in one action, > at the source, without asking Vitna.** | | Posture | Who bills the customer | Their kill switch | Reaches | |---|---|---|---|---| | C0 | Customer-held, customer-run: `vitna-runtime`, Deploy | Their provider, directly | Stop the process | Everything. Built | | C1 | Delegated and revocable at source: the OpenRouter key a consent screen mints | OpenRouter, directly | `openrouter.ai/keys/<sha256>` | Everything OpenRouter lists | | C2 | Federated and short-lived: assumed roles, workload identity, managed identity | Their cloud, directly | Delete the trust policy | Bedrock, Vertex, Azure | | C3 | The key rides the request, held for its life and never written | Their provider, directly | Rotate the key | OpenAI and Anthropic direct | | C4 | A stored provider key | Vitna could be | Ask Vitna to delete it | Refused | **C4 is refused and does not become available later.** Not because sealing a secret is unsolvable, but because "ask Vitna to delete it" is the customer's only remedy, which is the shape of dependency this whole architecture is arranged against. `PRICING.md` lists "provider keys stored by vitna.ai" among the things no plan includes, and this document does not reopen it. **C1 is three quarters built and is the interesting rung.** `docs/routing/connected-accounts.md` describes a control-plane connection that serves no traffic: OAuth PKCE, owner-only and checked twice, the verifier held in the session and never sent, sealed with AES-256-GCM under additional authenticated data binding provider, org and fingerprint, and written by a role whose column-list `SELECT` cannot read `sealed_key` back. Making that key serve traffic changes none of it. What it buys that nothing else on the ladder buys is a check Vitna does not control: > The customer reads what the key did at `openrouter.ai/logs?api_key_hash=<sha256>`, > at a party Vitna is not. The router's own ledger becomes falsifiable by > somebody else's record. That is the same move as pinning the public key out of band rather than reading it from the file it verifies. A claim somebody can check beats a claim somebody makes. It also survives `PRICING.md`'s narrowest exclusion, which is worth stating because it is the one a hosted router looks most likely to trip. "Consolidated billing for third-party model providers" is excluded; a PKCE key is minted under the customer's own OpenRouter account, so OpenRouter bills them directly and Vitna is a merchant of record for nothing. **C2 is where the cloud vendors already solved this.** Nothing at rest, credentials minted per request and expiring in minutes, revoked by deleting a trust policy the customer owns. The exact shape of each grant is a vendor fact and is not asserted here: it carries a retrieval date when an adapter is written against it, per the repository rule. **C3 is the honest weak rung and should not ship first.** The router sees plaintext, in memory, for every request. "Nothing is stored" is true and smaller than it sounds, because a compromised router harvests every key that passes through it. It also forces the caller to present a bundle rather than a key, since a fallback may cross vendors inside one request. ## 4. Custody and routing are one fact seen twice This is the part that makes the design cohere with what is already built rather than sitting beside it. `allowedProvidersOf` and `hasDataRequirement` in `platform/engine/policy.mjs` are read when a policy is emitted, so a lane carrying a data requirement is emitted with no OpenRouter name at all: the traffic cannot go there, rather than being asked not to. `RUNTIME.md` states the general form, that gates are compiled into the signed policy and never evaluated by Vitna while a request waits. Apply the same move to credentials. > The credentials an org has connected define its **reachable universe**, > established when the connection is made and compiled into the policy at emit > time. A rotation to a provider the org has not funded is unreachable, not > refused at request time. Three things fall out without being designed: - No synchronous permission check, so the router asks vitna.ai nothing while a request is waiting. That is the property, not a promise about it. - Disconnecting a credential is a policy re-emission, through the same signed artifact as everything else, rather than a runtime state change nobody can audit. - The console can say, per lane, reachable by which route or unreachable and why, which is the shape `/status` already answers in for keys and lanes. ## 5. One request ``` caller SDK -> router model: "vitna/deep-research" resolve lane the signed policy, digest pinned per request rules declared predicates, exact character counts cohort stable hash, 0 to 99, as traffic_share already buckets route C1 delegated, C2 federated, or C3 presented attach minted per request, or opened, or forwarded forward the provider's own endpoint, billing the customer fallback 5xx, 429 and transport only, each its own row ledger lane, digest, decision id, sku, route, cohort, tokens, latency, primary or fallback or shadow ``` Everything `platform/runtime/` refuses today it keeps refusing: no fabricated response, no retry of the same model without a row saying so, no fallback blended into the primary's numbers, no key in a log line or an error message including a prefix of one, and no reading of content beyond the counts routing needs. Two problems are genuinely new, and both are multi-tenancy rather than routing. Policies must not bleed between tenants, which the per-request digest pin is the answer to. And under C3 a credential must not outlive the request that carried it, which is a property to test rather than to state. ## 6. What the router spends, and how much comes back | Property today | What hosting costs it | Recoverable | |---|---|---| | An outage at vitna.ai is an event the runtime cannot observe | Gone. Vitna becomes an uptime dependency | Partly, and less than it first looks. See below | | Independence: an unflattering verdict costs nothing to publish | At risk the moment Vitna is in the money path | Yes, by never taking a token margin and by C1 to C3 keeping the customer billed by their own vendor | | Vitna holds no provider keys | Still literally true under C1 and C2. A delegated key and an assumed role are not provider keys | Yes, and the sentence gets more precise rather than retiring | | Multi-region availability | New, real, and permanent | No. This is R5's actual price | **The obvious rescue for the first row does not work, and it fails in exactly the wrong segment.** A client shim whose failure mode is calling the provider directly with the lane's decided model would turn a Vitna outage into a question of which model you get rather than whether you get one. But calling the provider directly requires the customer to hold provider keys locally, which is the precondition for running the runtime. So the shim protects the customers who did not need a router, and does nothing for the buyer who cannot run a process, who is the only reason to build one. State it as a limit rather than as a mitigation: **a hosted router's outage is an outage, for the customers it exists to serve.** That is what the third R5 precondition, multi-region availability as a paid promise, is the price of. ## 7. Where the merge disappears, and what has to replace it `RUNTIME.md` open decision 9, drafted 2026-08-30, splits the signing key: one Ed25519 key currently signs both the daily Merkle roots and every emitted config and policy, and a compromise of a customer-facing signer must not be able to forge the public record's roots. It then declines to overclaim, and the sentence it declines with is the most important one in this document: > A forgery still has to reach a runtime, and under the file and GitOps modes > that means a diff somebody merges, which is a second control. Under a pushed > policy it is thinner. **A hosted router is the thinnest case there is.** Vitna signs the policy and Vitna executes it, so there is no diff, no reviewer and no second party between a mistaken or forged policy and production traffic. The merge control is not thin here. It is absent. Two consequences, and neither is optional. **The key split stops being hygiene.** Under every other mode it bounds a compromise. Under a router it is the only control left on what the router executes, so the router does not ship before it. **The router owes a replacement control, and it is after the fact.** The candidate is section 3's independent trail: under C1 the customer can catch a serve their own register never decided, at OpenRouter, in a log Vitna does not control. That is weaker than a merge, because it catches after rather than before, and stronger than a promise, because the checker is not the party being checked. Any router that ships should ship with a documented way to run that check, not merely the possibility of one. **There was a hard ordering constraint here, and it was paid on 2026-08-31, which is the same day this page first said it was owed.** `policy-store.mjs` used to refuse a policy whose carried `publicKey` was not the single value `VITNA_PUBLIC_KEY` pinned, so a key could not be replaced without every deployed runtime being edited and restarted. It now parses a comma-separated set and a policy signed by any member verifies, which is how a signing key is replaced without an outage. `witness.signing_key` landed alongside it, so "the key the record publishes" is a fact with a purpose attached rather than a query against the newest attestation row, and that query would have started answering the wrong question the moment a policy key was split out. Both are prerequisites of this page rather than parts of it, and both are done. What that buys the router is narrow and worth stating exactly: the key split is now possible, not performed. Until it is, one key still signs both the daily Merkle roots and every emitted policy, and the argument above about a customer-facing signer is unchanged. ## 8. What to build, and in what order | | Step | What it clears | |---|---|---| | 0 | ~~`VITNA_PUBLIC_KEY` accepts a comma-separated set, and the record says which keys it acknowledges and what each signs~~ **Done 2026-08-31.** | It preceded everything, and it did | | 1 | ~~Open decision 9: the isolated asynchronous signer, the split key, its own least-privilege role~~ | **Built 2026-08-31.** `aperture:sign`, `vitna_signer`, and the policy key in its own file with its own unit | | 2 | ~~**Seller rotation under a standing decision, in the customer-run runtime**~~ | **Built 2026-08-31**, after 1, 3 and 5 rather than before them. `docs/routing/sellers.md` | | 3 | ~~C1 as a data-plane route, still customer-run~~ | **Built 2026-08-31, and hosted rather than customer-run.** The independent trail is `router:trail` | | 4 | Sell the ten Deploys. Record what each buyer refuses | The R5 demand precondition, measured rather than assumed. **Not done** | | 5 | ~~Hosted, C1 only, as its own product with its own risk review~~ | **The software is built. The product is not**, and the risk review this row is conditional on has not happened | **The order above was not followed, and it is worth recording that rather than tidying it away.** Steps 1, 3 and 5 were built on 2026-08-31, and step 2, which this section named as the place to START, was built last on the same day. The reason it was named first stands: it has no custody question, it proves the rotation mechanics on data already mirrored, and it is a feature of a runtime that already ships. What the order cost is smaller than it would have been and is not zero. Seller rotation now exists in the customer-run runtime, so it is no longer true that its first run anywhere will be inside a Vitna-operated process. What is still true is that it was written after the hosted router rather than before it, so it did not get to be the thing that found the rotation bugs before hosting could add its own. Nothing in this repository has yet routed a real request to a named seller. Step 4 is untouched, which means the demand precondition is exactly as unmeasured as it was when this page was written. The software existing is not evidence that anybody wants it. **Step 4 is the honest form of R5's demand precondition.** Deploy is built, complete, and priced at $79 a month, and `PRICING.md`'s rollout sells it by hand to the first ten. It is the product that absorbs "run it for me". So the router's evidence is not whether anybody asks for hosting, it is **what the first ten Deploy buyers refuse Deploy for.** If they stall on running a process in their own cloud, that is the signal. If they stall on price or on trust, which is what the open decision on $79 expects, there is no demand case yet. **Step 5 is a product and not a tier**, and `PRICING.md` says so in advance: if Vitna ever operates hosted traffic it becomes a different product with a different risk review and cost model, and it cannot be smuggled into a plan as an entitlement. ## 9. What a router may never do Inherited from `PRICING.md`'s exclusion list rather than invented here, and narrowed to this surface. - No token spread, no percentage of provider spend, no percentage of claimed savings. The fee cannot move with the recommendation, or the recommendation is not worth reading. - No paid provider placement, and no preferred-provider ranking. - No stored provider key, which is C4 and is refused above. - No consolidated billing. The customer's provider bills the customer. - No silent substitution of one model for another. Every serve names the decision that authorised it, per request, forever. - No prompt or response inspection beyond what routing counts, and no retention of either beyond the request. `PRICING.md` excludes prompts or responses transiting Vitna **by default**, and a router is the explicit non-default: it is an opt-in a customer takes deliberately, priced and reviewed separately, not a capability that arrives with a plan. ## 10. What is not verified The four questions open decision 8 asks of Connected mode, re-asked of this credential class, have answers in sections 3 and 4 rather than open ones: per account and keyed to the org that granted it, which the sealing already binds; reachable universe compiled at emit time; revocation makes a lane unreachable at the provider before Vitna notices, so fail static and report; and owner-only approval, checked twice, already built. Those answers are for a reviewer to refuse, which is the point of writing them down. What is genuinely unverified: - **Whether an OpenRouter key minted by a consent screen may serve `/chat/completions` at all.** `docs/routing/connected-accounts.md` says the key Vitna holds "serves no traffic by design", which implies the capability without establishing it. Issue #87 already found this grant narrower than expected once, for usage rather than for serving. Nothing in step 3 is worth starting until somebody checks this against OpenRouter's own documentation, with a retrieval date. - **The shape of each cloud's delegation grant**, which section 3 deliberately does not assert. - **Whether the buyer exists.** `COMPETITION.md` already says the audit buyer is asserted and has not been counted. The router's buyer is a subset of a population nobody here has observed, and step 4 is the only thing that changes that. --- <!-- https://vitna.ai/docs/routing/rules.md --> # Rules within a lane The per-request question, answered without an oracle: a declared predicate on an observable request attribute, routing to a named sku, shadow first. ## The refused alternative, and why this shape Within one lane, a request to find three citations and a request to synthesize forty sources are different work, and a per-lane incumbent prices both at the dearest one. The tempting answer is a classifier choosing models per request, and an unrecorded one is refused: that is a decision made a million times a day with no record, no threshold anyone wrote down and no brief anyone can regenerate. A rule is the opposite of that in every part: its predicate is declared, its terms are signed into the policy, and every request it touches resolves to a decision id. What the refusal names is those three absences, not choosing per request. Until 2026-09-29 this page said the classifier "is refused" without the qualifier, which was wider than the reason it gave. A per-request router can keep all three: [Switchyard](https://vitna.ai/switchyard), which Vitna is building, scores each request by rules written in its configuration, keeps one ledger row per request with the decision's reasons and no prompt text, and re-runs recorded decisions under a new policy before that policy serves. It is a separate program and not part of this runtime, and the two kinds below remain the runtime's per-request answer. ## The two kinds | Kind | Params | Matches when | |---|---|---| | `context_chars_over` | `--threshold`, in characters | The request carries more content than the threshold | | `traffic_share` | `--share`, a whole percent from 1 to 99 | The request falls in the first `share` buckets of a hundred | ## Registering one, which is always shadow ```bash npm run aperture:rule:add -- --workload convexity/deep-research \ --kind context_chars_over --threshold 800000 \ --sku anthropic/claude-sonnet-5 ``` ```bash npm run aperture:rule:add -- --workload convexity/deep-research \ --kind traffic_share --share 5 \ --sku openai/gpt-5.6-luna ``` Since 2026-08-29 the same act is on the workload's own page under **Start a rollout**, which is the same refusals through a form: the console builds the argument vector this command would have taken and hands it to the same functions, so a sentence you see in a browser is the one you would have seen here. Promoting a rule and taking one out stayed at a prompt on purpose, and [composing](console-composing.md) says why. There is no flag that registers a routing rule, and that is the design: **a rule's effect is evidence before it is behaviour.** A shadow rule reaches runtimes through the policy and logs, on each request it would have changed, what it would have served, as two fields on the [ledger](../runtime/ledger.md) row of what actually ran. It routes nothing. `--threshold` is in CHARACTERS of request content, counted exactly by the runtime, never tokens estimated by anybody. The record's [context bands](../concepts/price-period.md) price the provider's own token thresholds; how a character threshold maps onto a band is the operator's claim, made reviewable in the decision that promotes the rule. The refusals mirror a [fallback's](fallbacks.md): a lane nobody has ruled on, a sku the lane already serves, one from another market or that no request can name, a duplicate predicate, a held position. ## Promoting one, which needs a ruling that cites the shadows ```bash npm run aperture:rule:promote -- <rule-id> --under <decision-id> \ --readings 2026-08-20..2026-08-24 --source-ref "ledger, box 1" ``` The order of authority is the point. First the shadow rows accumulate. Then somebody weighs them, writes a brief that cites them, and that decision is accepted through the ordinary path. Only then does this command record that the ruling covers the rule, and it refuses: | Refusal | |---| | A decision of another workload, or one that is a proposal | | An accepted decision a later one has superseded: a rule routes under what stands | | A decision decided BEFORE the rule was registered, whose brief cannot have weighed shadow evidence that did not exist yet | | No `--readings` window, or no `--source-ref` naming where the cited rows live | The citation lands on the rule's own row (`promoted_readings`, `promoted_source_ref`), so "why does this rule route" answers from the register without opening a document. ## What a routed rule does at request time The rule's target goes to the head of the lane's attempts, and the lane's own order stands behind it: serving the decided model is always authorized, so a rule target that cannot answer degrades to the primary and its [chain](fallbacks.md) rather than to an error. Every attempt is its own ledger row, attributed to its own authority: the rule's serve to the decision that promoted it, the primary's to the lane's standing decision. ## Going stale Re-deciding a lane retires its rules exactly as it retires its chain: the [policy emitter](policy.md) refuses a shadow rule whose registering decision, or a routing rule whose promoting decision, is no longer the standing one, naming what to re-register or re-promote under. A rule assembled against one incumbent is not self-evidently right for its replacement. ## Removing one ```bash npm run aperture:rule:remove -- <rule-id> --reason "threshold was wrong" ``` Dated and kept, as everywhere in this schema: a rule that shadowed or routed traffic is part of what happened to that traffic. ## A share, and the ladder it completes A `traffic_share` rule is the canary this register could not run. It changes none of the epistemics: born shadow, promoted only under an accepted decision whose brief cites its shadow readings, removed by a dated row that is kept. What changes is that the promoted form may route five percent instead of all of it. ``` watching -> shadow -> 5% canary -> 25% -> accepted incumbent ``` Not every workload needs every rung. What matters is that an organisation can read the ladder off the record, with a decision at each one. ### It is a bucket, not a draw The assignment is `FNV-1a(cohort key) % 100 < share`. Three things follow, and each is why it is not a random number: - **The same request always lands the same way.** A canary is reproducible rather than merely measurable. - **The ledger can be audited.** Every row carries `cohort`, the bucket the assignment used, beside `serve: "rule:<n>"`, so a reader can recompute which arm any request belonged to. A draw would have made that impossible. - **Promoting is growing, not reshuffling.** The buckets are nested, so moving a share from 5 to 25 keeps everything already on the new side there. The readings before and after a promotion are about overlapping populations. ### Requests, not users, unless you say otherwise Without a cohort key, the bucket is derived from the request itself, so **a share is a share of requests**. Two requests from the same person can land on different sides, which means the reading measures the model and not the experience. Send `x-vitna-cohort` with something stable, a session or an account id, and the share becomes a share of whatever that identifies. The runtime reads that one header and nothing else about a caller's headers reaches routing. The bounds are the meaning rather than validation. Zero is a rule that does nothing, which is what leaving it unregistered already says. A hundred is not a share, it is the lane, and moving a whole lane is a decision about its incumbent rather than a rule inside it. --- <!-- https://vitna.ai/docs/routing/sellers.md --> # Rotating the seller, not the model A lane may name several hosts of the same weights and let a request go to the cheapest one that is up, without a new decision, because nobody decided to serve a different model. This is the fifth rotation in [the router design](router.md), and the only one that does not need a router. It runs in the runtime you already have. ## Why it moves under a decision that already stands The other four rotations change what answers a request. A lane moving to another sku, a rule promoting inside a lane, a canary widening, a fallback firing: each of those is a different model doing the work, so each is a quality question and each has a gate that ends in an accepted decision. Rotating the seller holds the model constant. Nobody chose different weights, so there is no quality question to re-ask, which is what lets the price be chased under the decision that already named the model. And the price is worth chasing. Vitna already mirrors what every seller of every model charges into the endpoints snapshot, key-free. Those prices move on a much shorter half-life than a model choice does, and a team applying decisions by hand cannot follow them. ## The gate, which is not the price `deepseek/deepseek-chat-v3-0324` was sold at four prices when the mirror last ran: | Seller | Declared | Input | |---|---|---| | DeepInfra | fp4 | 0.24 | | SiliconFlow | fp8 | 0.25 | | GMICloud | fp8 | 0.29 | | Crusoe | bf16 | 0.50 | The cheapest row is half the dearest and it is not the same artifact. The snapshot's own attribution says why this cannot be waved through: every figure is a seller's list price when the mirror ran and not what anybody paid, and **quantization is declared by the seller and verified by nobody.** So the gate is not "trust the declaration", which nobody can check. It is: > A seller is reachable only where the workload named the quantization it is > declaring. The declaration has to match something a person wrote down. A workload that never said fp4 was an acceptable way to serve its weights does not rotate to fp4, however cheap it is, and a seller declaring nothing at all is excluded for being uncheckable rather than for being bad. ## Turning it on A workload names what it will accept: ```json { "allowsQuantizations": ["fp8", "bf16"] } ``` Then emit as usual. The lane comes out carrying its admissible sellers, cheapest first, with the quantization each declared frozen into the document: ```bash npm run aperture:policy -- --org convexity --out policy.json ``` **A workload that says nothing gets exactly the policy it gets today**, down to the bytes. That is what keeps every runtime already deployed serving unchanged. Three things are reported rather than silently done. A workload naming a quantization no request can ask for refuses the whole file. A workload whose model has one admissible seller is emitted **without** a seller set and told why, because a one-entry list is a pin rather than a rotation and would stop the lane using a second seller the day one appeared. And a lane with no OpenRouter name is refused, because a seller is a host the substrate picks between and a direct route has none. ## What the runtime sends ```json { "model": "deepseek/deepseek-chat-v3-0324", "provider": { "order": ["siliconflow/fp8", "gmicloud/fp8", "crusoe/bf16"], "allow_fallbacks": false, "quantizations": ["fp8", "bf16"] } } ``` The order is the emitter's, from the prices the record mirrored, so it is in the signed artifact and the runtime consults no price while a request is waiting. A price that moved after the policy was signed changes nothing until somebody re-emits, which is the same staleness every other figure in a policy has. `allow_fallbacks: false` is the field that matters most. Without it an exhausted list is served from a host outside it, which is a silent substitution, and this runtime refuses those everywhere else. With it, an exhausted list is an error the caller sees. Field names read from [provider routing](https://openrouter.ai/docs/features/provider-routing), retrieved 2026-08-31. ## What an older runtime does It refuses the file, and that is the intended behaviour rather than a cost. A lane carrying a seller set adds lines to the bytes the digest is taken over, so a runtime that predates this recomputes a digest that does not match and declines to load the policy, keeping whatever it had. A lane carrying none adds nothing, so the same runtime keeps serving that policy exactly as before. The alternatives were both worse. Bumping the canonical version would make old runtimes refuse every policy, including ones they can execute perfectly. Putting the set outside the hashed body would leave them serving happily while ignoring a routing instruction, and would leave that instruction unsigned, which is the one thing no part of this system does. ## What is not verified **Whether the tags in the endpoints snapshot are the slugs `provider.order` reads.** They come from OpenRouter's own endpoints API, on the `tag` field, and they have the shape the routing documentation shows for variant slugs (`deepinfra/turbo`, `google-vertex/us-east5`). Nothing here has confirmed the two are one namespace, and no request has been made with one. That gap is bounded rather than open. `quantizations` says the same gate again in the vendor's own vocabulary, so it holds if the tags are wrong; and `allow_fallbacks: false` means a wrong tag produces an error rather than a request quietly served by a host nobody named. What would settle it is one real request against a rotating lane, which is the first thing to do with this. **Which host actually answered.** The chat completions response documents no field naming it ([API overview](https://openrouter.ai/docs/api-reference/overview), retrieved 2026-08-31), so the ledger records the set a request was **permitted** to use and not the one that served it. That is a real limit and it is why the row is named `sellers` rather than `seller`. Which host served is a fact OpenRouter keeps and this does not, and reading it there is the same independent trail [the hosted router](hosted.md) describes. A check somebody else keeps turns out to be useful for a second reason. --- <!-- https://vitna.ai/docs/routing/signing.md --> # Signing an artifact without an operator `aperture:sign` answers a queue of asks for signed artifacts, rebuilding each one from the record rather than signing what it was handed, so that a config or a policy can reach a customer without somebody standing at a prompt. ```bash DATABASE_SIGNER_URL=... VITNA_POLICY_SIGNING_KEY=... npm run aperture:sign ``` ## The problem it solves, which is not convenience [`aperture:policy`](policy.md) and [`aperture:config`](config.md) both sign, and both need a key the web process deliberately cannot see: one environment file, one reader, and a test that fails if anything a page can import so much as names the variable. That boundary is worth keeping. Its consequence was that a signed artifact existed only when a person ran a command, which is not something a customer can be given. So the site writes a row saying what it wants, and a timer unit answers it. The site still cannot sign, and now it does not have to. ## A request is an ask, and never a draft The row names an organisation and a decision. It carries no bytes at all. That is the whole security argument. A signer that signed the document it was handed would be an oracle: any bug upstream of it becomes a file with Vitna's signature on it, and the signature would be perfectly valid. Instead the signer re-reads the accepted decisions, rebuilds the artifact itself through the same code [`aperture:policy`](policy.md) runs, and the only thing a caller influences is which org and which decision the ask claims to be about. A caller who gets either wrong receives a refusal, not a signature. ## Four ways an ask is refused | Refusal | What it means | |---|---| | Out of scope | The named decision belongs to a different organisation | | Not accepted | The named decision is proposed or rejected, so nothing routes under it | | Superseded | The record has moved past it, so an artifact built now is not the one asked for | | Stale | The ask waited more than twelve hours, and the record may have changed underneath it | | Partial | A lane does not resolve, which the emitter already refuses to write around | Out of scope is checked before anything else about the decision, so an ask that names another organisation's decision learns only that it is not theirs. Superseded is why a request names a decision at all. The caller is saying "as of this one", and if the standing set has changed, the honest answer is to ask again rather than to receive a document that is quietly behind. An ask that names no decision means whatever stands right now, and cannot be superseded. ## A refusal is a row Every answered ask writes to `aperture.artifact`, which is append-only. A signed row carries the canonical bytes, their digest, the decisions that went into them, the key that signed and the signature. A refused row carries the reason and nothing that could be mistaken for an artifact. Refusals are stored for the reason the record stores everything else: a table of successes answers what was signed and not what was asked and declined, and the second question is the one an incident asks. ## The key that signs this is not the key that signs the record Until this existed, one Ed25519 key signed both each day's Merkle root and every config and policy emitted for a customer. While both ran from a timer nobody outside could reach, that was untidy. The moment signing happens on a customer's ask it is a category error: a compromise of the customer-facing signer would forge the public record's roots rather than one customer's routing. So there are two keys, in two files, with one reader each, and no unit reads both. `aperture:sign` refuses to start if its own key is not registered with the record to sign policies, so an artifact a customer could not verify is never produced in the first place. ## Registering a key, and the order that matters ```bash npm run witness:key:register -- --purpose policy --key <public half> npm run witness:key:register -- --list npm run witness:key:retire -- --key <public half> --reason "rotated 2026-09-01" ``` Registering publishes a key. It does not start signing with it, and the difference is an outage: 1. register the incoming key, so the record acknowledges and publishes it 2. operators add it to their runtime's `VITNA_PUBLIC_KEY`, beside the key they already pin, and restart 3. only then does the private half go where the signer reads it `VITNA_PUBLIC_KEY` accepts a comma-separated set from `vitna-runtime` 0.1.2 onwards, which is what makes step 2 a restart nobody has to hurry. This page said 0.2.0 until 2026-09-13; the set was released in 0.1.2, and `platform/runtime/policy-store.mjs` in that package is what parses it. Doing 3 before 2 signs policies every deployed runtime refuses, and each operator has to edit a file before their lanes come back. The whole order, retirement and a compromise included, is [rotating a signing key](../operations/key-rotation.md). What is registered at any moment is published at [`GET /api/v1/keys`](../api/v1/verification/keys.md), purpose first, and on `/witness`. Retiring is a date and a reason, never a delete. An artifact signed last year has to stay checkable against the key that signed it, and a verifier that could not tell a retired key from a key that was never Vitna's would have to call both of them forgeries. ## What it does not do **It is not an endpoint and must not become one.** It is a timer unit reading a table. Nothing in a request path calls it, nothing waits on it, and a customer collects what it wrote afterwards. An inference request still never calls vitna.ai. **Collecting it is a separate page.** The artifact sits in the table, and an owner asks for one at `/console/deployment`. Anybody signed in to the organisation downloads it from the same page, which prints beside each signed policy the two lines a runtime needs, with the key that signed that file. A machine with no session uses [Collecting a signed artifact](../api/v1/engine/artifacts.md), an authenticated read with the organisation's own key. **It ordinarily answers within about six minutes.** `deploy/vitna-sign.timer` fires every five minutes, on the twos and sevens, with up to thirty seconds of jitter, so an ask made just after a run waits for the next one. A refusal lands on the same schedule, with its reason. That is the schedule, not a promise: an attempt that fails is retried after five minutes and then twenty five, so an ask can wait half an hour or more and end refused. The console says so beside an ask that has waited longer than six minutes. **It signs nothing until a key is registered to sign policies.** The signer starts only with a key the register lists for that purpose, so on an instance where [`GET /api/v1/keys`](../api/v1/verification/keys.md) answers `"policyKey": null` nothing is signed, the console says so instead of the six minutes, and it takes no ask: one would answer nothing, block the next ask for that organisation, and be refused as stale if the signer started more than twelve hours later. vitna.ai was in that state on 2026-09-13. A registered key is what signing needs rather than proof that it runs; the signer also needs the private half and its database login on the box, which no page can see. **It signs policies only.** A config is a kind the queue accepts and this refuses by name, so an ask for one is answered rather than left pending. --- <!-- https://vitna.ai/docs/routing/stackfile.md --> # Declarative Stackfile and CI policy lock Check your organisation's workload register into git as `vitna.stack.json` (`vitna.stackfile.v1`), preview and apply register changes deterministically, and gate deployments in CI with `vitna-runtime-lock` and `vitna.lock.json`. ## Why two files A Stackfile (`vitna.stack.json`) declares what your team intends: workloads, unit profiles, quality and latency floors, data-handling boundaries (`requiresZeroRetention`, `requiresNoTraining`, `requiresRegions`, `allowsProviders`), private infrastructure rates, candidate pools, fallback chains, shadow rules, and watches. A [signed policy](policy.md) (`policy.json`) records what accepted rulings currently authorise `vitna-runtime` to serve. Keeping the two separate preserves the decision boundary: editing `vitna.stack.json` updates the register and its constraints, while routing never moves until a decision is accepted and a signed policy is emitted. `vitna.lock.json` (`vitna-policy-lock-v1`) binds the two together in git so CI refuses any pull request where `vitna.stack.json` tightened a boundary that `policy.json` does not satisfy, or where `policy.json` changed without updating the lockfile. ## Stackfile format ```json { "schema": "vitna.stackfile.v1", "org": "convexity", "privateSkus": [ { "slug": "vllm-h100-prod", "name": "vLLM H100 cluster", "classId": "model", "status": "active", "notes": "Dedicated inference nodes", "rates": [ { "dimension": "input_per_mtok", "amount": 0.18, "from": "2026-09-01", "notes": "Amortised cluster rate" } ] } ], "workloads": [ { "slug": "deep-research", "name": "Deep research synthesis", "classId": "model", "status": "active", "priority": "quality", "unitLabel": "briefs", "incumbentSkuId": "anthropic/claude-opus-4-8", "monthlyVolume": 12000, "unitProfile": { "inputTokensPerUnit": 24000, "outputTokensPerUnit": 3200 }, "constraints": { "qualityFloor": 92, "economicThresholdPct": 15, "requiresZeroRetention": true, "allowsProviders": ["anthropic", "openai"] }, "delivery": { "envKey": "MODEL_DEEP_RESEARCH", "policyAlias": "deep-research" }, "notes": "Customer-facing synthesis lane", "candidates": [ { "skuId": "openai/gpt-5.6-sol", "status": "watching" } ], "fallbacks": [ { "skuId": "anthropic/claude-sonnet-5", "notes": "Same-provider failover" } ], "rules": [ { "kind": "traffic_share", "share": 5, "skuId": "openai/gpt-5.6-sol", "notes": "Shadow canary" } ], "watches": [ { "kind": "price_move", "thresholdPct": 10, "direction": "down" } ] } ] } ``` Every field in `vitna.stackfile.v1` is validated by the same pure functions in `platform/engine/register.mjs` that guard `/console`, the CLI, and [`/api/v1/org/stackfile`](../api/v1/engine/org-stackfile.md). Unknown fields, negative profiles, out-of-range thresholds, duplicate lane aliases, and malformed identifiers fail immediately with the register's own refusal messages. A Stackfile may also carry an optional top-level `"pipelines"` array declaring multi-stage workflows (`slug`, `name`, `runLabel`, `monthlyRuns`, and ordered `stages` referencing declared `workloadSlug` entries with `invocationsPerRun` and `criticalPath`) for offline Bill of Materials evaluation and CI spend budget gates via [`aperture:pipeline`](../cli/aperture/pipeline.md). ## Exporting, planning, and applying from a checkout ```bash DATABASE_MIGRATE_URL=... npm run aperture:stackfile -- --export --org convexity --out vitna.stack.json npm run aperture:stackfile -- --validate vitna.stack.json DATABASE_MIGRATE_URL=... npm run aperture:stackfile -- --plan vitna.stack.json DATABASE_MIGRATE_URL=... npm run aperture:stackfile -- --apply vitna.stack.json ``` Reconciliation never deletes behind your back: - Workloads in the database that are omitted from `vitna.stack.json` stay untouched and are listed under `unmanaged.workloads`. Retire or pause a lane explicitly by setting `"status": "paused"` or `"status": "archived"`. - New lane rules declared in `vitna.stack.json` enter in `shadow` mode, matching `aperture:rule:add`. Promoting a rule to `route` requires an accepted decision citing shadow readings (`aperture:rule:promote`). - Existing armed watches omitted from `vitna.stack.json` stay armed and are reported under `unmanaged.watches` unless `--disarm-reason <note>` is supplied. ## Offline CI verification (`vitna-runtime-lock`) `vitna-runtime` ships `vitna-runtime-lock` with zero external dependencies and zero network calls. Pin `VITNA_PUBLIC_KEY` in CI secrets and run: ```bash VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ vitna-runtime-lock \ --policy policy.json \ --stackfile vitna.stack.json \ --write vitna.lock.json ``` Commit `vitna.lock.json` alongside `vitna.stack.json` and `policy.json`, then enforce it on every pull request: ```bash VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ vitna-runtime-lock \ --policy policy.json \ --stackfile vitna.stack.json \ --lockfile vitna.lock.json \ --max-age-hours 168 ``` From a checkout of this repository, the same command is available as an operator script: ```bash VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ npm run runtime:lock -- --policy policy.json --stackfile vitna.stack.json --lockfile vitna.lock.json ``` `vitna-runtime-lock` exits with code `3` when any invariant breaks: - `invalid_policy`: unreadable JSON, digest mismatch, untrusted signing key, or invalid Ed25519 signature. - `org_mismatch` / `lock_org_mismatch`: `policy.json`, `vitna.stack.json`, and `vitna.lock.json` disagree on the organisation slug. - `stale_policy`: `policy.json` is older than `--max-age-hours`. - `missing_lane` / `missing_required_lane`: an active workload's `delivery.policyAlias` (or `--require-lane`) is missing from `policy.json`. - `inactive_workload_routed`: `policy.json` still serves a workload marked `paused` or `archived` in `vitna.stack.json`. - `unapproved_provider` / `unapproved_fallback_provider` / `unapproved_rule_provider`: a primary model, fallback, or rule uses a provider outside `constraints.allowsProviders`. - `substrate_leak`: a lane, fallback, or rule carries an OpenRouter route while the workload requires zero retention, no training, regional residency, or excludes `openrouter` from `allowsProviders`. - `fallback_drift` / `lane_drift` / `policy_digest_drift` / `stackfile_digest_drift`: `policy.json` or `vitna.stack.json` moved without updating `vitna.lock.json`. --- <!-- https://vitna.ai/docs/routing/usage-import.md --> # Importing usage `aperture:usage:import` reads a consumer's own usage export back into the record, so it can say what actually served a workload. ```bash npm run aperture:usage:import -- \ --workload convexity/conversational --file ./august.csv \ --window 2026-08-01..2026-08-31 --source-ref "console export, 1 Sep" ``` ## Flags | Flag | Meaning | |---|---| | `--workload <id>` | Whose usage this is | | `--file <path>` | The export | | `--window a..b` | Calendar dates, inclusive at both ends | | `--source-ref <text>` | Where the export came from. Required. | | `--format json\|jsonl\|csv` | The file's format, guessed from the extension | | `--no-adopt` | Read the window without moving the workload's incumbent | ## It fetches nothing The export is a file somebody made. `--source-ref` is required for the same reason a measurement requires one: a reading with no provenance is a number. `jsonl` is the one format this repository also writes: a [runtime ledger](../runtime/ledger.md) is read by this command unmodified, with the same two fields and the same arithmetic as a console export. ## Units are the workload's own units Deliberately unclever about what an export contains. Units are reduced by whoever ran the export, because only they know whether a request is a unit or a hundred tokens are. Two columns are required and the rest ignored: the model as the provider names it, and the units it served. A row with no model, no units, or units that are not a whole number is a refusal naming the row. A usage file is somebody's afternoon of exporting, and a silently dropped row is a share that is quietly wrong. A third column, `provider`, is read where the export carries one and is never inferred where it does not. A blank cell and a missing column say the same thing, which is that this row did not say. Nothing is guessed from the model's own vendor: a model served through a substrate is served by somebody other than whoever trained it, which is the entire reason the column is worth having. A runtime's own ledger has written the field on every line since R1, so a `.jsonl` import carries providers without anybody adding a column. ## Windows `2026-08-01..2026-08-31`, inclusive at both ends, which is how a billing period reads and how anybody says "the first to the last". A window that ends before it starts is refused. ## Shares, and when a window is unambiguous Models are totalled, mapped onto skus where they are recognised, and each sku's share of the window computed. One sku must hold **95 percent** of a window before it is called what the workload is running. Not 100: a lane with a cross-vendor fallback answers a few calls elsewhere in any real month, and a rule that only fires on a clean sweep would never fire. ## Read against the lane's route, where there is one The threshold says whether one sku swept the window. It cannot say whether what else answered was supposed to, and until the import read the lane's published [route](fallbacks.md) those were the same question. A declared fallback holding 3 percent passed because 3 is under 5, and an undeclared sku holding the same 3 percent passed for exactly the same reason. They are opposite facts. A lane with a chain is now read against it as well: | What the window shows | What the import says | |---|---| | Only the primary and its declared fallbacks | The route was honoured, with each fallback's share named | | A sku the route does not name, at any share | A finding, whatever the share, because production is running something no decision covers | | A declared fallback leading the window | A finding about the primary, which is the model a decision named and the one meant to serve | None of this changes when the incumbent moves. Adoption still needs one sku to hold the window and to be the sku the standing decision named, so a finding can stand beside a confirmation: a primary that swept its window did sweep it, and an undeclared sliver beside it is a separate fact worth printing. A workload with no chain is read exactly as it was before chains existed. An environment target declares no fallbacks, and asserting a route for it would be judging a window against a policy nobody published. ## A model the registry does not hold Gets no row, and its units still count toward the window total. So a workload running something Vitna cannot price shows up as rows that do not sum to their window, rather than as a clean sweep. The denominator is stored on every row rather than derived, so one row still says what share it was, and a later import cannot quietly change what an earlier reading meant. ## What gets written One `aperture.deployment_observation` row per sku per window: | Column | Meaning | |---|---| | `workload_id`, `sku_id` | What ran, and for whom | | `window_start`, `window_end` | The period, inclusive | | `units`, `window_units` | This sku's units, and every unit the export carried | | `confirms_decision_id` | The accepted decision this reading confirms, where it confirms one | | `source_ref` | Where the export came from | | `imported_at`, `note` | When it was read, and any remark | | `provider` | Who served this sku's units, where the export named one provider for all of them. Null where it named none, where it named more than one, or where a runtime pushed the reading | | `origin` | `import` here; `runtime` on a reading [a runtime pushed](../api/v1/engine/usage.md) | Constraints: the window is ordered, units are positive, units fit inside the window total, and one reading exists per sku per window. Importing the same export twice is the same fact and must not become two readings. A null `confirms_decision_id` is the ordinary case for a workload nobody has ruled on lately, and is not a defect. ## Reading a window back at list price ```bash npm run aperture:reconcile -- --workload <id> --window 2026-08-01..2026-08-31 npm run aperture:reconcile -- --workload <id> --window 2026-08-01..2026-08-31 --spend bill.json ``` What an imported window cost at the tape's own prices, printed and written nowhere. Each sku's units are priced from the price periods that stood over the window and the workload's unit profile; a sku the record cannot price across the whole window is counted as coverage rather than costed at zero, and a window whose tape moved is a bound rather than a guess, because a reading is a window total and the record does not know how the units fell around the move. `--spend` takes lines off the bill, as JSON: `{ "amountUsd": 240 }` for the whole window, or with a `"sku"` for one row of it. Each comparable line prints the implied discount off list. A whole-window line is refused unless every unit in the window is priced, because a discount computed against part of the bill is a number that only looks like one. Nothing from a spend file enters the record: a negotiated price is [not a market](../concepts/source.md), and this is how it stays on the customer's side of the line. The workload page at `/console` renders the same reconciliation of the newest window, from the same module, so the page and the command cannot disagree. ## No new privileges Written by this command and by `aperture:workload:edit` as the owning role. No application role is granted anything, so the register still holds no `DELETE` and the site reads these rows only to put them on `/console`. --- <!-- https://vitna.ai/docs/routing/verifying.md --> # Verifying a routing config Checking a file you were handed, without needing the database or the key that made it. ```bash npm run aperture:config -- --verify /etc/convexity/models.env ``` ## What it checks without anything The digest is recomputed from the file's own canonical body, and the signature checked over the statement the file carries. Neither needs a database nor a signing key, so a consumer can verify a config on a machine that has neither. ## The part a signature does not settle A valid signature proves that whoever holds that key signed this file. It does not prove the key is Vitna's. So the command says so out loud. Without a database it reports the key as **not checked**, and points at the key published on `/witness` and `/api/v1/attestations` to compare against. With `DATABASE_URL` set it compares the key against the one the record publishes, and only then reports that this is Vitna's signature and not a well-formed forgery. If the record has signed nothing yet, it says that too, rather than reporting a check it did not make. ## The same distinction the attestations make This mirrors what [`attestation`](../concepts/attestation.md) says about the daily roots. A signature is a statement about custody of a key, and the step that ties a key to Vitna is published separately and has to be done. The signing key here is the same one the daily attestation uses. `npm run witness:attest -- --new-key` prints a pair. --- <!-- https://vitna.ai/docs/routing/watchdog.md --> # Rollout and decision watchdog Decision-oriented observability over live workloads and active lane rules: whether current prices, compliance terms, three-provenance capabilities, and runtime telemetry still support the standing decision, and where each `shadow` or `route` rule stands on the canary ladder. ## The two questions it answers Ordinary runtime metrics report request counts and latency percentiles without knowing what ruling authorized the lane or what assumptions that ruling relied upon. The watchdog (`platform/engine/watchdog.mjs`) joins the routing record back to the decision ledger to answer two concrete operational questions: ### 1. Does reality still support the standing decision? - **Price assumption drift**: compares current Witness or entered unit cost against the unit cost pinned in `standingDecision.inputs` (`pricePeriodIds` and `enteredPriceIds`), flagging moves above `maxPriceDriftPct` (default `+10%`). - **Compliance and capability eligibility**: re-evaluates the serving SKU against `witness.provider_term` (`requiresZeroRetention`, `requiresNoTraining`, `requiresRegions`, `allowsProviders`) and [three-provenance capability bounds](capabilities.md) (`requiresCapabilities`, `requiresVerifiedCapabilities`, `minContextTokens`, `minThroughputToksSec`, `maxTtftMs`). - **Quality and critical failures**: checks newest `aperture.evidence` measurements against `qualityFloor` and `edgeCaseVeto`. - **Operational reliability and latency**: checks fallback chain reliance (`fallbackSharePct <= maxFallbackSharePct`, default `10%`), lane unserved attempt rate (`unservedRatePct <= maxUnservedRatePct`, default `5%`), and observed latency against `constraints.latencyP90Ms`. ### 2. Where does each lane rule stand on the canary ladder? ``` watching -> shadow -> 5% canary -> 25% -> 50% -> accepted incumbent ``` For each active rule in [Rules within a lane](rules.md), the watchdog inspects its authority (`underDecisionId` and `promotedDecisionId`), candidate eligibility, evaluation evidence, and runtime telemetry (`shadowRule: "rule:<n>"` would-have matches and `serve: "rule:<n>"` routed canary attempts): | Rule verdict | Mode | When emitted | |---|---|---| | `hold_shadow` | `shadow` | Accumulating shadow matches (`< minShadowMatches`), awaiting evaluation evidence, or awaiting a post-shadow decision (`standing.decidedAt < rule.addedAt`) | | `promote_ready` | `shadow` | Shadow volume, eligibility, quality floor, post-shadow standing decision, and closed `readingsWindow` + `sourceRef` citation are all present | | `retire_shadow` | `shadow` | Target SKU fails workload eligibility, falls below `qualityFloor`, or hits a vetoed critical failure | | `healthy_canary` | `route` | Routed canary is serving within error, latency, compliance, and quality bounds while accumulating `minCanaryUnits` | | `expand_or_adopt_ready` | `route` | Routed canary cleared `minCanaryUnits`, `qualityFloor`, and economic threshold; ready for the next nested FNV-1a share rung (`5 -> 25 -> 50`) or full lane adoption | | `rollback_recommended` | `route` | Routed canary breached `maxUnservedRatePct`, `latencyP90Ms`, `qualityFloor`, `edgeCaseVeto`, or compliance/capability eligibility | | `stale_authority` | either | Rule's authorizing decision was superseded by a newer accepted decision on the lane | ## Epistemic boundary and zero silent mutation Two boundaries hold across the CLI, the org endpoint, and the stateless endpoint: - **Availability is never mistaken for answer quality.** Runtime unserved counts (`failed + refused + timed_out`) measure transport and provider availability (`unservedRatePct`). Answer quality (`successRatePct`) enters exclusively from recorded `aperture.evidence` evaluations. - **Zero silent mutation.** Neither [`GET /api/v1/org/watchdog`](../api/v1/engine/org-watchdog.md), [`POST /api/v1/watchdog`](../api/v1/engine/watchdog.md), nor [`aperture:watchdog`](../cli/aperture/watchdog.md) modifies `aperture.lane_rule` or `aperture.decision`. Instead, every actionable verdict emits the exact copy-ready operator command (`aperture:decide`, `aperture:rule:promote`, `aperture:rule:add`, or `aperture:rule:remove`) so every routing change resolves to an explicit, dated ruling. --- <!-- https://vitna.ai/docs/cli/overview.md --> # The CLI Everything an operator does to the record is a command, run from a checkout with the environment sourced. There is no admin UI, and that is a property of the design rather than a gap. ## Why the record is operated and not administered Every write that changes what the record says is an act with a person behind it, made through a role that holds the privilege. Every `UPDATE` grant in this database is column-level and there are two in `aperture`: the collector marks a trigger fired, and the console records a ruling on a decision. Neither `witness` nor `aperture` grants `DELETE` at all. So there is no endpoint that could quietly accept a decision. The one endpoint that accepts one is `/console/rule`, it does nothing quietly, and it cannot touch a decision's outcome, thresholds, inputs, reason or brief, because those columns are not in its grant. Everything else about the register is entered at a prompt, as the owning role, which is what the rest of this page is. ## The families | Family | Count | What it touches | |---|---|---| | [`witness:*`](witness/seed.md) | 5 | The record: seeding, collecting, replaying, signing, publishing rates | | [`event:add`](event-add.md) | 1 | The market log | | [`sku:*`](skus/candidates.md) | 4 | The discovery queue | | [`aperture:*`](aperture/workloads.md) | 24 | Workloads, evidence, triggers, decisions, compound pipelines, audit packs, fallback chains and rules | | [`org:*`, `auth:*`](access.md) | 4 | Which orgs exist and who may read `/console` | | [`org:key:*`](keys.md) | 2 | Who may land a reading over the network | | [`db:*`](database.md) | 2 | Migrations | | [`scripts/`](scripts.md) | 5 | Mirrors and tooling | ## Shared conventions **A dry run writes nothing.** `--dry-run` where offered, and several commands report rather than write unless given `--yes`. **A verify recomputes.** `--verify` on the signing and rate commands recomputes every row ever published and reports disagreement rather than trusting. **Refusals name the row.** A command that cannot do what it was asked says which input it refused and why, rather than doing part of the job. **Nothing invents data.** A command with a missing credential or an unreachable source fails and says so. It does not fall back to a plausible number. --- <!-- https://vitna.ai/docs/cli/access.md --> # Access commands Which orgs exist, who may read them at `/console`, and how a session is opened or revoked. ```bash npm run org:create -- --id <slug> --name "<name>" npm run org:agree -- --org <id> --note "<invoice or agreement reference>" npm run org:member:add -- --org <id> --github <login> [--role member] npm run org:member:remove -- --org <id> --github <login> npm run auth:session:create -- --github <login> [--github-id <n>] ``` ## An org that arrived by agreement is created, not opened Two doors make an org. `/console/signup` opens one for the account that submits the form, records that account as its opener, and the register meters it against its plan's allowance. `org:create` is the other: an operator at the prompt, as the migrate role, creating an org somebody agreed terms for. ```bash npm run org:create -- --id acme --name "Acme Ltd" ``` It leaves `aperture.org.opened_by_user_id` null and has no flag that could set it, and that null is what makes the org unmetered: the register refuses it no workload, the sixth or the fifty-first, so the number it may hold is the agreement's and nothing in the code enforces it. There is no plan id and no subscription row, because an agreement is not a subscription. `/console/billing` shows such an org as on agreement terms: no plan, no allowance count and no checkout, and a sentence saying the register does not meter it. `/console/checkout` refuses a checkout for it. It writes the org and nothing else. The first owner has to sign in once before a membership can be keyed on their account, so the command prints the `org:member:add` line to run when they have, with only the login to fill in. An id that already exists is refused (exit 3) and nothing is written, including the name, which `org:create` never changes. The refusal says which door made the org: an operator's needs only a member added, and a self-service org cannot be put on agreement terms by creating it again, so its refusal names `org:agree`. A malformed or reserved id is refused before the database is asked (exit 2), by the same rules the signup form uses. ## A self-service org that signs an agreement is moved, not re-created ```bash npm run org:agree -- --org acme --note "Invoice INV-2026-014" ``` `org:agree` records two facts on an org that was opened at `/console/signup`: when it moved onto agreement terms (`aperture.org.agreement_recorded_at`, the database's clock) and the invoice or agreement reference (`agreement_note`). It runs as the migrate role, and no web role can write either column. The org keeps its opener, so its account still cannot open a second org. The register meters an org only where it has an opener and no recorded agreement, so from that instant the org is unmetered in both directions: the register no longer refuses the workload its plan would have, and it will not refuse the fifty-first either. The report prints what changed, column by column, and says so. `/console/billing` then shows the org on agreement terms since that date, with the note. It cancels nothing at the processor. If the org still holds a plan, the report names it, and the billing page keeps its owner's button to the processor's own page so it can be cancelled. Exit 2 is flags refused before the database is asked: a missing `--org` or `--note`, a malformed id, or a note over 200 characters or carrying a control character. Exit 3 is an org the record refuses, with nothing written: one that does not exist, one an operator created (already on agreement terms), or one already moved, whose date and note stay as they were. There is no flag to backdate the date or amend the note. ## Signing in is not being allowed in `/console` is behind GitHub OAuth, and authentication only establishes who someone is. An account with no membership sees a page saying so, and the data of no org. ## A membership is keyed on the account id `org:member:add` refuses a login that has never signed in. GitHub logins are renameable and reusable. Keying a grant on a name would mean a renamed account keeps access it was not given, and a freed name inherits access somebody else had. So the account must have signed in once, which is what gives Vitna the immutable id to key the membership on. ## Removal applies on the next page load `org:member:remove` takes the access away and leaves the session alone. The session stays valid; it now belongs to somebody who is a member of nothing. Revoking the session too is a separate act, which is the right shape: losing access to one org is not the same as being signed out everywhere. ## Sessions from the box `auth:session:create` opens a session and prints its token once, for operating and for checking the console without a browser. It runs as the migrate role, so it grants nothing that shell access on the box does not already grant. That is the test for whether an operator command is safe to exist. --- <!-- https://vitna.ai/docs/cli/aperture/audit-pack.md --> # Audit attestation pack CLI Export, sign, and verify self-contained `vitna.audit-pack.v1` procurement, FinOps, and model-governance attestation packs from the register, from an offline Stackfile, or in an air-gapped CI runner. ```bash DATABASE_MIGRATE_URL=... npm run aperture:audit:pack -- --org convexity --runs 10000 --out ./audit-pack.json ``` ```bash npm run aperture:audit:pack -- --stackfile vitna.stack.json --policy policy.json --lockfile vitna.lock.json --sign ./pack.key --out ./audit-pack.json ``` ```bash npm run aperture:audit:pack -- --verify ./audit-pack.json --pack-key <base64-spki> --record-key <base64-spki> --policy-key <base64-spki> --require-signed --fail-on-rollback --max-monthly-cost-usd 5000 ``` Nothing is mutated in the database. Register export queries `aperture.*` and `witness.*` strictly read-only, while `--stackfile` and `--verify` open zero database connections and zero network sockets. See [Audit and procurement attestation packs](../../routing/audit-packs.md) for how the six verification layers and cryptographic key boundaries work. ## Modes and flags - **Register export mode** (`--org <slug>`): exports the organisation's canonical `vitna.stackfile.v1`, standing decisions, latest signed `policy.json` artifact (unless `--no-policy` is set), batched RFC 6962 Witness observation Merkle proofs and signed daily roots, customer-entered private SKU rates, provider compliance terms, three-provenance SKU capabilities, rollout watchdog evaluation, and compound pipeline Bill of Materials. - **Offline Stackfile export mode** (`--stackfile <vitna.stack.json>`): builds a deterministic `vitna.audit-pack.v1` with zero database access from a Stackfile, committed Witness market snapshots, and optional `--policy`, `--lockfile`, `--ledger`, `--evidence`, and `--terms` files. - **Standalone offline verification mode** (`--verify <pack.json>`): verifies all six layers of a `vitna.audit-pack.v1` file offline against optional pinned Ed25519 SPKI public keys (`--pack-key`, `--record-key`, `--policy-key`) and CI governance gates. | Flag | Meaning | |---|---| | `--org <slug>` | Export an audit pack for registered organisation `<slug>` | | `--stackfile <file>` | Build an offline audit pack from a `vitna.stackfile.v1` file | | `--verify <pack.json>` | Verify an existing `vitna.audit-pack.v1` file offline | | `--stages <spec>` | Optional comma-separated `<slug>[:<invocationsPerRun>][:bg]` list in `--org` mode | | `--pipeline <slug>` | Select one named entry from `stackfile.pipelines` in `--stackfile` mode | | `--runs <n>` | Monthly pipeline execution count (`monthlyRuns`) | | `--discover-peers` | Include active Witness class-peer exploratory spend levers in pipeline BOMs | | `--ledger <file.jsonl>` | Supply `vitna-runtime` JSONL rows for rollout watchdog and `x-vitna-run` attribution | | `--agent <agentId>` | Filter `--ledger` rows to one `x-vitna-agent` identifier | | `--window YYYY-MM-DD..YYYY-MM-DD` | Restrict observation proofs and `--ledger` rows to a closed UTC date range | | `--evidence <evals.json>` | Supply offline evaluation measurements in `--stackfile` mode | | `--terms <terms.json>` | Supply offline provider compliance terms in `--stackfile` mode | | `--policy <policy.json>` | Embed a signed `policy.json` envelope file (overrides register lookup) | | `--no-policy` | Omit the latest signed `aperture.artifact` policy in `--org` mode | | `--lockfile <vitna.lock.json>` | Embed a `vitna.lock.json` runtime policy lock file | | `--label <text>` | Optional human-readable release or procurement review label | | `--sign <key-file>` | Sign the exported pack with an Ed25519 private key PEM file (or pass `env` to read `SIGNING_PRIVATE_KEY`) | | `--out <pack.json>` | Write the exported `vitna.audit-pack.v1` JSON to `<pack.json>` | | `--pack-key <pub>` | Pin the expected outer pack Ed25519 SPKI public key (base64, PEM, or file path) in `--verify` mode | | `--record-key <pub>` | Pin the expected Witness `record` Ed25519 SPKI public key in `--verify` mode | | `--policy-key <pub>` | Pin the expected routing `policy` Ed25519 SPKI public key in `--verify` mode | | `--require-signed` | Fail `--verify` (`exit 4`) if the outer pack is unsigned | | `--require-attested-prices` | Fail `--verify` (`exit 4`) if any Witness price proof lacks a signed daily Merkle root | | `--fail-on-rollback` | Fail `--verify` (`exit 4`) if `rolloutWatchdog.verdict` is `rollback_recommended` | | `--max-monthly-cost-usd <usd>` | Fail `--verify` (`exit 4`) if `totalEstimatedMonthlySpendUsd` exceeds `<usd>` or is unknown | | `--json` | Emit structured JSON verification report (or print raw pack JSON to stdout when `--out` is omitted) | ## Exit codes | Code | Meaning | |---|---| | `0` | Export succeeded, or `--verify` passed (`verified` or `verified_with_warnings`) | | `1` | `--org` mode requested with neither `DATABASE_MIGRATE_URL` nor `DATABASE_URL` set | | `2` | Invalid flags, unreadable input file, key conflation refusal, or unknown organisation/workload | | `4` | `--verify` failed any cryptographic check, citation-boundary check, or CI governance gate | --- <!-- https://vitna.ai/docs/cli/aperture/bill.md --> # How much of a bill the record can price One window of somebody's spend, bucketed by whether the record holds a figure for it. ```bash npm run aperture:bill -- --window 2026-08-01..2026-08-31 --spend ./august.json ``` Nothing is written. It is a report over the record, exactly as [`aperture:counterfactual`](counterfactual.md) is, and a customer's negotiated rate never enters the record from here. The file is the one `aperture:reconcile` already takes: an array of lines, each carrying `amountUsd` and optionally the `sku` it covers, where a line with no sku is the whole window's stated spend. One file, two questions. The reconciliation asks what a window would have cost at list. This asks how much of it the record can speak to at all. ## Three buckets, never one number | Bucket | What it means | What it is for | |---|---|---| | Priced | Resolves to a sku the record held a figure for across the whole window | The claim | | Declared, unpriced | Falls in a class the record declares and holds no figure for | The roadmap, in the customer's own money | | Outside the record | Names no sku the record carries | The scope question, and sometimes the answer is that it stays outside | The middle bucket carries two different facts and the run says which one every line is. A class the record declares and has never wired is one. A sku the record knows, in a class it does price, that carried no figure over the window is the other. Both are honestly unpriced, and somebody deciding what to wire next needs to know which they are looking at. ## Priced means the whole window, not a corner of it A price period either covers the window or it does not, and the same walk `aperture:reconcile` uses decides it. A rate first read three weeks before a four month window ends overlaps that window and prices almost none of it, so it is not priced here; it is declared and unpriced, with the reason naming how much of the window it did cover. This was wrong until 2026-09-10, and the run that found it is worth keeping. A real bill over 130 days came back at a hundred percent priced, because every sku on it carries an open period that began in late August. The record held figures over about 21 of those days. The test asked whether a period overlapped the window, and the answer to that question is not the answer to this one. So a long window is now legible rather than flattering: the header prints the window's length, a priced line says it was priced over all of it, and a partial line says how many days of how many it actually had. If any line is partial the caveats name the thinnest one, because that is the number a reader would otherwise never meet. A bill line names a sku and not a meter, so coverage is asked of each of that sku's dimensions and the best-covered one answers. A sku whose input rate spans the window and whose batch rate does not is one the record can price across the window. That is a claim about the sku, and on 2026-09-10 the same bill showed what it does not claim. It carried $18.44 of cache-write spend and reported it priced, correctly by this rule, because the sku it was spent on is one the record prices. The record held no cache-write meter at all. Nothing was miscounted and the fix was not here: the [dimension](../../concepts/dimension.md) was registered instead, and the model class carries two of them now. What the run still does not do is say when a bill's own vocabulary is finer than the record's, which is an open call in `CAPTURE.md` rather than a defect with a patch waiting. ## The denominator is the whole design A coverage figure computed over only the lines somebody itemised would flatter itself exactly where it matters, so the run states what it divided by: - A bill that **states a window total** is measured against that total, and the difference between the total and the lines is printed as **unitemised**. That money is neither priced nor unpriced, because nobody said what it bought. - A bill that **states no total** is measured against the sum of its own lines, and the run says so in as many words. If those lines are a subset of the invoice, the figure is higher than the invoice's own would be. The percentage travels inside the sentence rather than above it, because that sentence is the one that gets pasted into a message, and a coverage figure that travels without the money it was taken over is the failure this command is shaped to avoid. This is also why a percentage is allowed here and refused on [/coverage](https://vitna.ai/coverage). A share of one bill has a denominator the reader is holding. A share of the AI market has none that anybody publishes. ## What it refuses | It refuses | Because | |---|---| | A bill stating two window totals | Two denominators, and no answer uses both | | A negative line | A credit is not spend, and netting one against the total hides what was bought | | A line with no `amountUsd` | A line that is not money cannot be a share of one | | Lines exceeding the stated total | It reports the overrun rather than clamping it, since a clamp would print a share against a total nobody wrote | An unknown sku is not a refusal. It is the third bucket, which is the point. ## It takes no workload, and it needs a database No workload, deliberately. The reconciliation needs one because it weighs units against prices and units come from an imported usage export. A bill needs none: the question is what the record prices, and the answer is the same whether or not anybody has registered the work that produced the spend. That is what lets this run against the first bill a prospect sends rather than only against a customer already through the loop. A database is required and there is no fallback to the committed snapshot, because the question is dated. "Priced over your window" is a claim about what the record held in that month, and the snapshot holds today's figures with today's read date. Answering from it would be a coverage figure quietly computed against the wrong month. --- <!-- https://vitna.ai/docs/cli/aperture/counterfactual.md --> # What it would have cost somewhere else One window of a workload, priced on a sku that did not serve it. ```bash npm run aperture:counterfactual -- --workload convexity/deep-research --window 2026-08-01..2026-08-31 --candidate openai/gpt-5.6-luna ``` Nothing is written. It is a report over the record, exactly as [`aperture:reconcile`](../../routing/outcome.md) is. ## Every figure in it is real The units were counted by whatever served them. The prices were read off a provider's own document, archived and dated. The arithmetic is the reconciliation's, so a price that moved inside the window makes both figures bounds rather than points, and a range that straddles zero is printed as one rather than resolved into a saving that may not exist. ## It runs nothing, so it scores nothing **No observed quality claim without an observed candidate response.** That sentence is the whole boundary, and the last caveat on every run says it in those words. No request was sent to the candidate, so nothing here says whether it would have answered as well, held its format, or called a tool successfully. A score for a request nobody answered has no source, and this record does not carry one. A **replay** is the other thing and this is not it. A replay sends the saved requests to the candidate, so a candidate really answers and the score has a run behind it. That needs a capture the runtime does not perform: the ledger records metadata and holds no prompt or response, by design, so there is nothing on any disk to replay. See `BUILD.md` for where that sits. ## What it refuses | It refuses | Because | |---|---| | A window that served no units | There is nothing to have cost anything anywhere | | A candidate that already served part of the window | That is a reading, not a counterfactual | | A candidate the record cannot price across the whole window | A gap priced as zero would be the interpolation the tape refuses | | A candidate from another market | A unit profile spends the dimensions of one market | ## The caveat that is easy to miss A window served by more than one sku is priced here as though one sku had served all of it, and the run says so. **A split happens for reasons a price does not know about.** A lane that split because one model could not answer some of the work would not have been cheaper on that model; it would have failed on it. --- <!-- https://vitna.ai/docs/cli/aperture/decide.md --> # Deciding Running the engine, and ruling on what it proposes. ```bash npm run aperture:decide -- --workload <id> --candidate <sku> npm run aperture:accept -- <decision-id> npm run aperture:reject -- <decision-id> --reason "..." ``` ## Deciding writes a proposal and nothing else `aperture:decide` runs against the live tape and the recorded evidence, prints the brief, and writes a **proposed** decision. Nothing is accepted and nothing is routed. A fired trigger is a question put to a person, and this is the command that formulates the question. ## The engine is pure `decide.mjs`, `brief.mjs` and `triggers.mjs` import nothing and read no clock, so the Decision Lab on the site, this command, and the collector all run the same code against the same inputs. A brief you read on the site is the brief the CLI would print. ## Accepting `aperture:accept` accepts a proposal and arms the next review, superseding the decision it was taken against. > [!WARNING] > Accepting is refused if the record moved under the proposal. A brief argued > against a tape that has since changed is a stale argument, and accepting one > would put a decision into force on reasoning that no longer holds. Accepting does **not** move the workload's incumbent, because whether traffic actually moved is not something Vitna can see from here. That gap is what [usage attribution](../../routing/usage-import.md) closes. ## Rejecting Takes a reason, kept on the row. No acceptance is written. A refused proposal stays in the register with why it was refused, so the next person to ask the same question can see that it was already asked. Refusing also puts back the watch that produced the proposal. A firing spends its watch, so without that a refusal would quietly end the watching: a workload watched for a price move that refused the resulting proposal would stop being watched for price moves. Only the watch that fired comes back, because the ones armed beside it never fired and are still armed. A review date comes back on a new date rather than the one that has passed, which would otherwise fire again on the next collector pass. The command prints what it re-armed, or that there was nothing to put back when the proposal came from a hand-run rather than a firing. ## Ruling from /console An owner of the org can accept or refuse a proposal on [/console](../../routing/console-ruling.md) instead of at a prompt. Both surfaces rule on the same record through the same refusals, in `platform/engine/ruling.mjs`, so a proposal the record moved under is refused identically either way. The console arms the defaults and offers no flags. `--review-on` and `--price-move-pct` stay here, because a threshold typed into a web form without the tape in front of you is a number chosen to fill a field. A ruling made at this prompt records `ruled_at` and leaves `ruled_by` null, because its attribution is who has shell access on the box rather than a signed in account. Both are set for a ruling made at /console. ## Inputs are published Every decision names the exact price period and evidence ids the run read, all append-only, so a brief can be regenerated and checked at any distance. See [`/api/v1/proposals`](../../api/v1/engine/proposals.md). --- <!-- https://vitna.ai/docs/cli/aperture/evidence.md --> # Candidates and evidence Putting a sku in play, and recording what was measured about it. ```bash npm run aperture:candidate:add -- --workload <id> --sku <sku> [--status watching] npm run aperture:evidence:add -- --workload <id> --sku <sku> --kind eval --label "..." --measured-at <date> --n <int> --metric successRatePct=94.0 --source-ref <url> ``` ## Evidence requires a source reference A URL or a commit. A measurement with no provenance is a number somebody remembers, and a [brief](decide.md) is only as checkable as the rows it cites. `--n` is the sample size, recorded because a success rate over eleven runs and one over eleven thousand are not the same claim. ## A metric the engine never reads is refused Not stored and ignored. Refused. A recorded metric nothing consumes looks like evidence and functions as decoration. Worse, it makes a workload look better evidenced than it is, which is exactly the misreading a decision register exists to prevent. ## A run through the runtime can write one Where a workload runs on a [Vitna runtime](../../runtime/overview.md), the [ledger](../../runtime/ledger.md) already holds what each request cost in time and tokens. This reads it back and writes one measurement per sku it served, through the same refusals a typed measurement passes. ```bash npm run aperture:evidence:from-run -- --workload <id> --file /var/lib/vitna/ledger.jsonl --label "Harness run, 10 Sep" [--lane <alias>] [--window 2026-09-10..2026-09-10] [--dry-run] ``` It writes `latencyP90Ms`, nearest-rank over the requests that were served, and `n`, the units served. Both rows from one run carry the same `--source-ref`, derived as `run:<file>@sha256:<digest>` over the exact rows the figures came from, so a [brief](decide.md) that weighs the two can say they were observed together. Each sku's row is labelled with the run's label and its own id, and the whole run enters in one transaction or not at all. It also prints what a unit cost, which is the observed tokens spent against the price record rather than read off a vendor's bill, and the [`workload:edit`](workloads.md) line that enters the profile it observed. ## It measures, and it does not grade Vitna is not an eval platform, and this is where that boundary is load-bearing rather than a slogan. A run yields a latency and a sample size because those are things the ledger observed. It never yields `successRatePct`. The tempting number is the share of requests that came back, and it is the wrong number: a refused or failed request is a request nobody served, not a wrong answer. An outage and a bad answer are not the same finding and only one of them is a quality problem. So refusals and failures are counted, written on the row in words, and never turned into a metric, and a sku that served nothing at all yields no measurement rather than a rate of zero. `successRatePct` keeps arriving from wherever the team already measures quality. ## Importing from CI eval harnesses Teams already running Promptfoo, Braintrust, or UK AISI Inspect AI in CI can POST raw JSON outputs directly to [CI evaluation evidence](../../api/v1/engine/org-evidence.md) with `adapter` set to `promptfoo`, `braintrust`, or `inspect`, alongside `vitna` for native payloads. Every adapter funnelled through `evidence.add` enforces the same metric whitelist (`successRatePct`, `latencyP90Ms`, `edgeCaseFailures`, `verifiedCapabilities`), requires a provenance source reference, rejects future dates, and keeps append-only label uniqueness (`409 duplicate_evidence` on a re-submitted label). ## Evidence is append-only There is no edit and no delete. A measurement that turned out to be wrong is superseded by a later one, and both stay, because which one the engine read at the time is part of what makes a past decision checkable. --- <!-- https://vitna.ai/docs/cli/aperture/pipeline.md --> # Compound pipeline CLI Compute a multi-stage, multi-class AI pipeline Bill of Materials, attribute empirical `x-vitna-run` trace fan-out from a runtime ledger, discover eligible lower-cost Witness class peers, and enforce per-run or monthly spend budgets in CI. ```bash DATABASE_MIGRATE_URL=... npm run aperture:pipeline -- --org convexity --stages web-search:6,doc-parse:10,deep-research:1 --runs 10000 --discover-peers ``` ```bash npm run aperture:pipeline -- --stackfile vitna.stack.json --pipeline research-report --ledger ./ledger.jsonl --max-run-cost-usd 0.15 ``` Nothing is written to the database or disk. Every actionable optimization prints the exact operator command (`aperture:decide`, `aperture:candidate:add`, `aperture:evidence:add`, or `aperture:rule:add`). See [Multi-class unit profiles and compound pipelines](../../routing/pipelines.md) for how stage fan-out, provenance splits, and optimization buckets are derived. ## Modes and flags - **Register mode** (`--org <slug>`): reads active workloads, standing decisions, registered candidates, `evidence` rows, live Witness and customer-entered private rates, provider compliance terms, and three-provenance SKU capabilities. Pass `--stages <slug:calls,...>` to define stage order and fan-out, or omit `--stages` to evaluate all active workloads in the organisation. - **Offline Stackfile mode** (`--stackfile <vitna.stack.json>`): runs with zero database connection for CI checks and architecture planning, reading pipeline definitions from `stackfile.pipelines` (or `--stages`) against the committed Witness market snapshot and any `privateSkus` declared in the Stackfile. | Flag | Meaning | |---|---| | `--org <slug>` | Evaluate workloads registered in `<slug>` | | `--stages <spec>` | Comma-separated `<slug>[:<invocationsPerRun>][:bg]` list | | `--stackfile <file>` | Run offline against a `vitna.stackfile.v1` file with no database | | `--pipeline <slug>` | Select a named entry from `stackfile.pipelines` | | `--runs <n>` | Monthly pipeline execution volume (`monthlyRuns`) | | `--run-label <label>` | Plural label for one pipeline run (default `runs`) | | `--ledger <file.jsonl>` | Group `vitna-runtime` JSONL rows by `runId` (`x-vitna-run`) for empirical stage fan-out | | `--agent <agentId>` | Filter `--ledger` rows to one `x-vitna-agent` identifier | | `--window YYYY-MM-DD..YYYY-MM-DD` | Restrict `--ledger` rows to one closed UTC date range | | `--evidence <evals.json>` | Supply offline evaluation measurements in `--stackfile` mode | | `--terms <terms.json>` | Supply offline provider compliance terms in `--stackfile` mode | | `--discover-peers` | Scan active Witness SKUs in each stage's class for eligible lower-cost exploratory levers | | `--max-run-cost-usd <usd>` | CI gate: exit `4` if `totals.costPerRunUsd` exceeds `<usd>` or any stage is unpriceable | | `--max-monthly-cost-usd <usd>` | CI gate: exit `4` if `totals.monthlyCostUsd` exceeds `<usd>` or `monthlyRuns` is unset | | `--json` | Emit structured JSON report | ## Exit codes | Code | Meaning | |---|---| | `0` | Evaluation succeeded and cleared any configured spend budget gates | | `1` | Register mode requested with neither `DATABASE_MIGRATE_URL` nor `DATABASE_URL` set | | `2` | Invalid flags, malformed stage spec, unreadable input file, or unknown organisation/workload/pipeline | | `4` | `--max-run-cost-usd` or `--max-monthly-cost-usd` gate was exceeded (or a stage was unpriceable under a gate) | --- <!-- https://vitna.ai/docs/cli/aperture/private.md --> # Your own infrastructure Registering something this organisation runs that no provider publishes a price for, and entering what it costs. A company-hosted deployment, an internal fine tune, a regional vendor under contract, an owned cluster. Until these commands existed, a workload could only name infrastructure Vitna could observe, which is a narrower thing than the infrastructure an organisation can actually use. ## Register one ```bash npm run aperture:private:add -- --id acme/vllm-prod-a --name "vLLM, Llama 3.3 70B, 4xH100" --class model ``` | Flag | Meaning | |---|---| | `--id` | `org/slug`, the same shape a workload id has. The org half is what scopes it | | `--name` | What a person reading a brief will see | | `--class` | The market it competes in, from the ones the record already names | | `--note` | What it actually is, in your own words. Nothing derives anything from it | ## Enter what it costs ```bash npm run aperture:private:price -- --sku acme/vllm-prod-a --dimension gpu_hour_on_demand --amount 2.49 --from 2026-08-01 ``` | Flag | Meaning | |---|---| | `--sku` | Which of yours | | `--dimension` | What the rate is per. It has to be one the market quotes, or no unit profile could spend it | | `--amount` | USD per one of that dimension's units | | `--from` | The date the rate became true. A date in the future is refused | | `--note` | Where the figure came from in your own world: an invoice, a contract, a capacity model | ## It is not a sku on the tape, and it never becomes one This is the part worth being exact about, because the word "sku" invites the opposite assumption. A private sku lives in `aperture`, scoped to one org. The boards, the spreads, the [reference rates](../../concepts/reference-rate.md) and the JSON Feed all query `witness.sku`, and it is not in that table, so it cannot reach any of them by construction rather than by a filter somebody has to remember. Nothing about it is published. Vitna makes no claim about it to anybody. ## The price is entered, and every surface that reads one says so There is no fetch behind an entered rate, no archived document, no sha256 and no [attestation](../../concepts/attestation.md). What the record can say is who entered it and when. So a decision priced partly from one: - names the `aperture.private_price` rows in `inputs.enteredPriceIds`, its own list, separate from the `witness.price_period` ids - prints them under their own heading in the brief - carries a sentence saying a reader cannot recompute them from anything they can fetch, which a witnessed price they can A figure a reader can recompute from a fetchable document and a figure somebody typed are not the same claim, and a brief that put them in one list would be saying the second is the first. ## A rate is not revisable `aperture.private_price` is append-only like every other record table. A rate that turns out to be wrong is corrected by entering the better one from the date it became true, and the old row keeps saying what a decision taken last quarter was taken over. The engine reads the newest rate effective on or before the day it runs. A rate dated ahead of today is refused rather than entered and silently ignored, because a person who entered one would otherwise believe the next run reads it. ## What a private sku cannot have **A watch on the market.** Four of the six [trigger](triggers.md) kinds fire on a `witness.market_event`: a price move, a delisting, a deprecation, a launch. Nothing observes a private deployment, so no event will ever name one, and a watch armed over one would sit in the register looking like a watch and be skipped silently on every collector run. Arming one is refused. The two that stay are the two that watch the organisation's own record rather than the market: `evidence_added` and `date`. Those are exactly the watches a private deployment can have. --- <!-- https://vitna.ai/docs/cli/aperture/seed-notify.md --> # Seeding and notification Loading the engine's reference data, and draining the outbox into the feed that stands in for email. ```bash npm run aperture:seed npm run aperture:notify ``` ## `aperture:seed` The engine's reference data, idempotent: the org, the first workload, its candidates, its evals, and the six decisions of its history. Those six are human decisions taken before the engine existed. They are seeded as decisions rather than reconstructed as proposals, because they are what the workload actually did, and a register showing only the machine half would misreport its own history. ## `aperture:notify` Drains the outbox. Notification is a feed rather than email, and the reasoning is recorded in the source: email needs SMTP details nobody has, an inbox nobody has agreed to monitor, and a deliverability problem this project does not need. A feed needs no credential, and every consumer already knows how to read one. So the published channel is [`/api/v1/feed.json`](../../api/v1/log/feed.md), and email stays documented as an add-on for whenever there is an inbox somebody reads. ## What a notification says it did not do A fired trigger's feed item states that nothing was routed and nothing was accepted. That sentence is in the payload rather than only in documentation, because a notification implying the system took an action it did not take is the worst thing this channel could carry. --- <!-- https://vitna.ai/docs/cli/aperture/triggers.md --> # Trigger commands Arming a watch on the record, and seeing what has fired. ```bash npm run aperture:trigger:arm -- --workload <id> --kind price_move --threshold-pct 10 npm run aperture:trigger:disarm -- <id> --reason "..." npm run aperture:triggers npm run aperture:watch ``` ## Arming Refused where the kind names one sku and the workload names no incumbent. A `price_move` watch with nothing to watch the price of is not a watch. ## Firing spends a trigger A trigger fires once. It will not fire again until a decision arms a new one. That is what stops one volatile cell producing the same notification every hour until somebody mutes it, and a muted watch is a watch nobody reads. The cost is that a second move during an unresolved review does not notify, which is the right trade: the first one is already waiting for a person. ## Disarming takes a reason Kept on the row. Calling a watch off is a decision about what to stop paying attention to, and the register keeps it rather than the trigger quietly disappearing. ## Listing `aperture:triggers` prints what each workload watches and what has already fired, with the event that fired it and the proposal it produced. The same view is public at [`/api/v1/watches`](../../api/v1/engine/watches.md), and the sentence describing a watch comes from the same function, so the CLI and the API cannot describe one watch two ways. --- <!-- https://vitna.ai/docs/cli/aperture/watchdog.md --> # Rollout and decision watchdog CLI Evaluate standing-decision drift, shadow rule promotion readiness, and routed canary health across an organisation's active workloads or offline against a checked-in Stackfile and runtime ledger. ```bash DATABASE_MIGRATE_URL=... npm run aperture:watchdog -- --org convexity ``` ```bash npm run aperture:watchdog -- --stackfile vitna.stack.json --policy policy.json --ledger ./ledger.jsonl --fail-on-rollback ``` Nothing is written to the database or disk. Every finding prints the exact operator command (`aperture:decide`, `aperture:rule:promote`, `aperture:rule:add`, or `aperture:rule:remove`) required to advance or roll back a rollout. See [Rollout and decision watchdog](../../routing/watchdog.md) for how each verdict is derived. ## Modes and flags - **Register mode** (`--org <slug>` or `--workload <org/slug>`): reads active workloads, standing decisions (recovering decision-time unit costs from pinned `pricePeriodIds` and `enteredPriceIds`), lane rules, fallback chains, pushed `deployment_observation` windows, `evidence` rows, live Witness and entered rates, provider compliance terms, and three-provenance SKU capabilities. May optionally pair a local runtime `.jsonl` file via `--ledger <path>`. - **Offline Stackfile mode** (`--stackfile <vitna.stack.json>`): runs with zero database connection for CI pipelines and GitOps checks, pairing the Stackfile with an optional signed `--policy <policy.json>`, `--ledger <file.jsonl>`, `--evidence <evals.json>`, and `--terms <terms.json>`. | Flag | Meaning | |---|---| | `--org <slug>` | Evaluate all active workloads in `<slug>` | | `--workload <org/slug>` | Narrow evaluation to one active workload | | `--window YYYY-MM-DD..YYYY-MM-DD` | Restrict observation and ledger rows to one closed UTC date range | | `--ledger <file.jsonl>` | Summarize per-request `vitna-runtime` ledger rows (`shadowRule`, `serve`, `cohort`, `latencyMs`) | | `--stackfile <file>` | Run offline against a `vitna.stackfile.v1` file with no database | | `--policy <policy.json>` | Pair offline Stackfile evaluation with a signed `policy.json` | | `--evidence <evals.json>` | Supply offline evaluation rows in `--stackfile` mode | | `--min-shadow-matches <n>` | Minimum shadow matches before `promote_ready` (default `50`) | | `--min-canary-units <n>` | Minimum served canary units before `expand_or_adopt_ready` (default `100`) | | `--max-unserved-pct <n>` | Maximum unserved attempt rate percentage (default `5`) | | `--max-fallback-pct <n>` | Maximum fallback share of served traffic percentage (default `10`) | | `--max-price-drift-pct <n>` | Maximum unit-cost increase vs standing decision percentage (default `10`) | | `--json` | Emit structured JSON report | | `--fail-on-rollback` | Exit `4` if any workload verdict is `rollback_recommended` or `decision_drifted` | ## Exit codes | Code | Meaning | |---|---| | `0` | Evaluation succeeded (and no rollback or decision drift occurred when `--fail-on-rollback` is set) | | `1` | Register mode requested with neither `DATABASE_MIGRATE_URL` nor `DATABASE_URL` set | | `2` | Invalid flags, malformed date window, unreadable input file, or unknown organisation/workload | | `4` | `--fail-on-rollback` was passed and at least one workload returned `rollback_recommended` or `decision_drifted` | --- <!-- https://vitna.ai/docs/cli/aperture/workloads.md --> # Workload commands Entering and correcting the things you run. ```bash npm run aperture:workloads npm run aperture:workload:add -- --id <org/slug> --name "..." --class model --incumbent <sku> --unit-label "..." --priority economics npm run aperture:workload:edit -- <id> [flags] npm run aperture:workload:profile -- --workload <id> [flags] ``` `workload:add` also takes `--volume N`, and repeatable `--profile k=v` and `--constraint k=v`. ## Listing `aperture:workloads` prints every workload, what it is running now, and where one cannot be decided on, the reason. Printing the blocking reason beside the workload is the point. A register that listed workloads and left you to discover which were undecidable would send you to the failure one command at a time. ## How far round the loop each one has come Every workload prints a rung count and one next step: ``` progress 4 of 10 steps of the loop next npm run aperture:candidate:add -- --workload acme/lane --sku <sku> Something to move to. A firing with no candidate has nothing to propose. ``` The ten rungs are the loop in the order a workload climbs it: registered, an incumbent named, a unit profile, a candidate in play, a measurement, a decision taken, a ruling, a watch armed, production coming back, and a runtime rather than a person being what sent it. **It is a meter and not a score.** A workload on four rungs is one somebody has got four steps into, and the line under the count says what to type rather than what is missing. There is no percentage, deliberately. The next step is always the first unmet rung you can act on now, which is why a waiting proposal is what you are told to rule on rather than being told to arm another watch, and why a workload with nothing proposed is sent back to the engine rather than to an acceptance with nothing to accept. Two rungs are worth reading carefully. **Production coming back** is any window of usage the register holds for the workload, however it arrived. **A runtime pushed it** is narrower: a reading carrying latency and tokens, which only a runtime produces, since a reading typed from an export carries none of them. Nothing here can see a policy file or a running process, because neither leaves a row, and a rung the record cannot answer would be a rung that lies. ## The four refusals on entry `aperture:workload:add` refuses: | Refusal | |---| | An incumbent the registry does not hold | | An incumbent from another market | | A profile key the class does not spend | | An incumbent with no profile to spend it against | All four are the same principle: a workload the engine cannot reason about should fail at entry, loudly, rather than produce a confident brief later from inputs that never made sense. ## A workload is a registry row, not a record `aperture:workload:edit` corrects one, because a workload is a description of something you run and descriptions are wrong sometimes. Evidence and decisions are not editable. Those are the record, and they stay append-only. `--env-key` and `--policy-alias` set the [delivery target](../../routing/delivery.md): the first is a variable a consumer loads, the second a lane a runtime resolves, and a workload may carry both. They merge, and an empty value removes that half. --- <!-- https://vitna.ai/docs/cli/database.md --> # npm run db:generate, npm run db:migrate Generating a migration from the schema and applying it, forward only, with the grants that carry the record's rules kept narrow. ```bash npm run db:generate npm run db:migrate ``` | Command | What it does | |---|---| | `db:generate` | Generate a migration from `platform/db/schema.ts` | | `db:migrate` | Apply `drizzle/` to `DATABASE_MIGRATE_URL`, forward only | ## Forward only There are no down migrations. Rolling back a schema change means writing the next migration, and recovery from a bad one is a restore. That is a deliberate trade: a down migration is code that runs once, under pressure, having never been rehearsed against production data. ## Hand-written migrations are normal here Several migrations are written by hand rather than generated, because the constraints and grants that carry the record's rules do not come out of a schema diff. The check constraint on a workload's [delivery target](../routing/delivery.md) is one example, and the comments explaining why are part of the migration. ## Roles Migrations run as the owning role. Application roles are granted narrowly, and the shape of the narrowness is that **every `UPDATE` in this database is column-level**. There is not one grant on a whole table anywhere. Two of them are in `aperture` (the collector marking a trigger fired, the console recording a ruling), five are the collector's in `witness`, and two are the auth role's in `site`. There is no `DELETE` in `witness` or `aperture` at all. A migration that widens any of that is a decision, and the tests enumerate the grant lists so widening one is visible in a diff. --- <!-- https://vitna.ai/docs/cli/event-add.md --> # npm run event:add Add one curated [market event](../concepts/market-event.md). ```bash npm run event:add ``` ## The inclusion rule is enforced, not observed Refused without a primary URL on the provider's own domain that answers today. The domain list is a column on `witness.provider`, so "is this the provider's own announcement?" is settled by data rather than per entry by whoever is typing. The consequence, stated plainly: an accurate news report of a price change cannot become a curated event. Only the provider's own page can. ## Answers today The URL is fetched at entry and the day recorded as `source_verified_on`. A citation that already 404s does not enter the log. ## Detected events are not entered here Change detection writes those, and a detected price move must name the periods on both sides of the comparison. This command is only for the curated half. --- <!-- https://vitna.ai/docs/cli/keys.md --> # Push keys Who may land a reading over the network, per org, minted and revoked at the prompt. The web process authenticates against these and cannot mint one. With console keys on, an owner can revoke one from the console, and can mint a platform key there through a separate minter (below). ## Minting ```bash npm run org:key:create -- --org convexity --label "runtime, box 1" ``` The token is printed once and stored nowhere: the record holds its sha256, by the same constraint that keeps a session token out of the nightly dump. It goes in `VITNA_PUSH_KEY` on the box whose [`runtime:push`](../runtime/pushing.md) will use it, and authenticates `POST /api/v1/usage` for that org's workloads only. `--label` is required and is for the operator: with three boxes pushing, "which key is this" has to be answerable at the prompt. A reading's own provenance carries the key's id rather than its label, so a relabel cannot rewrite history. The command ends by listing the org's keys, so what exists is visible at the moment one is added. ## Revoking ```bash npm run org:key:revoke -- <id> --reason "box decommissioned" ``` Revocation is a date and a reason on the row, never a delete: which keys existed and when they stopped is part of how a pushed reading's provenance is audited, and the row holds a hash, so keeping it keeps no secret. It applies on the key's next push, which is refused with the date and the words "mint another", so a forgotten pusher learns to stop rather than retrying forever. Losing a token means revoking and minting again. Nothing can recover one, because nothing holds one. ## Seeing what exists The console lists them, per org, under **Push keys** on [the settings page](https://vitna.ai/console/settings): the label, when the key was minted, whether it is live or revoked and why, and whether any reading the register currently holds arrived on it. Push and router keys are minted only at the prompt, because both need the migrate credential, which is strictly more privilege than a push key grants. The role the page reads with is granted every column of the table except the hash, so the page cannot show a token and no later query written against it could either. ## Platform keys from the console On an instance with `VITNA_CONSOLE_KEYS=on`, an org owner can mint a platform key on the settings page and revoke any of the org's keys there. The web process still cannot mint one. The page files a request, the owner confirms it with a passkey, and `vitna-mint` (`npm run org:key:minter`, as `vitna_minter`) mints the key and seals it to a key the owner's browser made for that request. The page opens it, shows it once, and forgets it. On the box the minter is `vitna-mint.service`, reading `/etc/vitna/env.minter`. By hand it needs `DATABASE_MINT_URL`, `VITNA_SITE_ORIGIN` and `WEBAUTHN_RP_ID`, and refuses to start without all three: ```bash npm run org:key:minter ``` The minter mints only when every one of these holds: - the request is less than ten minutes old and names an org the requester owns; - its passkey assertion answers a challenge that is a digest of the request itself, so it cannot be spent on another request or another sealing key; - the passkey belongs to the requester, verified the person (a PIN or a biometric), and is approved for minting with the same key it holds now; - the org has minted fewer than 10 console keys in a day, and the owner fewer than 5 in an hour. An operator approves a passkey once, after its owner reads back the fingerprint the settings page shows beside it: ```bash npm run org:passkey:approve -- --github <login> npm run org:passkey:approve -- --github <login> --passkey <id> --by "<who checked, and how>" npm run org:passkey:approve -- --github <login> --passkey <id> --withdraw ``` This is what keeps a compromised web process from minting on its own: it can add a passkey to an account, but not approve one. What it can still do is serve a page that asks the owner for a passkey touch and binds that touch to a request of its choosing, because an authenticator shows the site and not the request. So the guarantee is no key without a live touch of an approved passkey, not no key the owner did not mean. Revoking from the page writes the revocation's date, reason and who revoked; a trigger keeps a revoked key revoked for every role. Every request and its answer stay in `site.key_request`, which nothing rewrites. One column is easy to over-read. **Named by a reading** is not "last used": the register keeps the newest reading per workload, so a hand-run export landing on top of a runtime's window makes that column go quiet while the key is still pushing, and a key that pushed once in June still shows if nothing has landed since. It is evidence a key reached the record, never evidence of when it last did. --- <!-- https://vitna.ai/docs/cli/scripts.md --> # The standalone scripts The mirrors that write `app/data/`, the reconcilers that run behind them, and the rest of what is run with `node` rather than through npm. | Script | What it does | |---|---| | `export-snapshot.mjs` | Mirror the live tape into `app/data/*.json` | | `export-catalog.mjs` | Mirror the public model list into `app/data/catalog.json` | | `export-speed.mjs` | Mirror OpenRouter's measured throughput and latency into `app/data/speed.json` | | `export-endpoints.mjs` | Mirror what every seller charges, and what it declares about its route, into `app/data/endpoints.json`, and one uptime reading a day per seller into `app/data/uptime.json` | | `export-images.mjs` | Mirror what each image model costs, per endpoint and in the unit each price is published in, into `app/data/image-models.json` | | `export-weights.mjs` | Mirror what each open-weight listing's Hugging Face repository says about itself into `app/data/weights.json` | | `export-eci.mjs` | Mirror Epoch AI's Capabilities Index into `app/data/eci.json` | | `export-usage.mjs` | Mirror OpenRouter's usage datasets into `app/data/usage.json`, with a key | | `reconcile-catalog.mjs` | Make the speed, seller, uptime, weights and sku-map documents agree with a fresh catalog | | `reconcile-registry.mjs` | Record which registered skus the boards have stopped pricing | | `triage-candidates.mjs` | Sort the discovery queue into promote, reseller, tiered and unlocatable | | `shoot.mjs` | Capture every route as full-page and viewport images | | `edit-fixture-price.mjs` | An edited copy of an archived document, for scenarios only | ## The mirrors ```bash node scripts/export-snapshot.mjs https://vitna.ai node scripts/export-catalog.mjs --dry-run node scripts/export-speed.mjs --limit 20 ``` All three write into `app/data/`, which is what the site renders when it has no database. That keeps two logs at two resolutions: git is the daily one, with a commit per day and a readable diff, and the database is the hourly one. `export-catalog.mjs` is key-free and writes only when a row moved, so a run that changes nothing produces no commit. ## The two reconcilers ```bash node scripts/reconcile-catalog.mjs --check node scripts/reconcile-registry.mjs https://vitna.ai --check ``` Each runs immediately after the mirror it belongs to, in the same job, so that one commit carries a document and everything that document implies. Neither decides anything; both carry a decision already made upstream into the files that restate it. `--check` reports and writes nothing. They exist because a withdrawal used to stop a mirror dead. A product leaves a market, the fresh document correctly stops carrying it, and a hand-written file somewhere still names it, so a test refuses the pair and the only file that could resolve it is the one being refused. Between 2026-08-25 and 2026-09-08 both the catalog and the snapshot mirrors failed on that shape every single day, and the site could not accept a new model for a fortnight. `reconcile-catalog.mjs` reads no network. It deletes the speed and seller rows for listings the fresh catalog no longer carries, recounting each document so it still clears the floors a freshly read one has to, and records which of the sku map's ids have been withdrawn in `platform/catalog/delisted.mjs`, which the map is read through. `reconcile-registry.mjs` reads the boards and `/api/v1/sources/health`, and writes `platform/witness/withdrawn-skus.mjs`. Nothing is deleted from the register: an adapter replaying an archived document, a deprecation event naming its sku, and a sku's price history all need the product to stay registered after it stops being sold, so the withdrawal is a list beside the register rather than a cut into it. The health read is the part worth understanding. A board absence means "no current price", and a source whose parser is failing produces no current price for anything it quotes, so under a broken adapter an absence says nothing about the market at all. On the day this was written, eleven of seventeen absences were real withdrawals attested by sources that had read their pages that morning; the other six were Novita products still on sale behind a parser that had failed 208 times running. Only a healthy source can put a row in `withdrawnSkus`. The rest go to `unattestedSkus`, which the boards are compared against too, so a broken source no longer stalls the mirror, and the file says which of the two happened to every row. `export-speed.mjs` is key-free too, and reads differently: OpenRouter's API defines `latency_last_30m` and `throughput_last_30m` and returns null for both on every endpoint, so the figures are read from each model's own page instead, one page per model, three at a time. It asks robots.txt once before the run and sends the same honest agent as the collector. The list of pages is the committed catalog, so it can never read a page the catalog does not already carry. Unlike the catalog, this one changes almost every run, because the market's speed does. A page that will not answer is a recorded gap in `meta.failures`, never a zero, and a run that reached fewer pages and measured fewer models than the committed document refuses to write. `--limit` and `--only` make a smoke test cheap. Neither writes the document: a partial read is not a picture of the market. `export-images.mjs` is the price list for images. It reads OpenRouter's image models list, then each listed model's endpoints document, key-free, three at a time after asking robots.txt once. It does not start from the catalog, because most image models are not in it. Every price is kept in the unit it is published in, per image, per megapixel, per token or per request, beside what it bills and the condition it holds under, and nothing converts one unit into another: a price per token is a price per image only after a run says how many tokens the image took. A listed model nobody serves is named in `meta.unserved`, a served model that publishes no price in `meta.unpriced`, and a document that did not answer in `meta.failures`, because those are three different answers to whether a picker can show a price. It claims no licence: OpenRouter's CC BY 4.0 grant covers its keyed Data API and, by its own terms, nothing published outside it. `export-usage.mjs` is the one mirror that needs a key. OpenRouter publishes the figures behind its rankings page, tokens per model per UTC day and tokens per app over the window, at two dataset endpoints that answer 401 without a valid OpenRouter key. The data is CC BY 4.0 and the citation names OpenRouter and the `as_of` instant each response carries. The key is read from `OPENROUTER_API_KEY`, put in one request header and nowhere else, and the workflow passes the secret to that one step. Two requests a run, against a limit of 500 a day. ```bash OPENROUTER_API_KEY=... node scripts/export-usage.mjs --dry-run node scripts/export-usage.mjs --from <dir> ``` `--from` parses `rankings-daily.json` and `app-rankings.json` saved in a directory, dated by the `date` response header saved beside each as `<name>.headers`, so the document can be rebuilt from a saved read without a key. A read with a new dataset version, a row after the day's `other` row, a count past the exact range of a number, or fewer than 20 days, 30 models or 20 apps is refused and the committed document stands. `export-weights.mjs` is the one mirror that reads a host other than OpenRouter. The catalog names a Hugging Face repository for every listing whose weights are published, and the repository's own metadata carries four facts the list does not: the licence the uploader tagged it with, the parameter count its safetensors declare, the model class its config names, and whether the download is gated behind accepting the maker's terms. One request per repository, key-free, three at a time, after asking robots.txt once. The document is keyed on the repository rather than the listing, because a base listing and its variants point at one set of weights. What it deliberately does not read is downloads and likes. Those are adoption, which the catalog pages say they are not, and a figure that moves every hour would make every run write. A licence is printed as the uploader tagged it, never as verified, and a repository that does not answer is a recorded gap in `meta.failures`, never a model with no licence. `export-eci.mjs` carries one published index rather than a table of scores: Epoch AI's Capabilities Index, every model it places with the interval around its figure. It runs in the board mirror's job, behind the licence watch, because it is covered by the same licence row as Epoch's board, and it refuses to fetch anything if that row stops permitting republication. Like the boards, a publisher that does not answer leaves the committed document standing and the run green; a document that answers and no longer parses stops the run. It reads two files and republishes one. The second, the results behind each figure, mixes Epoch's own runs with results collected from other publishers, whose licences still apply, so only the count per model is kept. `--pairs` prints which catalog listing each model is. The pairing takes a reviewed decision, an equal display name, or an exact slug through the same collision-refusing index the board names use, and it refuses a listing that was on sale more than a fortnight before the model was released, which is what pairing a later snapshot onto an earlier listing looks like. ## `edit-fixture-price.mjs` is for scenarios only It produces a deliberately edited copy of an archived document, so a test can exercise the change detection path against a known before and after. It exists so that no test ever needs to invent a document. The edited copy is labelled as one and never enters the evidence archive. --- <!-- https://vitna.ai/docs/cli/skus/candidates.md --> # npm run sku:candidates The priced rows the collector saw that the registry does not track. ```bash npm run sku:candidates npm run sku:candidates -- --all ``` By default it shows rows seen at least twice. `--all` includes rows seen once. ## Why a second sighting A row seen once may be a parse artefact of one malformed page. A row seen twice is something the provider is actually publishing. Same discipline as the confirmation rule on [price periods](../../concepts/price-period.md), applied to existence rather than to value. ## The collector counts, a person decides `witness.sku_candidate` carries a status, and the collector holds `UPDATE` on the counting columns only, not on the status. So discovery is automatic and admission is not. A row becomes a [sku](../../concepts/sku.md) by [promotion](promote.md), is [linked](link.md) to one registered by hand, or is [ruled out](reject.md). --- <!-- https://vitna.ai/docs/cli/skus/link.md --> # npm run sku:link Close a candidate against a [sku](../../concepts/sku.md) the registry already holds. ```bash npm run sku:link -- <id> --sku <sku id> [--note "..."] ``` For a product registered by hand in `platform/witness/seed-data.mjs`, which is how a batch of joins lands with its adapter rows and fixtures. The registry already holds the slug, so [promotion](promote.md) refuses the candidate, and [rejection](reject.md) would record a product Vitna tracks as one it decided not to. Without a link the row stays pending and counts on /witness as work nobody has done. ## What it refuses - A candidate that is not pending. A ruling is not made twice. - A sku sold by a different provider from the page the row was read on, however alike the names are. - A candidate whose page has not been read successfully since the row was last counted. Every good read counts every row the adapter does not claim, so a later read that did not count it is the evidence that the registered sku now covers the row. Before that read, the next one could count it again. No event is written. The sku was registered by the change that added it, and a link records only that a queue row and a registry row are the same product. The note defaults to naming the sku, and `--note` replaces it. --- <!-- https://vitna.ai/docs/cli/skus/promote.md --> # npm run sku:promote Register a candidate as a [sku](../../concepts/sku.md). ```bash npm run sku:promote -- <id> --slug <slug> --name <name> npm run sku:promote -- <id> --slug <slug> --name <name> --backfill ``` ## What promotion writes The sku, and a `launch` [market event](../../concepts/market-event.md) recording when the record first read the row. That event is the honest claim available: Vitna knows when it first saw something, not when a provider shipped it. The event says the former. ## `--backfill` Registers the sku and writes **no** event. For a back catalogue Vitna is reading for the first time rather than something a provider has just done. Writing a launch event for a two-year-old model would put a false statement in the log, and the log is the part of the record people quote. The distinction is the operator's to make, and it is the one judgement this command asks for. --- <!-- https://vitna.ai/docs/cli/skus/reject.md --> # npm run sku:reject Rule a candidate out. ```bash npm run sku:reject -- <id> --reason "..." ``` The reason is required and is kept on the row. No event is written: declining to track something is not a market event, and the log would be worse for carrying Vitna's filing decisions. ## Rejection is not deletion The row stays, with its status and its reason. The collector keeps counting sightings. So a rejection is a recorded judgement that can be revisited, rather than a row that vanishes and gets rediscovered as new on the next pass. --- <!-- https://vitna.ai/docs/cli/witness/attest.md --> # npm run witness:attest Hash a day's observations into a Merkle root and sign it. ```bash npm run witness:attest npm run witness:attest -- --verify npm run witness:attest -- --new-key ``` ## Flags | Flag | Meaning | |---|---| | (none) | Sign yesterday | | `--verify` | Recheck every root ever signed | | `--new-key` | Print a keypair. Touches no database. | ## Yesterday, not today A day is signed the morning after it closes, so a figure read today becomes provable tomorrow. Until then [`/api/v1/observations/{id}/proof`](../../api/v1/verification/proof.md) answers with a null attestation and says why. ## What signing proves That the record for that day has not changed since it was signed. It does not prove a provider published a price. See [attestation](../../concepts/attestation.md). ## The key The same key [`aperture:config`](../../routing/config.md) signs a routing config with, under a different statement prefix so the two cannot be confused. `--new-key` prints a pair and writes nothing, so generating one does not require access to the record. --- <!-- https://vitna.ai/docs/cli/witness/collect.md --> # npm run witness:collect One collector pass: fetch every enabled source, parse it, archive it, and write what changed. ```bash npm run witness:collect npm run witness:collect -- --dry-run npm run witness:collect -- --source anthropic-pricing ``` ## Flags | Flag | Meaning | |---|---| | `--dry-run` | Needs no database. Fetches and parses, writes nothing. | | `--source <id>` | Read one source, enabled or not | `--source` reading a disabled source is what pairs with [`witness:seed --hold-new`](seed.md): a new adapter gets watched against the live page before it is turned on for everyone. ## What one pass writes A `fetch_run`, a `fetch` row per source with a final status, an [observation](../../concepts/observation.md) per parsed figure, and whatever [price periods](../../concepts/price-period.md) and [market events](../../concepts/market-event.md) the comparison implies. A priced row the registry does not hold becomes a [candidate](../skus/candidates.md) rather than an observation. ## In-flight runs and freshness A run's row is written when it finishes. While it is working, the source's last completed run is still the most recent one, so a source can read as stale while it is actively being read. Check whether a run is live before acting on a stale reading. Re-enqueueing a job that is eighty percent through a backfill makes it worse. ## The engine runs at the end Trigger evaluation happens after the collector's writes, in the same pass, so a price move and the watch it fires are one unit of work rather than two that can disagree. --- <!-- https://vitna.ai/docs/cli/witness/rates.md --> # npm run witness:rates Publish yesterday's [reference rates](../../concepts/reference-rate.md). ```bash npm run witness:rates npm run witness:rates -- --verify npm run witness:rates -- --dry-run ``` | Flag | Meaning | |---|---| | `--verify` | Recompute every day ever published and report disagreement | | `--dry-run` | Write nothing | ## The rule has no free parameters The median list price among at least three distinct providers publishing a confirmed price that day, in exact whole millionths. Because the inputs are append-only and the rule has nothing to tune, an independent implementation reading [`/api/v1/skus/{slug}/history`](../../api/v1/prices/sku-history.md) should arrive at the same series. `--verify` is Vitna running that check against itself. ## A day is published once A series is never restated. A correction is a new row on a later day, which is what makes the series usable as a benchmark rather than as an opinion with a timestamp. --- <!-- https://vitna.ai/docs/cli/witness/rebuild-periods.md --> # npm run witness:rebuild-periods Replay every [observation](../../concepts/observation.md) into [price periods](../../concepts/price-period.md). ```bash npm run witness:rebuild-periods npm run witness:rebuild-periods -- --yes ``` Reports only, unless given `--yes`. ## Why replaying is possible at all Observations are append-only, so periods are derived rather than primary. A parser fix or a change to the confirmation rule can be applied to history rather than only to what arrives next. That is the practical payoff of the append-only rule: the readings are the record, and the periods are a view of them that can be rebuilt. ## The reporting default The default prints what would change and writes nothing. A rebuild rewrites the history the boards draw and the rates are computed from, so the diff is read before it is applied rather than after. --- <!-- https://vitna.ai/docs/cli/witness/seed.md --> # npm run witness:seed The registry: classes, providers, sources, dimensions, skus, and the curated market log. ```bash npm run witness:seed npm run witness:seed -- --hold-new ``` ## Idempotent by construction Safe to run after every deploy. Curated log rows carry a partial unique index on the day and the headline, so a second run inserts nothing. ## `--hold-new` Inserts a newly added source **disabled**. A source arrives with an adapter nobody has watched run against the live page. Landing it enabled means the first collector run after a deploy is also its first real test, and a bad parse enters the record before anyone looks. Held, it is read only by an explicit single-source run. ## What it does not do It does not remove anything. A source that leaves the seed data stays in the database, because rows already point at it and the record does not lose history to a config edit. --- <!-- https://vitna.ai/docs/cli/witness/terms.md --> # Recording a provider term Entering, by hand, what a provider published about how it handles the work: retention, training use, processing regions, whether a key you hold is accepted. Each reading is taken from a page on the provider's own domain. ```bash npm run term:add -- --provider openai --kind zero_retention --value yes --on 2026-08-27 --source-url https://openai.com/enterprise-privacy ``` | Flag | Meaning | |---|---| | `--provider` | The provider it is about | | `--kind` | One of `zero_retention`, `no_training`, `regions`, `byok` | | `--value` | `yes` or `no`, or comma-separated region codes for `regions` | | `--on` | The date somebody read the page. Never a date in the future | | `--source-url` | The page it was read from, on that provider's own domain | | `--note` | What the page scoped it to, if it scoped it | ## The four refusals it inherits They are the [curated log](../../concepts/market-event.md)'s, applied through the same function: 1. No source URL, no row. 2. `http` is not a source. 3. A host that is not the provider's is not the provider. 4. A URL that does not answer 200 today is not verified today. Plus two of its own. A value that is not one the kind accepts is refused rather than stored, because "the term answers whether the provider trains on customer data, and a hedge is not an answer this record can hold". And a second reading of the same kind on the same day is refused, because a term that changed is a new row on the day it was read, never an edit of the one a decision was taken under. ## Why a person and not an adapter A policy page is prose and the answer is a yes or a no. A parser deciding what "we do not retain your data" means would be interpolating exactly where the honesty contract forbids it. ## What it is for A workload requires one, and a candidate whose provider does not meet it is excluded before anything is weighed: ```bash npm run aperture:workload:edit -- acme/support --constraint requiresZeroRetention=true ``` [Provider term](../../concepts/provider-term.md) is exact about what a requirement means when the record holds no term to check it against, which is the case worth reading before you rely on one. --- <!-- https://vitna.ai/docs/operations/overview.md --> # Operations How the site is served, deployed, collected and backed up. One box, one Node process, no orchestration. `HOSTING.md` at the repository root is the operator's runbook and carries the exact commands, addresses and recovery notes. These pages are the shape of it: what the pieces are and why they are arranged this way. ## Topology | Layer | What | |---|---| | Edge | Cloudflare, proxied, SSL Full (strict), HTTPS forced | | Origin | One VPS. nginx terminates TLS and proxies to the production server on loopback | | Process | A systemd unit running the built server | | Database | Postgres 16 on the same box | | Collector | A oneshot unit fired hourly by a timer | | Backups | A nightly unit, dumping and mirroring off-box | `www` redirects to the apex, and HTTP redirects to HTTPS. ## Response headers come from two places The worker sets the content policy, the permissions policy and the cross-origin opener policy. The nginx site file sets the frame options, content type options and referrer policy, and nginx **appends** rather than replaces. Two of them are therefore sent twice with identical values, which is inert. The worker deliberately does not send frame options at all: it did once, and the response carried two headers that disagreed. The content policy says the same thing more strictly. A duplicate header is cheap. A contradictory one is not, so check the response headers before and after touching that nginx block. ## HSTS belongs to the edge A max-age served by mistake is cached by every browser that saw it and cannot be withdrawn. The application never sends one. ## The pages | Page | What it covers | |---|---| | [Deploys](deploys.md) | How code reaches the box | | [Verifying a deploy](verifying-a-deploy.md) | Which commit is actually live | | [Rolling back](rolling-back.md) | The two kinds of failure, and what push cannot undo | | [The collector](collector.md) | The hourly run and its guards | | [Backups](backups.md) | What is dumped, what is encrypted, and what fails loudly | | [Self-service checkout](checkout.md) | How somebody becomes a customer, and what the box needs for it | | [Rotating a signing key](key-rotation.md) | Replacing the policy key without refusing a runtime, and what a compromise needs | --- <!-- https://vitna.ai/docs/operations/backups.md --> # Backups A nightly unit dumps the database, keeps a fortnight locally, and mirrors both the dumps and the evidence tree off-box. ## An unset destination is a failure Once any document is archived, an unset backup destination makes the unit **fail** rather than log and exit zero. The archive is what the attestation attests to, and an attestation that exists on one disk is one failure away from not existing. ## What is encrypted and what is not | Contents | Encrypted | Why | |---|---|---| | Database dumps | Yes, symmetric AES-256, before leaving the box | They carry other people's email addresses, and object storage is a third party holding them | | The evidence tree | No | Other companies' public pricing pages, no personal data, and content-addressed, so encrypting it would break the deduplication that keeps it small | Only the encrypted dump is uploaded. The plaintext dump stays on the box beside the database it came from, where encryption would add nothing. If no passphrase exists and the destination is object storage, the run **fails** rather than sending the dump in clear. The waiver is an environment variable, and it is deliberately awkward. ## Copy, never sync Off-box is where a longer history belongs, so the local fortnightly sweep must not reach through and delete it. A missed night catches up on the next run rather than leaving a hole. ## A separate credential Vitna's own bucket and Vitna's own token, not shared with the other project on the box. Sharing one would mean rotating either breaks the other, and a compromise of one exposes both. --- <!-- https://vitna.ai/docs/operations/checkout.md --> # Self-service checkout How somebody becomes a customer without anybody at Vitna doing anything, and what the box needs configured before they can. Two doors, and they are deliberately separate. The first opens an org and costs nothing. The second sells the plan that raises what that org may enter. An instance can have the first without the second, which is what a developer machine and CI both are. ## The fee shape | Plan | Stored ID | Workloads | Monthly | Annual | |---|---|---:|---:|---:| | Free | `free` | 1 | $0, no expiry | $0 | | Builder | `builder` | 3 | $29 a month | $290 a year | | Pro | `deploy` | 5 | $79 a month | $790 a year | | Team | `team` | 25 | $299 a month | $2,990 a year | | Enterprise | agreement | Contracted | Custom | Annual contract | The first workload is free indefinitely rather than for a trial period, because the first payment's job is commitment rather than revenue. A membership is free and always has been: an org may admit as many accounts as it likes. What is metered per plan is active workloads alongside hosted API and compute quotas (pages, rendered browser pages, Browser CDP sessions, research jobs, Search, and Unblocker, defined in `platform/billing/plans.mjs`). Note that `deploy` is kept as the stored plan id for Pro for backward compatibility with existing billing rows. The fee is charged per plan and never as a share of what anything routes. That is a commitment recorded in `VERTICALS.md` rather than a price point: a fee that moved with the recommendation would give every brief an incentive problem. Workloads or volumes above Team (`25` active workloads) are not sold self-serve; they are refused with a path to commercial contracting for an agreed volume and SLA. ## Opening an org A signed-in account with no membership sees a form rather than the command it used to be told to go and ask somebody to run. Submitting it writes one org and one membership, and makes that account the owner. The limit is one org per account, and it is a partial unique index on `aperture.org.opened_by_user_id` rather than a check in application code, so two browser tabs cannot both pass it. Somebody willing to hold several GitHub accounts can hold several free orgs, and no rule here changes that. Orgs an operator created carry a null in that column. They are outside the index and outside the allowance entirely: an org that arrived by agreement is on the terms of that agreement. Such an org is made with `org:create`, which has no flag that could set the column (see [access commands](../cli/access.md)): ```bash npm run org:create -- --id acme --name "Acme Ltd" ``` A self-service org that later signs an agreement is moved onto agreement terms with `org:agree`, which keeps its opener and records the date and a reference beside it. The register meters an org only where it has an opener and no recorded agreement, so the moved org is unmetered from that date: ```bash npm run org:agree -- --org acme --note "Invoice INV-2026-014" ``` `/console/billing` shows an org on agreement terms of either kind without a plan, an allowance count or a checkout, and says the register does not meter it, with the date and note where they were recorded. `/console/checkout` refuses a checkout for such an org. A moved org that still holds Deploy keeps its owner's button to the processor's page, because `org:agree` cancels nothing. **Joining an org that already exists is still a command.** No endpoint can add an account to an org that has members. That is `npm run org:member:add`, run by a person at a prompt, exactly as before. ## What grants an entitlement One thing only: a webhook event whose signature this instance verified against `STRIPE_WEBHOOK_SECRET`. The browser coming back from a payment page grants nothing. The success URL is a link anybody can visit, so `/console/billing?paid=1` says a payment page was finished and then reads the record to say what the record currently holds. For a few seconds after paying that is usually still the old plan, and the page says so rather than congratulating anybody. `billing.subscription_state` is append-only. The current state of an org is its newest row ordered by the processor's own `event_created_at`, never by arrival: a retried failure and a fresh cancellation can arrive in either order, and ordering by receipt would let a stale status overwrite a current one. A `past_due` subscription still carries its allowance. `unpaid` does not. The line is drawn where the processor stops retrying rather than where a card first fails. Nothing is ever taken away. An org whose plan lapses keeps every workload it entered; what it loses is the ability to enter another. ## What the box needs Environment variables in `/etc/vitna/env.site`, two written by the provision script and the Stripe/signup credentials set by hand: | Variable | What | Written by | |---|---|---| | `DATABASE_SIGNUP_URL` | The `vitna_signup` role | `provision-checkout.sh` | | `DATABASE_BILLING_URL` | The `vitna_billing` role | `provision-checkout.sh` | | `STRIPE_SECRET_KEY` | The processor's API key | By hand | | `STRIPE_WEBHOOK_SECRET` | The endpoint's signing secret | By hand | | `STRIPE_PRICE_BUILDER` | Stripe price id for `builder` ($29/mo) | By hand | | `STRIPE_PRICE_DEPLOY` | Stripe price id for Pro / `deploy` ($79/mo; fallback alias `STRIPE_PRICE_PRO`) | By hand | | `STRIPE_PRICE_TEAM` | Stripe price id for `team` ($299/mo) | By hand | | `STRIPE_PRICE_BUILDER_ANNUAL` | Optional annual Stripe price id for `builder` ($290/yr) | By hand | | `STRIPE_PRICE_DEPLOY_ANNUAL` | Optional annual Stripe price id for Pro / `deploy` ($790/yr; fallback `STRIPE_PRICE_PRO_ANNUAL`) | By hand | | `STRIPE_PRICE_TEAM_ANNUAL` | Optional annual Stripe price id for `team` ($2,990/yr) | By hand | | `VITNA_WEB_SIGNUP_ENABLED` | Set `"true"` once live signup is armed | By hand | | `VITNA_WEB_SIGNUP_SECRET` | Optional shared secret / bot check key for web signup | By hand | Note that `platform/billing/stripe-api.mjs` checks that all three paid tier prices (`builder`, `deploy`, `team`) are configured distinctly before checkout starts, and `refusePriceDisagreement` validates both monthly (`$29`, `$79`, `$299`) and annual (`$290`, `$790`, `$2,990`) Stripe session totals against `platform/billing/plans.mjs` before redirecting. After deploying `/etc/vitna/env.site`, run a pre-flight check against `GET /api/stripe` (confirming `configured: true` and all paid tier price mappings) and `POST /api/v1/keys/verify` on a provisioned test key before setting `VITNA_WEB_SIGNUP_ENABLED=true`. Run the provisioning before the migration deploys: ```bash bash /home/ubuntu/vitna-web/deploy/provision-checkout.sh ``` It creates the two roles with generated passwords and appends their connection strings. Re-running is safe and is the expected shape: the first run creates the roles and stops, and the run after the deploy proves the grants. A migration that reaches a box where the roles do not exist fails, and that failure is safe rather than silent. The post-receive hook aborts before the restart and the previous build keeps serving. ## The webhook endpoint `POST /api/stripe`, configured at the processor as a webhook destination. Four event types are acted on: ``` checkout.session.completed customer.subscription.created customer.subscription.updated customer.subscription.deleted ``` The first binds an org to an account at the processor and grants nothing; the three subscription events carry the entitlement. Any other event type is recorded as received and ignored, which is a 200: the processor sends what the account is configured to send, and refusing an event type nobody uses would put a permanent red mark on the dashboard for no reason. ## The processor account is shared, and that shapes the endpoint Vitna and Convexity bill under one processor account on purpose. A webhook destination is a property of the account rather than of a product, so every destination receives every event it subscribed to, for every customer on the account. Convexity's subscription events arrive at `vitna.ai/api/stripe`, and Vitna's arrive at Convexity's destination. **A subscription is this product's when it carries `vitna_org` in its metadata**, and it is nobody else's business when it does not. The key is namespaced rather than a bare `org` for exactly that reason: `org` is the obvious name, which is why the other product might use it, and two products silently claiming each other's subscriptions is the worst outcome available here. Every checkout this site opens writes `vitna_org` onto both the session and the subscription, so every renewal, failure and cancellation carries it for as long as the subscription lives. **A foreign event is ignored, never refused, and the status code is the whole point.** A refusal is a 400, which tells the processor to retry, and a retry of somebody else's subscription can never succeed. It would redeliver on backoff indefinitely, fill the journal with a word that means "look at me", and a destination that fails for long enough is disabled at the processor, which would take the real events down with the foreign ones. So a foreign event is answered 200, recorded once with outcome `ignored`, and logged as `stripe: foreign`. A run of `stripe: foreign` is the other product being used. It is not a problem and needs nothing done about it. **Signature failures are never told what was wrong.** The response says the request was not signed by this instance's processor. The specific sentence, and whether it was the timestamp or the digest, is in the journal. ### The edge blocks some clients before the app sees them The site is behind Cloudflare, which refuses a request from a user agent it reads as a bot with a `403` and an HTML body carrying `error code: 1010`. That answer never reaches this application, so nothing about it appears in the journal. Checked on 2026-08-31, posting the same body three ways: | User agent | Answer | |---|---| | `Stripe/1.0 (+https://stripe.com/docs/webhooks)` | `400`, reached the app | | absent | `400`, reached the app | | `Python-urllib/3.12` | `403`, blocked at the edge | The processor's own agent gets through, which is the case that matters. The one to know about is the third: a hand-rolled probe in Python looks like a blocked bot, answers `403`, and reads exactly like a broken endpoint. Send probes with `curl`, or post to `http://127.0.0.1:3200/api/stripe` on the box to bypass the edge entirely. ## The two secrets are checked in opposite directions `STRIPE_SECRET_KEY` opens a payment page. `STRIPE_WEBHOOK_SECRET` verifies what comes back. An instance holding the first and not the second can send somebody to pay and can never learn that they did, so check both are set before turning the endpoint on at the processor. An instance holding neither refuses both routes with a 503 and renders neither button, which is the intended state of anything that is not production. ## A published price that the processor would not charge The figure on the page lives in `platform/billing/plans.mjs` and the price id lives in an environment file, which makes them two claims about one number kept in two places. The checkout route asks the processor what it would charge, compares it with the published figure, and refuses to redirect anybody when the two disagree. It logs `checkout: price disagreement` at error level. That line means every visitor to the plan page is reading a figure this site will not honour, so it is worth waking somebody for. ## Reading it back ```bash journalctl -u vitna-web -g 'signup:' journalctl -u vitna-web -g 'checkout:' journalctl -u vitna-web -g 'stripe:' ``` An abandoned checkout is a row in `billing.checkout` with no subscription state after it, which is a question somebody can answer rather than an event that left no trace. `stripe: foreign` is the other product on the shared account and needs nothing. `stripe: refused` is the one word here that means something is wrong and somebody should look. --- <!-- https://vitna.ai/docs/operations/collector.md --> # The collector A oneshot unit fired hourly with jitter, running one [collector pass](../cli/witness/collect.md) out of the deployed checkout. Scheduled runs are tagged with a run trigger so they are told apart from a hand-run in the `fetch_run` table. ## What one pass does Reads every enabled document, archives each under the sha256 of its bytes, appends the observations it parsed, runs change detection, records what it saw and did not see, evaluates the armed triggers, and closes the run with a summary. ## Three phases, three transactions Detection, discovery and triggers each run in their own transaction with their own log line. A phase that fails is rolled back and named on stderr, and the run still records everything it read. That isolation matters most for the third one: a broken trigger must never cost the market record an hour of reads. ## A read reports three things | Outcome | Where it goes | |---|---| | Cells found | Observations | | Registered skus **not** found | An absence count | | Priced rows the registry never heard of | The [candidate queue](../cli/skus/candidates.md) | Three consecutive successful reads without a sku close its periods and mark it delisted. A read that carries it again opens a new period and makes it active. **No market event is written by any of that.** A delisting is what the record saw. Calling it a deprecation is a claim, entered by hand. ## The half-catalogue guard A read where more than half of a source's registered skus are absent is recorded as a parse failure with the count, whatever the adapter returned, and touches no period. Half a provider's catalogue vanishing in one hour is a page that changed shape. Delisting a whole board on that guess is the expensive mistake, and the guard exists because the cheap mistake, one hour of missing reads, is recoverable. ## One failing source does not fail the run Each source runs in its own transaction. A failure is recorded with its error, every other source still lands, and the run exits zero. Only a run that could not start at all exits non-zero. A source whose adapter the deployed revision does not carry is recorded as a parse failure naming that reason, so a configuration error is named rather than appearing as a source that quietly disappeared. --- <!-- https://vitna.ai/docs/operations/deploys.md --> # Deploys Push-based. The box never talks to GitHub. ## Why push and not pull The organisation disables repository deploy keys, and the house posture is push-based anyway. So the source of truth on the box is a bare repository, and deploying means pushing to it. ```bash git remote add vps ssh://ubuntu@<host>/home/ubuntu/vitna-web.git git push vps main ``` ## What the hook does A `post-receive` hook checks main out, runs `npm ci` and the build, applies migrations, seeds, and restarts the service. Every deploy appends to a log on the box. The order matters: it builds, migrates and seeds **before** it restarts anything, and aborts on the first failure with the previous build still serving. ## Automatically The `deploy` job runs on every push to main and `needs` both test jobs, so a red CI stops the deploy. It was its own workflow until it was folded in, which meant it raced CI rather than waiting for it. The mirror workflows push with the default token, which never triggers other workflows, so each carries its own copy of the deploy step gated on whether a commit actually landed. A shared concurrency group keeps the two from pushing at once. ## The credential is narrow The Actions key is bound on the box to a forced command permitting exactly one operation, receive-pack on that one bare repository. No shell, no pty, no forwarding. The workflow pins the host key rather than trusting first use. ## What the automatic check proves Git ignores a `post-receive` exit code and the hook writes to a log file, so a failed build or migration reaches Actions as silence: the push itself succeeded. The job reads the health endpoint afterwards and compares the revision it reports against the commit the job pushed. The hook records that revision into the environment file before it restarts, so it is the commit the running process was built from and not the one the bare repository points at. Those two disagree exactly when a deploy did not take, which is why a green deploy job is now the claim that **your commit is serving**, rather than only that something is. A box whose hook predates that check reports no revision at all, and the job refuses rather than passing on a claim it cannot check. Install the current hook, which is a copy and not a link: ```bash cp /home/ubuntu/vitna-web/deploy/post-receive \ /home/ubuntu/vitna-web.git/hooks/post-receive ``` ## A failed deploy rewinds the ref The hook fires only when a ref moves, so a bare repository left pointing at a revision that never deployed makes the obvious retry a no-op: git answers `Everything up-to-date`, nothing is built or migrated, and the job goes green against the build that is still serving. The hook rewinds `refs/heads/main` on every failure, which is what makes pushing again mean what it appears to mean, and gives the ref an honest meaning: the revision that last deployed, rather than the one that was last received. Read the log before retrying anyway. A push that appears to have done nothing has a reason, and the reason is on the box. Doing it by hand is written down in [verifying a deploy](verifying-a-deploy.md). --- <!-- https://vitna.ai/docs/operations/key-rotation.md --> # Rotating a signing key Replacing the key that signs customers' policies without refusing a single running runtime, and what a compromise needs instead. There is no scheduled rotation. Every rotation of the policy key costs every operator a restart, so it happens on a compromise, on the personnel change that implies one, or when the trust set is being edited anyway. `RUNTIME.md` records that decision, and this page does not reopen it. ## The two keys | Purpose | Signs | Private half lives in | Read by | Pinned by | |---|---|---|---|---| | `record` | Each day's Merkle root | `/etc/vitna/env.attest` | `vitna-attest` | Nobody | | `policy` | Configs and policies for customers | `/etc/vitna/env.policy` | `vitna-sign` | Every runtime, in `VITNA_PUBLIC_KEY` | The difference in the last column is the whole of why the policy key is the hard one. A record key can change on any day: every attestation row stores the key that signed it, so the record remembers. A runtime only knows what its operator told it. What is registered at any moment, with what each key signed, is at [`GET /api/v1/keys`](../api/v1/verification/keys.md) and on `/witness`. ## A planned rotation, in order The order is not a preference. Signing with the new key before operators pin it refuses every deployed runtime until each one catches up. **1. Make the pair and register the public half.** ```bash npm run witness:attest -- --new-key npm run witness:key:register -- --purpose policy --key <public half> --note "incoming, 2026-10-01" ``` `--new-key` prints both halves once and stores neither. Registering publishes the key and signs nothing with it. A key that has signed the record's roots is refused here, because the signer would then sign customer policies with the record's own key. Check it appeared: ```bash npm run witness:key:register -- --list curl -s https://vitna.ai/api/v1/keys/policy-key.pub ``` The `.pub` file now carries two keys, comma separated, oldest first. **2. Operators add the new key beside the old one, and restart.** ```bash VITNA_PUBLIC_KEY="<old key>,<new key>" ``` `vitna-runtime` 0.1.2 and later accept the set. Both keys verify, so the policy already on disk keeps loading and nothing is urgent. Tell operators, with the fingerprint from `/api/v1/keys` so they can check what they pasted, and give them time. This is the step that costs other people a restart, and nothing here can tell whether they have done it: a runtime never calls vitna.ai. **3. Sign with the new key.** Put the private half in `/etc/vitna/env.policy` as `VITNA_POLICY_SIGNING_KEY`. `vitna-sign` is a oneshot on a five-minute timer, so its next run reads it. It checks its own key against the register before signing anything and refuses a key that is not registered to sign policies, which is what makes step 1 enforceable rather than remembered. From here every new artifact carries the new key, and `/console/deployment` prints that key beside each one. Policies already downloaded still verify against the old key, which is still pinned. **4. Retire the old key, with a dated reason.** ```bash npm run witness:key:retire -- --key <old key> --reason "rotated 2026-10-01, replaced by the key registered that day" ``` The reason is public. `/api/v1/keys` serves it to anybody and `/witness` prints it beside the key, and no command here edits it afterwards, so write what a stranger may read. Retiring is a date and a reason, never a delete. Everything the old key signed stays checkable against it, and `/api/v1/keys` keeps listing it as retired. It drops out of `policyKey` and out of `policy-key.pub` at once. Operators drop the old key from `VITNA_PUBLIC_KEY` when they have a policy signed by the new one, and restart. Not before: a runtime restarted with only the new key refuses a policy file signed by the old one. ## A compromise Everything above assumes the old key is still trustworthy while operators catch up. After a compromise it is not, and the overlap in step 2 is the window an attacker uses. What changes, and what nothing here does for you. **Stop signing with it, now.** Remove `VITNA_POLICY_SIGNING_KEY` from `/etc/vitna/env.policy`. The next `vitna-sign` run refuses to start and signs nothing, which is the right state until the replacement is ready. **Retire the old key next, before anything is registered.** ```bash npm run witness:key:retire -- --key <old key> --reason "compromised 2026-10-01" ``` The order is the point. `policy-key.pub` serves every policy key that is not retired, so a replacement registered first would sit beside the compromised key in that file, and `/api/v1/keys` would call the pair a rotation in progress and say to pin both. Anybody who read the file in that gap would pin the key this section exists to get rid of. Retiring first has no dependency on the replacement: the signer already refuses a retired key. Until the replacement is registered, `policy-key.pub` answers `404` `no_policy_key` and `/api/v1/keys` says no key is registered to sign policies. That is the true state, and it is the safe one to be read. Both answers are cached for a minute and never served stale past it, so a minute after the retire nobody is handed the old set. The reason is public, served by `/api/v1/keys` and printed on `/witness`, and it cannot be edited afterwards. Say that the key was compromised and when, and keep how it happened, which host, and what is known in the incident record. **Register and install the replacement without waiting for operators.** Steps 1 and 3 above, back to back. There is no overlap to protect, because an operator still trusting the old key is exactly the exposure. Be exact about what retirement does and does not do. It publishes that the key is out of service. It does not stop a runtime trusting it: that runtime cannot be told, by design, and a key stops being trusted there only when its operator removes it from `VITNA_PUBLIC_KEY`. So every operator has to be contacted directly, with the new key's fingerprint, and told to remove the old key rather than add the new one beside it. **Re-signing is not automated.** A signed artifact is an append-only row and is never rewritten, so replacing an organisation's policy means a new ask answered by the new key. Today that ask is made per organisation: an owner presses **Ask for a signed policy** at `/console/deployment`, or an operator writes one ask per organisation into `aperture.artifact_request` as the owning role. There is no command that asks for every organisation at once, and one that writes queue rows in bulk should arrive as its own reviewed change rather than as a statement typed during an incident. Each operator then downloads the new file, sets `VITNA_PUBLIC_KEY` to the new key alone, and restarts, in that order. **What cannot be undone.** A policy the attacker signed with the old key verifies against the old key forever. The register can say the key was retired and when; it cannot say which signatures made before that moment were Vitna's. `aperture.artifact` is the list of what Vitna actually signed, digest by digest, and a file whose digest is not in that organisation's listing did not come from the signer. ## If the record key is the one compromised Nothing pins it, so there is no operator step. Generate a pair, put the private half in `/etc/vitna/env.attest`, and the next nightly run signs with it; the new key enters the register the moment it signs a root. Register the old key and retire it with the reason, since a key known only from attestation rows has no row to retire until it is registered. Roots signed with the old key before the compromise stay checkable, because each was published on its day and a later forgery has to contradict what was already out. Roots dated inside the compromise window are the ones to treat as unproven. --- <!-- https://vitna.ai/docs/operations/rolling-back.md --> # Rolling back Most failures roll themselves back. One kind needs a person. ## The kind that rolls itself back The hook builds, migrates and seeds before restarting anything, and aborts on the first of those that fails. The previous build keeps serving and the service never moved. The log names which step failed. Nothing to undo. Fix it and push again. ## The kind that needs a person A deploy that succeeded and should not have. Push the last good commit to the box. Because that moves the branch backwards it has to be forced: ```bash git push --force vps <last-good-sha>:refs/heads/main ``` The hook treats it as any other deploy, so [verify it](verifying-a-deploy.md) the same way. ## Two things a push cannot undo **Migrations are forward-only.** An older build meets a newer schema. The schema is additive by construction and an older build ignores what it does not know about, so this is usually fine. A migration that **dropped or narrowed** something is not a rollback, it is a restore from the nightly dump. **The bad commit is still the remote's main.** The next push redeploys it. Revert it there too, or the box returns to the state you just left. That second one is the trap, because the box is correct and the repository is not, so everything looks fixed until somebody else pushes. --- <!-- https://vitna.ai/docs/operations/verifying-a-deploy.md --> # Verifying a deploy A push that exits zero is not a deploy. Three checks say which commit is live. ## The three | Check | What it proves | |---|---| | The deploy log ends with a done marker | The hook ran to completion | | The service restart timestamp is newer than the push | The process actually moved | | The served page carries a change-unique marker from the pushed commit | The bytes in front of users are yours | All three, in that order. Any one of them alone can read as success while the deploy did not happen. ## Why a change-unique marker Not the title, not a version string that a cached build would also carry. Some string that exists only in the commit you just pushed. The failure this catches is the one that looks most like success: the hook aborted during the build, the previous build is still serving, and it is serving healthily. Health checks pass. The site is up. The code is old. ## What the automatic check covers The deploy job reads the health endpoint and fails unless three things hold at once: it answers `200`, its database is reachable, and the revision it reports is the commit that job pushed. That third one is the check that used to be missing, and it is the one that catches a failed build. The hook records the revision it restarted, so the endpoint reports what the running process was built from; a build that aborted leaves the previous revision reported, and the job goes red naming both. A box whose hook predates that reports no revision, and the job refuses rather than passing on a claim it cannot check. ## Re-pushing retries, but read the log first A failed deploy rewinds `refs/heads/main` on the box, because the hook only fires when a ref moves and a ref left advanced turns the obvious retry into a silent no-op. So pushing again does now retry. That is a repair, not an invitation. A push that appears to have done nothing has a reason, the reason is in the deploy log, and retrying before reading it turns one question into two. --- <!-- https://vitna.ai/docs/search/overview.md --> # Vitna Search Private beta. A separate search API for agents with independent Google/Bing SERP retrieval and Vitna's own crawler and web index. SERP requests return actual search-engine organic listings; indexed search and cited research use successfully indexed pages. It includes a browser interface, dedicated client keys, and shared request limits. The private-key beta is hosted at `https://search.vitna.ai`, with an API reference at `/docs`. Its product page and a playground are at `/search` on vitna.ai. Keys are issued by invitation, requested at [vitna.ai/start](https://vitna.ai/start#search). Customer signup and search billing are not enabled. ## The service Use `POST /v1/serp` to retrieve actual Google or Bing text search result pages through Vitna's transport and parser, without a Bright Data call. The compatibility alias `POST /request` accepts the supported URL-based request shape. This alpha returns parsed organic listings or fetched HTML; it does not promise full vendor coverage for ads, shopping, maps, other SERP features, geographic egress, or challenges. See the [SERP API contract](serp.md) for requests, operator setup, and limitations. The native contract returns ranked URLs, titles, excerpts, source identity, and observed crawl timestamps, and on request each page's stored text and verbatim highlights ([inline contents](#inline-contents)). It uses keyword retrieval with title-weighted BM25 ranking by default, removes duplicate URLs and fragments, and identifies result origins in `provenance`. Among its first 20 candidates, a result list leads with at most two pages per host (`www.` aside), and that host's further candidates follow the other sites'. Nothing is dropped, so a search confined to one site still fills its page. An instance configured with Vitna's own embedding engine adds [semantic and hybrid retrieval](#semantic-retrieval) and [similar pages](#similar-pages). A result carries a publication date only where the page itself declares one ([declared publication dates](#declared-publication-dates)); it never infers one, and it does not generate an answer. Query-guided expansion can discover and crawl a bounded set of pages, then make those pages searchable in Vitna's own index. Exact single-brand queries prefer a matching indexed homepage under `.com`, `.ai`, or `.org`, including its `www` form. The homepage must still match the query and satisfy all filters; other results keep their BM25 ordering. Running an instance is operator work, from a checkout of Vitna's private repository, and is described at the end of this page under [Operator reference](#operator-reference). Nothing there is needed to call the hosted API. The process is separate from the site and the market collector. It has no access to the site's database and holds no signing key. Its optional SQLite search file belongs to this service alone. Provider search credentials belong only in this process's environment, not in the site's environment. Runtime search results and usage never become Witness price observations. ## Google/Bing SERP ```text POST /v1/serp POST /request ``` Send the search bearer key and JSON content type. For an actual Google result page: ```json {"engine":"google","query":"lunar science","country":"us","language":"en","maxResults":5,"format":"json"} ``` The response contains `general`, `input`, `organic`, and `meta`, with `meta.provenance: "search_engine"` and `meta.coverage: "organic_only"`. The organic entries expose `rank`, `title`, and `link`, with optional `description`, `display_link`, and observed `sitelinks`. Use `engine: "bing"` for Bing or `format: "html"` for the fetched HTML. SERP retrieval does not require an index, start a native crawl, or produce cited research. Read the complete [SERP contract](serp.md) before migrating a URL-based client. For up to 100 distinct Google links across observed result pages, use the asynchronous [multi-page jobs](serp.md#multi-page-google-jobs) contract. Its target can finish partial; the single-page endpoint remains capped at ten. Collection requests default to `fresh: false`: exact same-client requests can join an active job or reuse a target-complete result for thirty seconds. Use `fresh: true` to bypass completed reuse. Capture timestamps remain unchanged, and cancelling a shared job affects its joined callers. Enabled SERP retrieval defaults to `auto`: ordinary routes prefer a browser for Google and direct HTTPS for Bing, or pinned proxy browser routes when proxies are configured. A single deadline bounds distinct attempts, route concurrency, and cooldowns. Successful JSON results have a short process-local cache; send `fresh: true` for a new capture. Read the complete [retrieval policy](serp.md#retrieval-routes-and-retries) for operator settings and safe route/cache metadata. These controls do not establish production access to a search engine. ## Search ```text POST /v1/search ``` Send `Content-Type: application/json` and a search bearer key. For example, with the issued credential in the caller's `VITNA_SEARCH_KEY` environment: ```bash curl https://search.vitna.ai/v1/search \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"query":"agent search APIs","mode":"index","maxResults":5}' ``` | Field | Default | Allowed values | |---|---|---| | `query` | Required | 1 to 1,000 characters, no control characters | | `maxResults` | 5 | Integer from 1 to 100 | | `mode` | Explicit `index` recommended; browser default | `index`; mutually exclusive with `provider` and `fallback`; removed `live` and `hybrid` values return HTTP 400 | | `provider` | Legacy `auto` when `mode` is omitted | `auto`, `vitna`, `brave`, `exa`, `tavily`; cannot accompany `mode` | | `fallback` | Legacy `false` | Boolean; `true` only with legacy `auto`; cannot accompany `mode` | | `domains` / `excludeDomains` | Unset | 1–10 public hostnames; includes subdomains; `index` mode or legacy Vitna without external fallback | | `maxAgeDays` | Unset | Integer 1–3,650; observed crawl age; `index` mode or legacy Vitna without external fallback | | `startPublishedDate` / `endPublishedDate` | Unset | A date (`YYYY-MM-DD`, a whole UTC day) or an ISO 8601 date-time with a UTC offset, both inclusive; only pages that [declare a publication date](#declared-publication-dates) can match; `index` mode or legacy Vitna without external fallback | | `asOf` | Unset | Point-in-time snapshot cutoff (`YYYY-MM-DD`, end of UTC day, or ISO 8601 date-time with a UTC offset); searches indexed snapshots captured on or before that instant with zero lookahead, disables coverage expansion and paid fallback (`as_of_requires_vitna`), and echoes `meta.asOf` | | `index` | `web` | `web` or one of the 23 vertical indexes reported by `GET /v1/indexes` | | `effort` | Unset | `fast`, `balanced`, or `deep`; `fast` prefers keyword retrieval without reranking, `balanced` uses configured hybrid retrieval without default excerpts, and `deep` attaches `contents.excerpts` and applies native reranking when available; echoed in `meta.effort` | | `retrieval` | `hybrid` where [semantic retrieval](#semantic-retrieval) is configured, otherwise `keyword` | `keyword`, `semantic`, or `hybrid`; Vitna index without external fallback; `semantic` cannot accompany `additionalQueries` | | `contents` | Unset | `text`, `highlights`, and/or `excerpts` read from stored pages, optional `maxTotalCharacters` / `maxTotalTokens` budget, and `injection` guard; see [inline contents](#inline-contents); Vitna index without external fallback | | `expand` | Unset; permits automatic name/topic discovery | Boolean; `true` requests bounded expansion; `false` opts out of automatic discovery | | `discover` | Unset; permits automatic name/topic discovery | Boolean; explicit `true` requires `expand: true`; `false` disables site discovery | | `urls` | Unset | 1–8 public HTTPS page URLs without query strings; requires `expand: true` and must satisfy domain filters | Unknown fields are refused. Request bodies are limited to 8,192 bytes. `maxResults` above 10 needs `mode: "index"` or legacy `provider: "vitna"`; a request that may reach an external provider keeps 1 to 10. The caller cannot set upstream provider endpoints, headers, or credentials. Domain, crawl-age and publication-date filters, and `retrieval`, require the Vitna index with external fallback disabled; otherwise the filters return HTTP 400 `filters_require_vitna` and `retrieval` returns HTTP 400 `retrieval_requires_vitna`. With `discover: true`, `domains` is limited to three hostnames and supplied URLs to three distinct origins; ordinary Index domain filters still allow ten. Use `mode: "index"` for Vitna's stored corpus. It makes no paid search-engine call, including when it returns no matches. `maxResults` limits the immediate search response; it does not increase crawl work. A whole-request search deadline is separate from the background job's bounded crawler work. An index search can return up to 100 results. A longer list extends a shorter one: its first results are those a 10-result search returns, with the same host spread among the first 20, and later results follow in relevance order. Every list keeps its duplicate suppression and filters. Caller-supplied `additionalQueries` are read as deep as the list asks, ten pages each or `maxResults` when larger, and pages found only below each query's top ten follow the rest; `meta.index.retrieval.perQueryLimit` reports the depth. On a synthetic 50,000-page index with 384-dimension vectors, measured in one process on a Windows arm64 laptop and without the query embedding, 100 results took 30 ms median and 41 ms at the 95th percentile by keyword, against 11 ms and 13 ms for 10, and 61 ms and 77 ms hybrid, against 35 ms and 38 ms: far inside the default 10-second deadline. A repeated search is answered from the query workers' 30-second cache, which weighs a cached answer by its results, ten to a unit, so a 100-result answer takes the room of ten 10-result ones. An empty native name/topic search, or a topic search with only partial-term matches, can automatically enqueue discovery when enabled, with no `domains` or supplied `urls` and no external fallback. For example, this illustrative name query needs no crawl flags: ```json {"query":"examplebrand","mode":"index"} ``` When configured, automatic discovery first uses its own directory of observed names. Explicit expansion can also use matching known URLs. A narrow distinctive single ASCII brand can also produce up to three candidate hosts using `.com`, `.ai`, and `.org`. Those hosts are suggestions, not verified official identities. Name detection is heuristic; common topic words and multiword topic queries produce no guessed hostnames. Broad queries can match a publisher catalog's descriptions or, when configured, a local English Wikipedia title catalog to select candidate pages. Catalog entries are discovery starting points, not indexed coverage. A query without a suitable name, topic, or concept match can return `no_candidates`. This is not whole-web search. Current results return first with any job in `meta.expansion`; automatic admissions include `meta.expansion.automatic: true`. Send `expand: false` or `discover: false` to prevent automatic discovery. For explicit target-only refresh, send `expand: true, discover: false`. For explicit site discovery, send `expand: true, discover: true`; domains remain optional guidance, not a prerequisite. With `expand: true`, current indexed results return with `meta.expansion`: a durable job with a `statusUrl`, or an explicit `no_candidates`, `queue_full`, `client_job_limit`, `discovery_unavailable`, or `unavailable` status. Admission failure does not discard valid indexed results. With `discover: false`, the job crawls known relevant URLs and supplied targets with scope `known_urls_and_supplied_targets`. Add `discover: true` to examine selected sites' homepages and sitemaps before choosing target pages. Optional `domains` guides site selection; without it, explicit discovery can use supplied URL origins, matching known URLs, observed names, narrow name candidates, topic-catalog matches, or configured concept-title matches. Automatic discovery skips incidental known-URL matches and uses the name or topic planners. The job has scope `site_discovery`, with at most three origins, 2,000 sitemap candidates, eight target pages, 40 total HTTP requests, and 60 seconds of crawler work. Candidate names do not establish official identity or promise whole-web coverage. An initialized empty index may accept expansion. See [coverage expansion](general-web.md#query-guided-coverage-expansion). Bounded lexical and site hints, including names, candidate origins and their provenance, are encrypted in private jobs that expire after 24 hours. The raw original query and full request are not stored or logged. Deduplication uses a private HMAC digest. Plaintext queries and terms are absent from jobs and receipts; metadata receipts still record request outcomes and usage. The operator must configure a private `VITNA_SEARCH_DEMAND_SECRET_FILE` before accepting site-discovery jobs. Authenticate `GET /v1/search/jobs/<id>` with the same client's key to receive `{"job": {...}}`. Status is `queued`, `running`, `completed`, or `failed`. Completed means the bounded pass finished; inspect its summary for indexed, blocked, failed, and unmet targets, then repeat the original search to retrieve fresh results. A manual rerun can send `expand: false, discover: false` to retrieve only the updated corpus. `summary.discovery` reports integer counters for `origins`, `candidates`, `selected`, and `unavailable`. Selected candidates are not indexed pages; inspect `summary.indexed`, `summary.unchanged`, and unmet targets for actual coverage. Job checks have a separate 60-per-minute limit and don't consume the search quota. Jobs expire after 24 hours. Unknown, expired, and other clients' jobs return the same 404 response. ## Semantic retrieval Semantic and hybrid retrieval are optional. They embed text with Vitna's own embedding engine: vitna-anchor's OpenAI-compatible `POST /v1/embeddings`, which runs on the search host as `vitna-embed.service` at `127.0.0.1:8795` for query embeddings and `vitna-embed-batch.service` at `127.0.0.1:8797` for vector maintenance, serving `bge-small-en-v1.5` (384 dimensions). No hosted embedding, rerank, or search API is called. Both stay off unless `VITNA_SEARCH_EMBED_URL` and `VITNA_SEARCH_EMBED_MODEL` are set, and an unconfigured instance answers exactly as before. | `retrieval` | What runs | |---|---| | `keyword` | The BM25 retrieval described above | | `semantic` | The pages whose stored vectors are nearest the query's embedding, by cosine similarity | | `hybrid` | Reciprocal rank fusion, with constant 60, of the top 20 keyword candidates and the top 20 nearest pages | When the two hybrid lists swap a pair, their fused scores tie exactly and the embedding order decides. A request for more than 20 results reads both lists that deep; pages found only below both lists' top 20 follow the fused top 20, so the leading results stay those of a shorter request. A configured instance defaults to `hybrid`. Every mode applies the same enabled-source, domain and crawl-age filters, suppresses duplicate extracted content, honours `maxResults`, and labels results `kind: "index"` in `provenance`. The domain and crawl-age filters restrict the nearest-page scan itself, and the scan widens past copies of a nearer page, so neither empties a list that other admitted pages can fill. A matching brand homepage still leads only when it also matched the query's words. A page found only by its vector shows a query-focused excerpt when a query term occurs in it, otherwise its leading words. `meta.index.retrieval` reports what ran. Illustrative values: ```json {"requested":"hybrid","used":"hybrid","model":"bge-small-en-v1.5","vectors":49012,"candidates":{"keyword":20,"semantic":20}} ``` It is present whenever semantic retrieval is configured or a request names a mode. `vectors` counts pages with a stored vector, which can trail `meta.index.documents` until the maintenance job catches up. `meta.index.match` keeps its lexical meaning: it describes the keyword half of a hybrid search, for the first query as in keyword retrieval, and is absent from a `semantic` one. Automatic discovery follows the keyword half, so a `hybrid` search queues it exactly when a keyword search would have: when that half found nothing (`candidates.keyword` is 0) or matched only some terms (`partial`), even though nearest pages fill its results. Pages that `additionalQueries` found count as found, in either mode. An opted-in `semantic` search queues discovery only when empty. The search never fails because of semantic retrieval. When the embedding call fails or exceeds `VITNA_SEARCH_EMBED_TIMEOUT_MS`, or no compatible vectors are loaded, the response carries keyword results with `used: "keyword"` and a `fallback` code: `semantic_not_configured`, `embed_unavailable`, `embed_timeout`, `embed_paused`, `embed_rejected`, `embed_invalid_response`, `embed_model_mismatch`, `vectors_unavailable`, `vectors_incompatible`, `vectors_over_limit`, or `snapshot_as_of` (when `asOf` runs lexical retrieval over historical snapshots). A fallback result is not cached. Receipts record the retrieval used and any fallback code, never the query. After a query embedding times out or the engine cannot be reached, query embeddings pause for 30 seconds and searches report `embed_paused`, so an outage costs one timeout per pause rather than one per search. The first search after the pause tries the engine again while the rest stay paused, and a success ends the pause. Maintenance batches never pause. Because the embedding wait counts against the request deadline, startup refuses a `VITNA_SEARCH_EMBED_TIMEOUT_MS` above half of `VITNA_SEARCH_TIMEOUT_MS`, which leaves the keyword fallback at least half the deadline. Page vectors live in their own SQLite file, `vectors.sqlite` beside the index unless `VITNA_SEARCH_VECTORS` names another, and never in `index.sqlite`. The file carries its own SQLite application id. The maintenance job refuses a vectors path that names the index, frontier, concept catalog or quota file, and the store refuses any file that belongs to another database; a vectors file from before the id existed is adopted, header only, on the job's next run. A file holds one model: its identifier and width are recorded once, and the API and the maintenance job refuse any other, so changing models means a new file. Each row keeps the fingerprint of the text embedded: by default the page title and its leading text, up to `VITNA_SEARCH_EMBED_MAX_CHARACTERS` (1,500) under a conservative token estimate, because single-window `bge-small-en-v1.5` refuses texts over its 512-token window with HTTP 400 `context_length_exceeded` rather than truncating them. Operators can enable bounded overlapping passage chunking (`VITNA_SEARCH_EMBED_MAX_CHUNKS`, `VITNA_SEARCH_EMBED_CHUNK_OVERLAP`, `VITNA_SEARCH_EMBED_MAX_DOCUMENT_CHARACTERS`) and configure multilingual or long-context models (`VITNA_SEARCH_EMBED_MAX_TOKENS`, `VITNA_SEARCH_EMBED_MULTILINGUAL`); chunked matches deduplicate to the canonical document and pass matched passage byte/code-point offsets into `contents.highlights`. A maintenance job keeps the file current; operators run it as described under [Maintaining the vectors file](#maintaining-the-vectors-file). The query embedding is made on the API's main thread, so a slow engine never occupies a query worker, and its failure becomes keyword retrieval before any worker is involved. The nearest-neighbour scan runs in each query worker beside BM25. Each worker holds its own copy of the vectors: vectors × (width × 4 + 8) bytes, room for about 5% growth, and their URLs. For 49,063 pages at 384 dimensions that measured 79.5 MB per worker, with a 434 ms full load. `VITNA_SEARCH_VECTORS_MAX_MB` (512 per worker) bounds each copy. Workers apply new vectors and removals incrementally when the file changes: 500 changed vectors took 5.3 ms. A file larger than that bound, such as one holding vectors for millions of imported Wikipedia articles, is searched through an approximate nearest-neighbour index instead, which the operator [builds](#maintaining-the-vectors-file) beside it (`vectors.ivf` beside `vectors.sqlite`). It is an inverted-file index: the vectors are split into up to 4,096 lists by two levels of spherical k-means, and each vector keeps one bit per dimension of its difference from its list's centre plus that difference's mean size, 52 bytes at 384 dimensions instead of 1,544. A search scores every list centre, estimates each vector in the `VITNA_SEARCH_VECTORS_PROBES` (64) nearest lists with one table read per eight dimensions, and rescores the best `VITNA_SEARCH_VECTORS_RESCORE` (400) estimates exactly from the floats stored in `vectors.sqlite`. Vectors stored or replaced after the index was built are held and searched exactly beside it, and a removed vector drops out because rescoring finds no row, so the index stays usable between rebuilds; the index and those newer vectors together must fit `VITNA_SEARCH_VECTORS_MAX_MB`. The index is plain JavaScript over typed arrays, like the exact scan, so the search host needs no native build. A file within the bound is always searched exactly. A file over the bound without an index built from it (a missing index, one from another vectors file or model, or more newer vectors than fit) answers with `vectors_over_limit`, and a damaged index with `vectors_unavailable`; either way the search falls back to keyword results as below. A query worker reads the index in the background, since at millions of vectors that takes longer than a query's deadline: after it starts, its semantic and hybrid searches fall back with `vectors_unavailable` until the index is in, and after a rebuild the index in use keeps answering until the new one replaces it. Limitations: - The default single-vector `bge-small-en-v1.5` configuration embeds only English text well; multilingual retrieval requires deploying a multilingual profile (such as `bge-m3` or `multilingual-e5-small`/`multilingual-e5-base` with `VITNA_SEARCH_EMBED_MULTILINGUAL=true`). - Unless multi-passage chunking (`VITNA_SEARCH_EMBED_MAX_CHUNKS > 1`) is enabled, only a page's title and leading text (within `VITNA_SEARCH_EMBED_MAX_CHARACTERS` and `VITNA_SEARCH_EMBED_MAX_TOKENS`) are embedded, and text further down a page is found by keyword retrieval alone. - Nearest neighbours always exist, so `semantic` returns the closest pages even when none is relevant; there is no relevance threshold. On the authored negative queries, semantic and hybrid retrieval answered none with an empty result, where keyword retrieval answered 8% to 33% of them correctly. - Within the bound the scan is brute force and grows linearly with the number of vectors and their width. It measured 24.7 ms median for 49,063 vectors of 384 dimensions on a desktop Ryzen 7 3700X; the default 512 MB bound holds about 347,000. Past that, the approximate index answers instead, and can miss some of the exact nearest pages; see the [measurement](general-web.md#wikipedia-at-full-scale). - A search embeds its query first: about 10 ms on an idle engine, longer while a maintenance batch is in progress, because the engine serves one request at a time. - New and changed pages are found by keyword alone, or by their previous vector, until the next maintenance run. - Source brief research and indexed text extraction remain keyword based. The [general web beta](general-web.md#measure-coverage-before-expanding-access) records the measured comparison of the three modes. ## Legacy provider routing Legacy gateway instances can omit `VITNA_SEARCH_INDEX`, explicitly select `VITNA_SEARCH_PROVIDERS=brave,exa,tavily`, and configure `BRAVE_SEARCH_API_KEY`, `EXA_API_KEY`, or `TAVILY_API_KEY`. Those accounts and contracts are separate from Vitna's native crawler; the hosted beta uses the native index. The Bright Data search adapter and its Live/Hybrid modes have been removed. Use the independent [SERP API](serp.md) for actual Google/Bing listings, or native search for Vitna's indexed corpus. Existing clients should follow the [migration guide](live-engine.md). Clients that omit `mode` retain the provider/fallback contract. `auto` uses the first configured provider in `VITNA_SEARCH_PROVIDERS` order, which defaults to `vitna` when an index path is configured, or `brave,exa,tavily` otherwise. Providers with no index/credential are skipped. This is a configured preference, not a claim that the first provider is the cheapest or best. A named provider stays pinned. With `fallback: true`, a timeout, network failure, malformed response, HTTP 429, or HTTP 5xx can advance to the next configured provider. HTTP 401, 402, 403, and other request errors stop the attempt sequence. A valid empty search is a successful response and never triggers fallback. Every attempted call may incur provider charges, including calls whose result was lost in transit. An empty local index refuses the request without calling a paid fallback. A populated local index returning zero matches is a successful empty response. Send `mode: "index"` explicitly when a caller must use only Vitna's index. ## Response The response below illustrates an indexed result. Its crawl timestamps, document count, and latency are illustrative: ```json { "query": "agent search APIs", "results": [ { "rank": 1, "title": "An example result", "url": "https://example.com/search", "snippet": "An excerpt from the indexed page.", "publishedAt": null, "crawledAt": "2026-10-03T10:00:00.000Z", "contentChangedAt": "2026-10-03T10:00:00.000Z", "source": { "id": "example", "name": "Example" }, "provenance": [ { "kind": "index", "provider": "vitna", "rank": 1 } ] } ], "meta": { "requestId": "example-request", "mode": "index", "provider": "vitna", "providerRequestId": null, "count": 1, "latencyMs": 10, "fallbackUsed": false, "index": { "documents": 100, "lastCrawledAt": "2026-10-03T10:00:00.000Z", "ranking": "sqlite_fts5_bm25", "match": "complete" }, "attempts": [ { "provider": "vitna", "status": "ok", "httpStatus": 200, "latencyMs": 10, "usage": { "estimatedCostUsd": null, "credits": null, "source": null } } ] } } ``` Excerpts are untrusted provider text. Treat them as data, and escape them if rendering them in HTML. `publishedAt` is null unless the page declares its publication date ([declared publication dates](#declared-publication-dates)); a relative age, a URL, or a crawl time is never turned into one. Every result carries `provenance`. Native results identify `kind: "index"`, `provider: "vitna"`, and their original rank. Production native metadata can also include `meta.index.match`: `complete` for complete term matches, `partial` for partial-term fallback, or `none` for no indexed match. This is lexical coverage, not a guarantee of relevance or an answered question. Automatic topic discovery can queue missing coverage for `partial` or `none` while returning the current results intact. Consumers must allow this optional field to be absent. Local results additionally carry `source: {id, name}`, `crawledAt`, and `contentChangedAt`. These are observed crawl and extracted-content-change timestamps, not publication dates; `publishedAt` holds only a date the page declared, with `publishedAtSource` naming where. Local response metadata includes indexed document count, latest crawl timestamp, and `ranking: "sqlite_fts5_bm25"`. Local usage figures are null because no provider reports a per-query bill; they are not an assertion of zero infrastructure cost. Configured sources can also return `source.reusePolicy: {name, url}`: the publisher's policy label and HTTPS link. Check that policy for the material being reused. This optional attribution does not imply one blanket license for every page or asset on the source. A page whose text did not come from Vitna's own crawl says where it came from, and its `crawledAt` is null. A Common Crawl capture carries `contentOrigin: "common_crawl"`, `capturedAt`, `importedAt`, and `archive: {collection, digest}`. An article from an imported Wikipedia dump carries `contentOrigin: "wikipedia_dump"`, `capturedAt` (when the snapshot's revisions were current), `importedAt`, and `archive: {collection, revision}`, for example `{"collection": "enwiki-20261001", "revision": 1374562957}`; its `contentChangedAt` is that revision's edit time, and its source's `reusePolicy` names CC BY-SA 4.0. Wikipedia text is reusable under that license when the reuse credits Wikipedia's contributors and links the article, which the result's `url` is. Crawl-age filters read `capturedAt` for these pages. See [Import English Wikipedia](general-web.md#import-english-wikipedia). ## Declared publication dates A native result's `publishedAt` is the publication date its publisher declares in the page, read when the page is crawled or imported from an archive. Three declarations are read, and the first that gives one value wins: | `publishedAtSource` | Where the page declares it | |---|---| | `article_published_time` | A `<meta property="article:published_time">` (or `name=`) tag | | `json_ld_date_published` | `datePublished` on a JSON-LD article or posting, or failing that on the page itself (`WebPage` and its kinds), at the top level, in `@graph`, or as their `mainEntity`; a site's or a listed item's date is not read | | `article_time_element` | A `<time datetime>` inside an `<article>`: the one marked `itemprop="datePublished"` or `pubdate`, or else the only value the article's time elements give | When a source gives two different values, such as several articles' dates on one listing page, it declares nothing and the next source is read. A value with a time and a UTC offset is returned as that instant in UTC, `"2024-05-01T08:30:00.000Z"`. A bare date, or a time without an offset, stays the calendar day the page named, `"2024-05-01"`, because the zone it was written in is unknown. A date more than a day after the crawl, a date before the year 1000, and the Unix epoch's first day, which content systems print for an empty field, are not stored. Pages without a declaration have `publishedAt: null` and no `publishedAtSource`, exactly as before. Nothing is inferred: not from a relative age such as "3 days ago", a URL path, visible text, a modification date, an HTTP header, or the crawl time. A page crawled before the index stored dates has none until it is crawled again, and an article imported from a Wikipedia dump has none: the dump declares revision times, not publication dates. `startPublishedDate` and `endPublishedDate` filter on this declaration and apply only to pages that have one: a page without a declared date never matches a date filter, so a filtered search covers less of the index than an unfiltered one. Both bounds are inclusive. A date bound means the whole UTC day; a date-time bound needs a UTC offset and is compared as the instant it names. A page that declares only a date matches when that date falls within the bounds' dates. Semantic and hybrid retrieval restrict their nearest-page scan to dated pages in range; through the [approximate index](#semantic-retrieval) the range is applied as candidates are read back, as the crawl-age window is, so a narrow range there can return fewer pages than it admits. ```json {"query":"lunar missions","mode":"index","startPublishedDate":"2024-01-01","endPublishedDate":"2024-12-31"} ``` The declaration is the publisher's claim, not a verified fact. A publisher can backdate a page, or declare an original print date for a page posted later. ## Inline contents A native search can return each result's stored page text and verbatim highlights with the results, so a caller needs no extraction request per URL. Send `contents`: ```json {"query":"Galilean moons","mode":"index","maxResults":5,"contents":{"text":{"maxCharacters":4000},"highlights":true}} ``` | Field | Default | Allowed values | |---|---|---| | `contents.text` | Off | `true` for up to 20,000 characters per result, or `{"maxCharacters": n}` / `{"maxTokens": t}` (1 to 100,000 characters, 4 characters per token, mutually exclusive) | | `contents.highlights` | Off | `true` for a 1,000-character passage budget per result, or `{"maxCharacters": n, "maxTokens": t, "query": "..."}` (1 to 4,000 characters, optional 1-1,000 character query) | | `contents.excerpts` | Off (auto on `effort: "deep"` for native index searches with extraction) | `true` for a 1,000-character contiguous excerpt window per result, or `{"maxCharacters": n, "maxTokens": t, "query": "..."}` (1 to 4,000 characters) | | `contents.maxTotalCharacters` / `contents.maxTotalTokens` | Unset | Positive integer response-wide character budget from 100 to 400,000 (`maxTotalTokens` from 25 to 100,000 at 4 characters per token, mutually exclusive); distributes budget across results and redistributes unused slack fairly | | `contents.injection` | Off | `"scan"` or `"strip"`; scans extracted `text`, `highlights`, and `excerpts` for prompt-injection phrases and invisible Unicode controls, attaching `contents.injection` (and stripping invisible controls when `"strip"`) | | `contents.snapshotAsOf` | Unset | Point-in-time snapshot cutoff (`YYYY-MM-DD` or ISO 8601 date-time with UTC offset) for historical inline extraction; when paired with `asOf`, both values must match | `false` leaves a part out, and a `contents` object that asks for no content part (`text`, `highlights`, or `excerpts`) returns an error. Other fields are refused. Contents require the Vitna index with external fallback disabled, otherwise HTTP 400 `contents_requires_vitna`, and an instance with page extraction, otherwise HTTP 503 `extraction_unavailable`. Both are checked before the request is admitted. When `contents.maxTotalCharacters` or `contents.maxTotalTokens` is requested, `meta.contextBudget` reports `{ requestedCharacters, returnedCharacters, estimatedTokens, truncated }`. When `contents.injection` is `"scan"` or `"strip"`, each extracted item's `contents.injection` reports `{ risk, findings }` (plus `stripped`, the count of removed invisible Unicode characters when `"strip"` is used). Contents come from the same stored text `/v1/extract` returns, read after ranking, under the stored source's and publisher's current policy. No page is fetched, and nothing is summarized or generated. They belong to the one admitted search and consume no further quota. Pages are read one after another, so a search with contents takes longer than one without. Each result then carries a `contents` object; these values are illustrative: ```json { "status": "ok", "contentChangedAt": "2026-10-03T10:00:00.000Z", "totalCharacters": 5120, "text": "Jupiter has dozens of known moons. …", "characters": 4000, "truncated": true, "highlights": [ { "text": "The four largest moons are called the Galilean moons.", "start": 812, "end": 865 } ], "offsetUnit": "unicode_code_points" } ``` | Field | Meaning | |---|---| | `status` | `ok`; `page_not_indexed` when the page is unavailable under current stored source or publisher policy; `extraction_unavailable` when the read failed. Otherwise only `status` is present, and the result keeps its rank | | `contentChangedAt` | The stored text version that `text` and the offsets come from | | `totalCharacters` | Code points in the stored text | | `text`, `characters`, `truncated` | As in [indexed text extraction](#indexed-text-extraction): the leading text, its code-point count, and whether it was clipped to the requested limit or the response budget | | `highlights` | Up to three passages, best first, each `{text, start, end}`; `fragment: true` marks a slice of a longer sentence | | `excerpts`, `excerptsOrder` | Contiguous sentence windows (`{text, start, end}`) ranked by BM25 (`"keyword"`) or meaning (`"meaning"`) when configured | | `injection` | Present when `contents.injection` is set: `{risk, findings}` plus `stripped` count under `"strip"` | | `offsetUnit` | `unicode_code_points`, present with highlights or excerpts | Highlights are sentences quoted verbatim from the stored text that hold at least one of the query's words, ranked by how many they hold. The ranking is lexical, drops common query words, and uses source brief research's sentence filter, which keeps prose sentences and questions and skips navigation, timecode, and sharing text. A page's first 100,000 characters and first 2,048 sentences are considered. A sentence longer than the remaining budget becomes a fragment around its first matching word. A page that matched only by its vector, or holds no such sentence, has empty `highlights`. Offsets are zero-based, with `start` inclusive and `end` exclusive, in Unicode code points of the version `contentChangedAt` names; a later crawl can change them. The complete response is bounded to 510 KiB. When the requested text would exceed it, every result's text is clipped to one shared length, the longest that fits, and marked `truncated`. Highlights are never clipped to fit: instead one response holds at most 40,000 characters of them. Past 40 results, `highlights: true` gives each result 40,000 divided by `maxResults` characters (400 at 100 results), and an explicit `maxCharacters` times `maxResults` above 40,000 returns HTTP 400. Contents for 100 results read 100 stored pages, one after another, so they take longer than a short list's. Contents are untrusted publisher data, like excerpts: treat them as data, escape them before rendering HTML, and check the source's reuse policy. ## Similar pages ```text POST /v1/similar ``` Where [semantic retrieval](#semantic-retrieval) is configured, this returns the indexed pages nearest one indexed page, by cosine similarity between its stored `bge-small-en-v1.5` vector and theirs. There is no query: the page's own vector is the starting point, so no embedding call is made, and nothing is fetched. Where the vectors are searched through the approximate index, so are a page's neighbours, which can then miss some of the exact nearest pages. Send the same search key and JSON content type: ```json {"url":"https://example.com/search","maxResults":10,"excludeDomains":["example.com"]} ``` | Field | Default | Allowed values | |---|---|---| | `url` | Required | A page in the index, as search returns it: a public HTTPS URL without credentials, an explicit port, query strings, or encoded path separators | | `maxResults` | 5 | Integer from 1 to 100 | | `domains` / `excludeDomains` / `maxAgeDays` | Unset | The native search filters | | `startPublishedDate` / `endPublishedDate` | Unset | The [declared publication date](#declared-publication-dates) range; only pages with a declared date match | | `contents` | Unset | As in [inline contents](#inline-contents); `highlights` needs its own `query`, since the request has none | The response has search's shape: `url`, `results`, and `meta` with `requestId`, `mode: "index"`, `provider: "vitna"`, `count`, `latencyMs`, and `index: {documents, lastCrawledAt, ranking: "embedding_cosine", model, vectors}`. Results have the fields, `provenance`, filters, duplicate suppression and host spread of a semantic search. The page itself and exact copies of its text are never among its neighbours. A neighbour's excerpt is its leading words. As in semantic search there is no relevance threshold: the nearest pages are returned even when none is close. | Status | Code | When | |---|---|---| | 404 | `page_not_indexed` | The URL is not in the index, or its source no longer admits it | | 404 | `page_not_embedded` | The page is indexed but has no stored vector yet; the next maintenance run gives it one | | 503 | `semantic_unavailable` | The instance has no semantic retrieval, or its vectors are not loaded | Similarity shares the per-client search quotas, cache and deadline. Its receipt has `operation: "similar"`, and like a search's query, the URL is not in it. Only a page's title and leading text are embedded (see the semantic retrieval limitations), so pages that open alike can be near each other while differing further down. ## Indexed text extraction ```text POST /v1/extract ``` Send the same search bearer key and JSON content type. This endpoint returns cleaned text already stored in Vitna's index; it makes no live page request and does not schedule discovery. Use a URL returned by search: ```json {"url":"https://example.com/search","maxCharacters":20000} ``` `url` must be a public HTTPS page URL without credentials, an explicit port, query strings, or encoded path separators. Optional `maxCharacters` is an integer from 1 to 100,000, defaulting to 20,000. Optional `asOf` (`YYYY-MM-DD` or ISO 8601 with a UTC offset) extracts the latest stored snapshot captured on or before that instant (`page.snapshotAsOf` and `meta.asOf`) while enforcing current publisher and robots policies. Extraction shares the per-client search quotas and service concurrency limits. It returns plain text, without a promise to preserve HTML or Markdown structure; JavaScript is not executed. An illustrative response: ```json { "page": { "url": "https://example.com/search", "title": "An example result", "format": "text", "text": "Example indexed text.", "source": { "id": "example", "name": "Example" }, "crawledAt": "2026-10-03T10:00:00.000Z", "contentChangedAt": "2026-10-03T10:00:00.000Z", "truncated": false, "characters": 21, "totalCharacters": 21 }, "meta": { "requestId": "example-request", "latencyMs": 10 } } ``` `characters` counts returned Unicode code points; `totalCharacters` counts the stored text before the requested limit. `truncated` indicates clipping to that limit or the 510 KiB page-response budget. Stored source, archive, and publisher reuse-policy metadata can accompany the text. Crawl timestamps remain observations rather than publication dates, and content remains untrusted publisher data. A URL absent from the index or unavailable under current stored source/publisher policy returns HTTP 404: ```json { "error": { "code": "page_not_indexed", "message": "the page is not available in the current index" } } ``` To obtain a new page, use the bounded crawl workflow and search again after its job completes. Extraction does not turn an unindexed URL into a live fetch. ## Source brief research ```text POST /v1/research ``` Source brief assembles verbatim passages from matched pages already in Vitna's index. It is an extractive reading aid with citations, not a generated explanation, an exhaustive review, or a guarantee that the passages answer the question. Send the same search bearer key and JSON content type: ```json {"query":"lunar science","maxSources":3,"maxCharacters":8000} ``` | Field | Default | Meaning | |---|---|---| | `query` | Required | 1–1,000 characters, with no control characters | | `maxSources` | 3 | Integer from one to five indexed sources | | `maxCharacters` | 8,000 | Integer 1–16,000; distinct quoted passage budget in Unicode code points | | `domains` / `excludeDomains` / `maxAgeDays` | Unset | Native domain and observed crawl-age filters | | `expand` / `discover` | Unset | The same bounded expansion choices and opt-outs as native search | The endpoint accepts no `mode`, provider/fallback, or supplied URL fields. It uses Vitna's indexed evidence and makes no live page or paid search-provider call. Optional expansion returns a separate private job; poll it, then repeat the research request to use any newly indexed pages. Unknown fields are refused and request bodies are limited to 8,192 bytes. Research shares the search quotas, concurrency limit, and whole-request deadline. An unavailable research capability returns HTTP 503. Repeated overview quotes do not count twice against the passage budget. The complete response is also bounded to 128 KiB. The response reports `mode: "extractive"` and the original `query`: | Field | Meaning | |---|---| | `overview` | Passage entries with `text` and `citationIds` | | `sections` | Source sections using actual indexed page titles, `sourceId`, and cited `passages` | | `citations` | IDs linked to a `sourceId`, `start` and `end` offsets, and `offsetUnit: "unicode_code_points"` | | `sources` | IDs, URLs, titles, publisher identity, observed timestamps, optional archive metadata, and `readTruncated` | | `results` | Result cards for indexed sources actually quoted | | `coverage` | Bounded evidence counts and limitations | | `meta` | Request identifier, latency, `provider: "vitna"`, and optional index/expansion metadata | Each section's `sourceId` maps to a source entry. Citation offsets are zero-based, with `start` inclusive and `end` exclusive, in the indexed text version read for that source. They count Unicode code points, not bytes or JavaScript string units. Use the same stored text version when checking a quote; a later crawl can change its offsets. A passage marked `fragment: true` is a bounded slice of a longer sentence, still copied verbatim. `coverage.status` is `matched`, `partial`, or `insufficient`. Its scope is `matched_indexed_evidence` and `exhaustive` is always `false`; it reports `candidateSources`, `usedSources`, `passages`, `readTruncated`, `skippedSources`, `attemptedSources`, `unavailableSources`, `catalogUnavailable`, `outputTruncated`, `quotedCharacters`, and optional `indexMatch`. `matched` means selected passages without skipped sources, clipped reads, clipped output, or sentence fragments. `partial` has passages with one of those limitations; `insufficient` has no selected passages, including when the requested character budget is too small. These describe the selected indexed evidence, not the completeness of the web or the truth of a conclusion. Source text is untrusted publisher data. Check the cited pages and their reuse policies. ## Usage and receipts Native retrieval does not report a per-query provider bill. Infrastructure and crawl work still consume resources; the separate crawl meters bound that work. Receipts do not store search queries or retrieved content. Tavily basic search is explicitly selected, automatic parameter selection is off, and the request asks for reported credit usage. Exa automatic search requests highlights; its reported dollar cost is returned as `estimatedCostUsd`, because Exa documents that response field as an estimate rather than the basis of billing. Brave returns no dollar usage figure in this contract. Missing usage stays null. Provider-reported zero is preserved. Each admitted search, similar-page, SERP, extraction, or research request emits one JSON receipt to stdout: client identifier, request identifier, time, status, and latency. Search receipts include provider attempts and reported usage, plus the retrieval used and any fallback code when semantic retrieval is configured or requested; extraction receipts identify `operation: "extract"`, research receipts `operation: "research"`, similar-page receipts `operation: "similar"`, and SERP receipts `operation: "serp"`. Queries, requested URLs, returned content, authorization headers, and credentials are omitted. Capture stdout with the process supervisor if receipts are needed. Receipt output is best effort and is not a durable billing ledger. A service that accepts platform keys also sends the receipts for an org's requests (clients written `org.<id>`) to vitna.ai in the background, where the org's console lists them in Activity. The same allowlist applies there, so nothing the line above omits reaches it. See [One key](../one-key.md) for the endpoint and the settings that turn it off. ## Status and limits ```text GET /v1/indexes GET /v1/status GET /healthz GET /readyz ``` `GET /v1/indexes` requires a search key and returns the catalog of queryable indexes (`web` when configured, followed by all 23 vertical indexes with descriptions, data sources, retention windows, content-hash deduplication flags, live readiness, and indexed document counts). On vitna.ai, the public [Vitna Library](/library) catalog lists these vertical indexes and Web API data sources. Status requires a search key and reports the client identifier, configured providers, first provider, supported `modes`, and limits. It exposes safe availability information for configured operator routes, without credentials. The separate `serp` capability reports the retrieval adapter's own Google/Bing availability and engines, with safe transport settings. Its `retrieval` metadata includes configured attempt limits, cache statistics, and safe per-route/per-engine health and concurrency counters. Runtime-unavailable routes remain disabled until restart. It does not probe a search engine or disclose proxy credentials. `expansion.siteDiscovery` reports `enabled`, `automaticNames`, `automaticTopics` and `automaticConcepts` (true when configured), `limits: {origins: 3, candidates: 2000}`, and safe name-directory and publisher-catalog aggregate counts. `topicCatalog` identifies `source: "local_topic_descriptors"` and reports `publishers` (unexpired descriptors), with `observed` and `reviewed` counts. `candidates` counts separate unverified domain metadata; `imports` records dataset attribution and retrieval metadata. None of these are indexed document counts. Optional `conceptCatalog` reports `source: "wikipedia_titles"`, `language: "en"`, `titles`, `snapshot`, `importedAt`, `sourceUrl`, `license: {name, url}`, and a `changes` description of normalized lookup keys with original page titles retained. Its title count measures metadata, not indexed articles. Redirect titles have no canonical-target mapping in this catalog; article text becomes usable only after an allowed page is fetched and indexed. Health is public and reports only that the HTTP process is up; it makes no paid provider probe. Status also reports `extraction: {enabled, indexedOnly: true, limits: {defaultCharacters: 20000, maxCharacters: 100000}}` and `contents: {enabled, indexedOnly: true, limits}`, whose limits are `defaultTextCharacters`, `maxTextCharacters`, `defaultHighlightCharacters`, `maxHighlightCharacters`, `highlightsPerResult`, `highlightReadCharacters`, and `responseBytes`. `research` reports `enabled`, `mode: "extractive"`, and `limits: {maxSources: 5, maxCharacters: 16000}`. `similar` reports `{enabled, indexedOnly: true, limits: {maxResults: 100}}`, enabled where semantic retrieval is configured. Top-level `snapshotAsOf: true` indicates support for point-in-time `asOf` and `contents.snapshotAsOf` queries. Where semantic retrieval is configured, status adds `retrieval: {default: "hybrid", modes, semantic}`, whose `semantic` reports the `model`, the loaded `vectors` count, `dimensions`, the `bytes` one query worker holds, and a `state` of `ready`, `missing`, `empty`, `incompatible`, `over_limit`, or `unavailable`. When the vectors are searched through the approximate index, `semantic.approximate` adds `indexed` (vectors in the index), `exact` (vectors stored since its build), `lists`, `probes`, `rescore`, and `builtAt`. The index status counts `importedDocuments` beside `liveDocuments` and `archivedDocuments`, and a bulk source in `sources` reports its `contentOrigin`. The browser uses ordinary result search by default. Source brief is an explicit option when this capability is available; it shows only pages with usable quoted evidence and can return fewer cards than ordinary search. Readiness is also public and follows the default `/v1/search` route. When the native index is the default provider, readiness returns HTTP 503 while that index has no documents or its status cannot be read, whether or not SERP retrieval is enabled. A SERP-only instance reports its retrieval adapter's own availability, without probing a search engine. Authenticated status remains available with enabled SERP if an index status read fails, reporting `index: {ready: false, unavailable: true}`. Authenticated status includes local source counts, crawl freshness, and the latest crawl summary when the Vitna provider is configured. The root `/` serves a browser interface that keeps a pasted key only in that open page. | Environment variable | Default | Meaning | |---|---|---| | `VITNA_SEARCH_HOST` | `127.0.0.1` | Listening address | | `VITNA_SEARCH_PORT` | 8789 | Listening port | | `VITNA_SEARCH_PROVIDERS` | `vitna` | Unique ordered provider identifiers; external adapters require explicit selection and credentials | | `VITNA_SEARCH_INDEX` | `.search-data/web/index.sqlite` | Independent SQLite index; initialized if needed | | `VITNA_SEARCH_SERP_ENABLED` | `false` | Enables independent Google/Bing SERP retrieval without requiring a native index; also requires `VITNA_SERP_RESALE_APPROVED=true` | | `VITNA_SERP_RESALE_APPROVED` | `false` | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted | | `VITNA_SEARCH_SERP_TRANSPORT` | `auto` | Bounded direct/browser routes; proxies pin browser egress; explicit `direct` or `browser` also supported | | `VITNA_SEARCH_SERP_TIMEOUT_MS` | 10,000 | SERP retrieval timeout within the shared whole-request deadline | | `VITNA_SEARCH_SERP_PROXIES` | Unset | Private operator JSON array of browser proxy URLs | | `VITNA_SEARCH_MODE` | Unset | `web` selects the general beta's default index and frontier paths | | `VITNA_SEARCH_FRONTIER` | Unset outside web mode | Existing separate crawl-state file; adds authenticated crawl status | | `VITNA_SEARCH_CONCEPTS` | `.search-data/web/concepts.sqlite` in web mode; unset otherwise | Separate SQLite title-metadata catalog; optional native concept discovery | | `VITNA_SEARCH_DEMAND_SECRET_FILE` | Unset | Private file containing a 64-character lowercase hexadecimal key; enables encrypted site-discovery jobs in web mode | | `VITNA_SEARCH_QUOTA` | Unset | Separate writable SQLite quota file; persists per-client minute/day limits | | `VITNA_SEARCH_RPM` | 30 | Admitted search, SERP, extraction, and research requests per client per fixed minute | | `VITNA_SEARCH_DAILY_LIMIT` | 1,000 | Admitted search, SERP, extraction, and research requests per client per UTC day | | `VITNA_SEARCH_CONCURRENCY` | 4 | Simultaneous search, SERP, extraction, and research requests across the instance; a SERP search waiting for a busy route's slot counts, so keep it well above `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` | | `VITNA_SEARCH_TIMEOUT_MS` | 10,000 | One whole-request deadline, including sequential legacy fallback attempts | | `VITNA_SEARCH_EMBED_URL` | Unset | Embeddings endpoint ending in `/embeddings`, HTTPS or loopback HTTP; with the model, enables semantic retrieval | | `VITNA_SEARCH_EMBED_BATCH_URL` | Unset | Optional second endpoint, held to the same rules, that vector maintenance uses instead, so a query never waits behind a batch; queries always use `VITNA_SEARCH_EMBED_URL` | | `VITNA_SEARCH_EMBED_MODEL` | Unset | Model identifier sent to the engine; a different echoed model is refused | | `VITNA_SEARCH_EMBED_KEY` | Unset | Optional bearer token for a private remote endpoint | | `VITNA_SEARCH_EMBED_QUERY_PREFIX` | Empty | Prepended to each query, for example `Represent this sentence for searching relevant passages: ` for BGE models | | `VITNA_SEARCH_EMBED_DOCUMENT_PREFIX` | Empty | Prepended to each page's text and covered by its fingerprint | | `VITNA_SEARCH_EMBED_DIMENSIONS` | Unset | Expected vector width; responses of another width are refused | | `VITNA_SEARCH_EMBED_TIMEOUT_MS` | 2,000 | Query embedding deadline before the keyword fallback, 100–60,000 and at most half of `VITNA_SEARCH_TIMEOUT_MS` | | `VITNA_SEARCH_EMBED_BATCH_SIZE` | 4 | Pages per maintenance request, 1–256; an engine serves one request at a time, so without `VITNA_SEARCH_EMBED_BATCH_URL` a larger batch makes searches wait longer | | `VITNA_SEARCH_EMBED_BATCH_TIMEOUT_MS` | 60,000 | Deadline per maintenance request, 1,000–600,000 | | `VITNA_SEARCH_EMBED_MAX_CHARACTERS` | 1,500 | Characters of title and leading text embedded per page, 100–8,000 | | `VITNA_SEARCH_VECTORS` | `vectors.sqlite` beside the index | Separate vectors file, never the index, frontier, concept or quota file | | `VITNA_SEARCH_VECTORS_MAX_MB` | 512 | Bound on each query worker's copy of the vectors, 1–16,384; a larger file is searched through its approximate index, which with the vectors stored since its build must fit this bound | | `VITNA_SEARCH_VECTORS_PROBES` | 64 | Lists of the approximate index scanned per search, 1–4,096 | | `VITNA_SEARCH_VECTORS_RESCORE` | 400 | Approximate candidates rescored exactly per search, 10–4,096 | | `VITNA_SEARCH_EMBED_BULK` | `false` | `true` makes vector maintenance embed the pages of bulk imports too, after crawled pages | Search, SERP, extraction, and research requests that fail after admission consume a quota slot; this includes extraction 404 responses. A legacy upstream request may already have been paid for. Shared API `service_busy` refusals and invalid requests consume no quota. SERP route-busy and cooling responses occur after admission and consume one slot. Fallback attempts are part of one admitted search; at most four providers can be attempted, including at most three external providers. Without `VITNA_SEARCH_QUOTA`, quotas live in process memory and reset on restart. With it, per-client minute/day counters persist in SQLite and accounting failures refuse searches. The web deployment template sets a quota file. Run one alpha instance; these limits are not a distributed customer billing ledger. All responses use `Cache-Control: no-store`. Errors carry an `error` object with `code` and `message`. Completed attempt sequences also carry metadata. | Status | Meaning | |---|---| | 400 | Invalid JSON or unsupported request shape, including removed Live/Hybrid modes (`invalid_request`) | | 401 | Missing or invalid search credential | | 404 | Unknown service path, or an extraction URL unavailable in the current index | | 405 | Wrong HTTP method; `Allow` names the supported method | | 413 | Request body exceeds the byte limit | | 415 | Content type is not JSON | | 429 | Minute limit, day limit, or service concurrency limit; includes `Retry-After` | | 499 | Caller cancelled the request | | 502 | No attempted provider completed the search | | 503 | A provider, SERP, extraction, or research is unavailable, the required local index is empty, or quota accounting fails; also every SERP retrieval or parsing failure, a missed deadline included, told apart by its code | Upstream error bodies are never forwarded. Legacy search-provider redirects are refused, and their response bodies are capped at 1 MiB. SERP uses its own bounded, validated search-engine retrieval. Disconnecting a caller aborts the current request and prevents another fallback attempt. ## Compare retrieval outputs Compare speed and result counts within the same task and freshness policy. These endpoints return different evidence: | Route | What it returns | What a successful response establishes | |---|---|---| | Google SERP | Observed Google organic listings, engine snippets, page ranks, and resolved publisher links | A captured result page or bounded collection; snippets are not quotations fetched from the publisher pages | | Indexed search | Keyword-ranked matches from Vitna's stored corpus, or hybrid-ranked where semantic retrieval is configured, with crawl and source metadata | Matches in the configured index; neither a lexical match nor vector similarity is a finding supported by source evidence | | Native source brief | Verbatim passages from indexed pages with source references and Unicode citation offsets | Extractive indexed evidence with explicit coverage; it does not generate an answer or establish exhaustive research | An owned-route provider fee of zero excludes cloud hosting, browser CPU, network transfer, storage, crawling, and operating costs. Optional external search charges are separate. Compare result relevance, evidence quality, coverage, and freshness alongside latency; a larger link count is not the same deliverable as a smaller cited source brief. ## Provider protocol references The adapters follow the providers' own interfaces, checked on 2026-10-03: - [Brave Web Search](https://api-dashboard.search.brave.com/app/documentation/web-search/get-started), ranked web results and subscription-token authentication. - [Exa Search](https://exa.ai/docs/reference/search), highlights, request identifiers, and estimated cost metadata. - [Tavily Search](https://docs.tavily.com/documentation/api-reference/endpoint/search), bearer authentication, basic depth, and reported credit usage. ## Before a commercial launch The private beta serves configured clients and upstream accounts. Public signup, self-service key issuance and revocation, an invoice-grade usage ledger, customer billing, and public self-service deployment need a further implementation. The included search interface works on a configured instance. Search is a separate service from the existing routing subscriptions; no existing plan gains or is charged for search in this change. Check provider contracts before exposing their results commercially. Brave's [standard terms](https://api-dashboard.search.brave.com/documentation/resources/terms-of-service) prohibit resale/redistribution and replacing its service; ordinary API access does not authorize a public resale search service. Google and Bing results need the same decision: SERP retrieval starts only after the operator confirms, with `VITNA_SERP_RESALE_APPROVED=true`, that collecting and redistributing them is permitted. ## Operator reference The sections from here to the end are for the people who run a Search instance, which during the beta means Vitna's own operators, working from a checkout of Vitna's private repository. None of it is needed to call the hosted API at `https://search.vitna.ai`, and none of it is an offer to run a deployment of your own. Try the complete offline demo with `node platform/search/demo.mjs`, then open `http://127.0.0.1:8789`. Its four pages are clearly labeled illustrative fixtures; the demo makes no external requests. See [Run a focused search index](self-hosted.md) for live crawling, OVHcloud preparation, and R2 snapshots. The [general web beta](general-web.md) adds persistent cross-site discovery, day/month crawl budgets, file-backed API quotas, and the Virginia deployment package. Since 2026-10-07 a browser that opens the hosted instance, or its `/docs`, is sent to Search's page and this reference on vitna.ai, where the product's pages live: the operator sets `VITNA_SEARCH_PAGE_URL` and `VITNA_SEARCH_DOCS_URL`, and a self-hosted instance with neither set serves its own page and API reference. ## Starting an alpha instance Node.js 22.13 or later is required; Node 24 is recommended for the deployment package. Direct SERP retrieval and indexed search use built-in Node modules and need no npm dependencies or website build. The optional browser SERP transport, including browser attempts selected by `auto`, requires the separate Unblocker runtime. Indexed search uses the currently experimental `node:sqlite`. Startup opens `.search-data/web/index.sqlite` by default and initializes an empty local index when needed. Populate selected sources before serving useful native searches. SERP retrieval does not require populated indexed pages or crawler work. Mint a credential for each alpha client: ```bash npm run search:key -- pilot ``` The command prints a `vitna_search_` token once, followed by a JSON entry mapping the client identifier to its SHA-256 hash. Give the token to the client privately. Merge the hash entry into `VITNA_SEARCH_CLIENTS`, a JSON object. The service accepts these search keys and an org's platform key (`vitna_key_`, see [one key](../one-key.md)); push and router keys grant no search access. Remove a client's entry and restart to revoke its access. For native retrieval, set `VITNA_SEARCH_CLIENTS`. The default provider is `vitna`, with an index at `.search-data/web/index.sqlite`; override its path with `VITNA_SEARCH_INDEX`. Crawl selected sources before searching their content. See the [index setup](self-hosted.md). An operator can still select legacy external adapters explicitly, but native startup has no external provider fallback. For actual Google/Bing results, set `VITNA_SEARCH_SERP_ENABLED=true` and `VITNA_SERP_RESALE_APPROVED=true`, and follow the [SERP setup](serp.md#run-without-an-index). The second setting records the operator's decision that collecting those results and redistributing them through Vitna's API is permitted; startup refuses the first without it. The same client key authenticates both capabilities; neither requires a Bright Data key or zone. Start the API with: ```bash npm run search:start ``` For a local environment file, keep a private `.env.search` in the checkout, using the settings described in `deploy/vitna-search.env.example`. Start it with Node's environment-file loader: ```bash node --env-file=.env.search platform/search/start.mjs ``` By default the server binds to `127.0.0.1:8789`. Boot refuses an empty client registry, invalid limits, duplicate credential hashes, no enabled retrieval capability, or SERP retrieval enabled without its redistribution confirmation. The local index is initialized if needed; an empty index can start but native retrieval, and with it readiness, remains not-ready until a crawl succeeds, even with SERP retrieval enabled. A SERP-only instance can be ready without an index. Starting the process makes no upstream search request and does not run a crawl. ## Maintaining the vectors file Where [semantic retrieval](#semantic-retrieval) is configured, maintain the vectors file with: ```bash npm run search:embed ``` The job reads the index read-only, embeds new pages and pages whose embedded text changed, deletes vectors of pages that left the index, stops at `--limit` pages (default 25,000) or `--budget-ms` (default 480,000), and prints one JSON summary line. A vectors file holding another width or model stops the run with `vectors_dimensions_mismatch` or `vectors_model_mismatch`; any other error in the vectors file or the index exits 1 with the error on stderr and no summary. Every batch commits on its own, so an interrupted run can simply run again; a lease in the vectors file keeps two runs from overlapping. A text the engine refuses is retried with its character limit halved, and halved again while the engine keeps refusing (750, 375, then 187 characters for the default 1,500), stopping before the limit falls under 100 or once halving no longer shortens the text. It is then recorded and skipped until the page changes. The [general web deployment](general-web.md#semantic-retrieval-deployment) packages it as a service and timer. The job compares pages and vectors in document-id ranges of 100,000, so its memory stays bounded however large the index grows. Pages of a bulk import (an imported Wikipedia dump) are embedded only with `VITNA_SEARCH_EMBED_BULK=true`, and then after every crawled page that needs a vector; without it they are passed over and any vectors they already have are kept. The summary's `documents` counts the pages the job considered. Once the vectors no longer fit `VITNA_SEARCH_VECTORS_MAX_MB`, build the approximate index from the vectors file, with the same embedding settings as the API: ```bash npm run search:ann ``` It reads the vectors file only, writes `vectors.ivf` beside it under a temporary name and renames it into place, and prints one JSON line with the vector count, lists, file size and duration. Query workers load the new index on their own. Vectors stored after a build are searched exactly beside the index, so rebuild when they no longer fit beside it, or when a large share of the file has changed. --- <!-- https://vitna.ai/docs/search/general-web.md --> # Run the general web beta Vitna Search can grow its own index across public websites, keep a durable crawl queue, and serve authenticated searches without a paid search provider. This is a bounded beta with limited coverage. The API is deployed at `https://search.vitna.ai` on a reused Virginia OVH server, with private R2 backups. No additional OVH instance was purchased. Index is the browser's default search mode. Searches return the current corpus immediately. Empty name/topic queries and topic queries with only partial-term matches can automatically queue discovery of a few candidate sites, then crawl selected pages into the same index. Domain input is optional. This uses Vitna's own crawler and makes no paid search-engine call. It does not promise whole-web coverage or immediate access to pages that have not been indexed. Source brief can assemble cited, verbatim passages from the matched indexed pages; it does not generate an explanation. ## Current evidence Three local live runs through 2026-10-03 used 460 HTTP requests and recorded about 31.3 MB of response payload. They indexed 65 real pages, registered 250 origins, and left 27,066 URLs pending. Registered origins and queued URLs are discoveries, not searchable coverage. Some publishers refused requests, timed out, or had unavailable robots files; those failures remain recorded for bounded retries. A staging import of all of English Wikipedia (6,030,988 articles, snapshot 20261001) answered 67.7% of NQ-open's questions in keyword retrieval's top ten, against 16.1% on the production index; it is not deployed. See [Wikipedia at full scale](#wikipedia-at-full-scale). A separate synthetic test inserted 100,000 documents in 16.2 seconds and measured 79 ms median / 91 ms 95th-percentile retrieval on this PC over 100 queries. Its database was about 454 MB. These measurements concern the keyword index, not web relevance, real page sizes, or OVH performance. A later local HTTP test against 100,000 synthetic pages completed 100/100 requests with four simultaneous callers, two query workers, and caching off. Median search latency was 314 ms; the 95th percentile was 1,382 ms. Public health checks had a 1.1 ms 95th percentile. This is a short local smoke test, not sustained throughput or a production capacity guarantee. ```bash npm run search:benchmark -- --documents 100000 npm run search:benchmark -- --documents 100000 --http --concurrency 4 ``` `--vectors 384` adds one synthetic 384-dimension vector per page. On 2026-10-06, `--documents 49063 --vectors 384 --http --concurrency 2` on a desktop Ryzen 7 3700X measured a query worker's full load of the 49,063 vectors at 434 ms and 79.5 MB, the top-40 scan at 24.7 ms median and 25.4 ms at the 95th percentile, and 500 changed vectors applied in 5.3 ms. Over HTTP with two workers, hybrid searches took 119 ms median against 72 ms for keyword, with 95th percentiles of 650 and 564 ms. The benchmark makes its query vectors in process, so those hybrid times exclude the engine call, about 10 ms on an idle engine. ## Crawl and preview locally Node 24 is recommended. The service uses built-in modules and needs neither an npm install nor a website build. Leave at least 10 GiB free on the data disk. Initialize local databases and a private pilot credential with: ```bash npm run search:setup ``` This creates `.env.search` and `.search-data/private/pilot.key` without printing the token. It preserves an existing environment file and key. Both are excluded from Git and release archives. The new environment selects the Vitna index and loopback port 8791. Start the authenticated local API with: ```bash node --env-file=.env.search platform/search/start.mjs ``` There is no crawl or cloud deployment during setup. An empty index stays not-ready until a successful crawl. Windows does not enforce POSIX file modes; store these private files in an account-restricted workspace. For the browser preview with its own temporary credential, use: ```bash npm run search:web -- --max-requests 80 npm run search:web:preview ``` The first command makes real public HTTP requests. Its override can only lower the configured run limit. The second opens a loopback-only service at `http://127.0.0.1:8790`, with a temporary key usable only by that preview. It searches the real local index and performs no crawl. Its quota database is in memory; production uses the file specified below. Live runs can exit with a failure status while preserving successfully indexed pages. Read the summary's `failed`, `errors`, and `stopReason`. Requests stopped by a run limit resume from the queue on the next invocation. The preview never creates provider calls or reads the site's database. ## Crawl scope and limits The starter `platform/search/web-seeds.json` contains 28 public sites spanning science, health, books, parks, government, art, economics, and software. Links can add other public HTTPS origins within the configured cap. Coverage and permission are evaluated as each origin is visited; a seed is not a promise that its site will be indexed. Pages in languages other than English can enter through discovered links. `platform/search/web-profile.json` sets these defaults: | Control | Default | |---|---:| | Total searchable documents | 100,000 | | Documents per origin | 2,000 | | Documents per bulk-imported source (a Wikipedia dump) | 8,000,000 | | Registered origins | 1,000 | | Queue entries | 250,000 | | Concurrent origin jobs | 4 | | HTTP requests per run | 1,000 | | Jobs per run / maximum duration | 4,000 / 15 minutes | | Requests per UTC day / month | 25,000 / 600,000 | | Metered payload per UTC day / month | 4 GiB / 120 GiB | | Daily/monthly request and payload capacity reserved for demand work | 20% | | Raw and decompressed page size | 1 MiB each | | Raw and decompressed discovery file size | 2 MiB each | | Sitemap files per origin / nesting depth | 6 / 2 | | Advertised feeds per page / feed page refresh | 2 / 24 hours | | Minimum origin spacing | 1 second, increased for robots rules | | Regular page refresh / robots cache | 7 days / 24 hours | | Discovered link depth | 4 | | Failed attempts before manual retry | 5 | | Minimum extracted words | 40 | | Minimum free disk space | 10 GiB | The queue, origin delays, robots cache, retry state, and day/month meters are stored in SQLite. Every HTTP request, including robots, discovery, and redirects, reserves budget before starting. Successful reads refund unused payload reservation; crashes leave the full reservation charged. These meters bound crawl work and payload, not transport headers, compute charges, or an account-wide cloud bill. Changing the profile does not prune existing queue entries or documents to a smaller cap. Validate a smaller profile against existing counts before activating it. The first two caps count crawled pages only (Common Crawl imports included). A [bulk import](#import-english-wikipedia) fills its own source, which has its own cap, `maxDocumentsPerBulkSource`, so millions of imported articles neither stop the crawler at its 100,000 nor count against `en.wikipedia.org`'s crawl allocation, and the crawler's caps are no looser than before. The crawler's source sync never trims or removes a bulk source. Background work stops when shared daily or monthly usage reaches 80% of its request or payload limit, reserving the remainder for user-requested jobs. Both lanes charge the same durable totals; the reserve does not create an additional budget or guarantee immediate crawl capacity. At a full frontier, background discovery updates known entries without registering new sites that cannot receive queue space. Each run may reclaim up to 64 automatically registered origins that have never been visited and have no queued URL, robots cache, lease, delay, or indexed document. Configured seeds and current demand targets remain protected. Newly reviewed starting sources also receive bounded queue admission when the frontier is full. Each pass admits at most eight missing seed pages across sources, replacing only untouched discoveries; indexed pages, crawl history, active leases and configured seeds stay protected. The catalog now includes 28 starting sources spanning science, government, health, general reference, travel, programming, reporting and textbooks. These are starting points for the crawl, rather than a claim of complete coverage on those sites. ## Query-guided coverage expansion In configured web mode, an empty native name/topic search or a topic query with only partial-term matches can queue site discovery automatically when no domains or URLs are supplied and external fallback is disabled. A plain request works without domain input or crawl flags: ```json {"query":"examplebrand","mode":"index"} ``` The browser enables this option by default. Untouched requests omit the `expand` and `discover` flags; disabling the option sends both as `false`. Either explicit `expand: false` or `discover: false` prevents automatic discovery. Automatic admissions include `meta.expansion.automatic: true`. Production `meta.index.match` can report `complete`, `partial`, or `none`; `partial` describes partial-term fallback. Discovery can address that weak coverage while leaving the current results intact. The field is optional, and complete term matches do not guarantee that a result answers the query. Name detection is heuristic. Common topic words and multiword topic queries do not generate guessed hostnames. Broad queries instead use publisher description matches from reviewed or observed publisher descriptors. A catalog domain is a discovery candidate, not an indexed document or a promise of access. A query without a suitable directory/name, publisher, or configured concept-title match can return `no_candidates`. Automatic name lookup uses Vitna's own directory of names observed on fetched pages; only explicit expansion also matches known URLs or caller-supplied sites. A narrow distinctive single ASCII brand can additionally suggest up to three hosts formed with `.com`, `.ai`, and `.org`. These are unverified candidates, not a claim that any of them is the official site or matches the intended entity. Topic discovery matches query words against publisher descriptions, then examines at most three selected publishers' homepages and sitemaps before choosing up to eight target pages. It shares the same request, time, policy, and ownership limits as name discovery. The catalog does not contribute search results until the crawler successfully indexes pages. When configured, a separate local English Wikipedia title catalog can match concepts to candidate article URLs. It contains title metadata, including redirect titles, rather than article bodies or canonical redirect mappings. Each candidate must still pass crawler policy and be fetched and indexed before its text is searchable or available to Source brief. Matching a title does not establish current page availability or whole-web coverage. Automatic name lookup uses exact directory names and bounded candidates; incidental matches in article addresses do not establish a site identity. Before a candidate enters the index, its HTTPS page must respect robots and `noindex`, contain useful readable text, and identify the requested name. Parking pages, challenges and unrelated redirects are rejected. A successful check establishes a self-identified content match, rather than verified ownership. The directory learns public names from indexed homepages, retains observations for 180 days, and stores up to 10,000 sites with eight aliases each. Failed probes are retained as private HMAC digests for at most 24 hours, preventing repeated failed requests without saving guessed names. Both tables live in the frontier database and are covered by the existing backup. To learn from homepages already in an initialized index, run this optional local command with the same index and frontier environment settings: ```sh npm run search:sites -- --backfill ``` It inspects at most 1,000 live homepages, respects enabled source scope and cached publisher rules, makes no network requests, and prints aggregate counts. Callers can also request work explicitly. Current indexed results return first with a separate `meta.expansion` job. Send `expand: true, discover: false` for target-only refresh. The service selects up to eight relevant known URLs using words in their addresses, plus supplied `urls`. `domains` can supply domain roots when no relevant known URL is available. This job's scope is `known_urls_and_supplied_targets`. Add `discover: true` to examine selected sites' homepages and sitemaps before choosing target pages. It requires `expand: true` and a configured private discovery key. Optional `domains` guides site selection; without it, sites come from matching known URLs, supplied URL origins, observed names, narrow brand-name candidates, topic-catalog matches, or configured concept-title matches. Automatic discovery skips incidental known-URL matching. With discovery, callers can provide at most three `domains` or three distinct supplied URL origins; ordinary native domain filters retain their ten-host limit. Site-discovery jobs have scope `site_discovery`. Each examines at most three origins, including robots files, homepages, and up to two sitemap files per site, and considers at most 2,000 sitemap candidates. It selects at most eight target pages within 40 total HTTP requests and 60 seconds of crawler work. Discovery requests and target fetches share those budgets. URL matches can be weak, publishers can refuse crawling, and selected candidates may never be indexed. This is bounded site discovery, not whole-web search. Jobs are durable in the frontier database, private to the authenticated client, and expire after 24 hours. Admission allows 64 pending jobs in total and four per client. Repeating the same normalized query, filters, discovery choice, and supplied targets returns the same job until expiry. Deduplication uses a private HMAC digest. Bounded lexical and site hints, including names, candidate origins and their provenance, are encrypted in private expiring discovery jobs. The raw original query and full request are not stored or logged. Plaintext terms are not persisted in jobs or receipts. Target URLs remain in the job and normal crawl state. `GET /v1/search/jobs/<id>` returns progress with the same bearer key. A completed job means its bounded pass finished; inspect `summary.indexed`, `summary.failed`, and `summary.demand.unmet` for actual coverage. `summary.discovery` adds integer counters `origins`, `candidates`, `selected`, and `unavailable`; provisional name jobs also report `probed`, `accepted`, and `cached`. Candidate and selection counts do not mean searchable pages; indexed/unchanged counts and unmet targets describe the crawl outcome. Search again to retrieve newly indexed pages; the browser's completed-job rerun sets `expand: false` to avoid repeating the work. Polling allows at most 60 checks per minute and does not consume the ordinary search allowance. An unknown, expired, or other client's job returns 404. Each job uses at most eight target attempts, 40 HTTP requests, and 60 seconds of crawl work. It follows the crawler's existing public-address validation, robots, noindex, origin spacing, free-disk guard, and shared daily/monthly budgets. It does not recursively crawl links or feeds. Ordinary target-only jobs do not fetch sitemaps; site-discovery jobs add only the bounded pass above. The same global index lease protects background and demand work. Never-fetched pending queue records may be replaced to admit explicit targets at the frontier cap; fetched history and active leases are preserved. The background runner releases its lease between 100-request or one-minute batches and checks demand jobs between batches. The demand timer checks every 30 seconds while the background pass is inactive. A busy crawler, publisher delay, or exhausted budget can postpone work; queued jobs report retry timing. ```bash curl https://search.vitna.ai/v1/search \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"query":"lunar science","mode":"index","expand":true,"discover":true,"domains":["nasa.gov"]}' ``` Set `expand: true, discover: false` to crawl only known and supplied targets. For name discovery without domains, send `expand: true, discover: true`. Supplied `urls` remain public HTTPS page addresses without credentials or query strings and must satisfy domain restrictions. Run one bounded demand pass manually: ```bash npm run search:demand ``` The installer includes `vitna-search-demand.service` and `.timer`; enable the timer alongside the API and background crawl units. Jobs are included in existing frontier backups. This deployment uses Vitna's own crawler and index. ## Initialize the topic catalog Run these explicit local commands with the existing `VITNA_SEARCH_FRONTIER` and `VITNA_SEARCH_INDEX` settings loaded into the process environment: ```bash npm run search:topics -- --init npm run search:topics -- --review-seeds npm run search:topics -- --backfill ``` Without those settings, the defaults are `.search-data/web/frontier.sqlite` and `.search-data/web/index.sqlite`. Initialization creates the frontier/catalog schema. Seed review records 28 curated publisher descriptions. Backfill reads at most 1,000 eligible already indexed public homepages, respecting current source scope and cached publisher rules. These commands make no network requests and neither register crawl origins nor start crawling. Startup does not automatically seed or download the catalog. An optional explicit command downloads a bounded domain-candidate dataset: ```bash npm run search:topics -- --import-majestic --limit 10000 ``` It uses the fixed [Majestic Million CSV](https://downloads.majestic.com/majestic_million.csv), retaining 10,000 candidates by default and at most 100,000. Reads are bounded to 100 MiB and 60 seconds. A durable 24-hour cooldown includes failed download attempts; no force flag or alternate URL is accepted. Failed refreshes preserve the previous candidate snapshot. Stored attribution identifies [Majestic Million](https://majestic.com/reports/majestic-million) and its [CC BY 3.0 Unported license](https://creativecommons.org/licenses/by/3.0/), retrieval information, and normalization to a bounded subset of HTTPS roots. That dataset license does not establish permission to reuse publishers' pages or assets. Imported candidates remain separate unverified metadata. Importing them does not create publisher descriptions, register origins, enqueue a crawl, or add searchable documents. Topic discovery matches reviewed or observed publisher descriptions; the candidate count does not measure its indexed coverage. ## Import a concept-title snapshot `VITNA_SEARCH_CONCEPTS` selects a separate SQLite catalog. Web mode defaults to `.search-data/web/concepts.sqlite`; the API and demand worker must use the same path, distinct from index, frontier, and quota files. Startup performs no title download. Import a chosen published English Wikipedia snapshot explicitly: ```bash npm run search:concepts -- --import-wikipedia --snapshot 20261003 ``` The date is an example published snapshot. Select a valid `YYYYMMDD` snapshot from the [official title dumps](https://dumps.wikimedia.org/other/pagetitles/). The importer constructs that fixed Wikimedia source URL; it accepts no alternate download URL or prefix-sized title subset. It reads the complete gzip stream, validates title records and database integrity, and publishes a staged replacement only after successful completion. A failed import preserves the existing catalog. | Import bound | Limit | |---|---:| | Compressed / decoded payload | 160 MiB / 2 GiB | | Title records / database size | 25 million / 8 GiB | | Minimum available disk space | 12 GiB | | Import deadline | 15 minutes | Original English main-namespace titles and redirect titles are retained with separate normalized lookup keys. There are no redirect-target mappings or article bodies. Status records the snapshot, import time, source URL, payload counts, and contributor/license attribution; consult that attribution and the publisher's policy when reusing material. This catalog cannot supply search results or Source brief passages by itself. Candidate pages must first be fetched and indexed within the crawler's policy and budgets. The R2 backup described below includes the title catalog once it holds titles, when `VITNA_SEARCH_CONCEPTS` is set in `web-backup.env`. Without that setting, preserve it separately or repeat its recorded snapshot import when restoring concept discovery. ## Private discovery jobs Configure `VITNA_SEARCH_DEMAND_SECRET_FILE` in the shared `/etc/vitna-search/web.env`, for example: ```dotenv VITNA_SEARCH_DEMAND_SECRET_FILE=/etc/vitna-search/demand.key ``` Create that private file with a cryptographically random 32-byte key encoded as 64 lowercase hexadecimal characters, with an optional final newline. Use a regular file, not a symbolic link. On Linux, give it to `vitna-search` with mode 0600; group or public permissions are refused. Use private account ACLs on Windows. Keep the key out of chat, source control, release archives, and receipts. The API, background web crawler, and demand unit must read the same file through their shared environment. The key is needed only for new site discovery; ordinary retrieval and target-only expansion remain available without it. Authenticated `GET /v1/status` exposes `expansion.siteDiscovery` with `enabled`, `automaticNames`, `automaticTopics` and `automaticConcepts` (true when configured), `limits: {origins: 3, candidates: 2000}`, and safe name-directory and publisher-catalog aggregate counts. `topicCatalog` reports `source: "local_topic_descriptors"`, `publishers` (unexpired descriptors), `observed`, `reviewed`, `candidates` (separate unverified domain metadata), and `imports` (dataset attribution/retrieval records). These do not count searchable documents. `conceptCatalog` identifies `source: "wikipedia_titles"`, `language: "en"`, title count, snapshot/retrieval information, source URL, license attribution, and normalization changes. Its title count is also metadata, not indexed pages. Status also reports `extraction` availability, `indexedOnly: true`, and its default/maximum text limits, plus `research` availability and extractive limits. Check `enabled` before submitting `discover: true`. The key's contents and decrypted query terms are never returned through public job status. Unavailable discovery is reported in `meta.expansion`; current indexed results remain available. A job whose encrypted context cannot be opened fails with `discovery_context_unavailable`, rather than silently changing into a different crawl. ## Crawler policies The crawler checks public DNS addresses and pins the checked address into the TLS connection. It refuses IP literals, private addresses, credentials, custom ports, query strings, login/cart paths, and download links. It identifies as `VitnaSearch`, honors robots allow/disallow rules, `Content-Signal: search=no`, noindex, and nofollow. Newly forbidden indexed pages are removed when the publisher's robots policy is refreshed. Other-origin robots redirects fail closed. HTTP 429/503 Retry-After pauses that origin; exhausted jobs require: ```bash npm run search:web -- --retry-failed ``` Publisher sitemaps, RSS/Atom feeds, and `llms.txt` add scoped same-origin page URLs. Nested sitemap indexes and gzip files share the origin's six-file budget; cycles are skipped and nesting stops at two levels. XML entity declarations are rejected. HTML-advertised feeds are fetched only when following links is allowed. Feed pages, seed pages, and imported archive URLs receive a 24-hour refresh interval; the queue and crawl budget can delay that further. Optional feed failures cannot discard an already fetched page. Discovery URLs are enqueued in batches. The crawler extracts HTML, plain text, and Markdown. JavaScript rendering, PDFs, model reranking, spam scoring, image/news search, and generated answers are outside this beta. Semantic embeddings are an optional [deployment](#semantic-retrieval-deployment) on Vitna's own engine. Weekly refresh is a minimum interval, not a freshness guarantee: queue size, failures, and caps can delay it further. ## API and production accounting Run the API with `VITNA_SEARCH_MODE=web`, `VITNA_SEARCH_PROVIDERS=vitna`, and dedicated client hashes in `VITNA_SEARCH_CLIENTS`. `search:key` creates a token and its hash; deliver the token privately and place only the hash in the service's private environment. See the [API contract](overview.md). Use three different files, never the site's database: ```dotenv VITNA_SEARCH_INDEX=/var/lib/vitna-search/web/index.sqlite VITNA_SEARCH_FRONTIER=/var/lib/vitna-search/web/frontier.sqlite VITNA_SEARCH_QUOTA=/var/lib/vitna-search/web/quotas.sqlite ``` Setting `VITNA_SEARCH_QUOTA` enables persistent per-client minute/day limits across API restarts. Failed accounting refuses the search. Authenticated `GET /v1/status` reports index coverage, queue/usage summaries, and `quotaStorage`. Receipts omit queries and credentials and remain best-effort logs, not an invoice ledger. Signup, customer billing, and self-service keys still need implementation before a commercial public launch. For a URL returned by search, `POST /v1/extract` returns the cleaned text already stored in the index, with source metadata and observed crawl times. It uses the same bearer key and search quotas. Optional `maxCharacters` defaults to 20,000 and is capped at 100,000. It performs no live fetch or JavaScript execution and does not promise Markdown structure. An absent or currently policy-unavailable page returns `page_not_indexed` with HTTP 404. See [indexed text extraction](overview.md#indexed-text-extraction) for the request and response contract. `POST /v1/research` provides Source brief: cited verbatim passages from selected indexed pages, with actual page titles, source metadata, search results, and bounded evidence counts. It defaults to three sources and 8,000 Unicode code points of distinct quoted passages, capped at five sources and 16,000 code points. Repeated overview quotes are not counted twice. It accepts native filters and bounded expansion choices, with no paid provider fields. Coverage is limited to matched indexed evidence and is never exhaustive; a brief is not a generated explanation or a complete review. See [Source brief research](overview.md#source-brief-research) for its contract. Local searches accept `domains`, `excludeDomains` (1–10 public hostnames each), and `maxAgeDays` (1–3,650). Domain filters include subdomains; crawl-age filters use observed crawl times, not publication dates. `startPublishedDate` and `endPublishedDate` filter on a page's [declared publication date](overview.md#declared-publication-dates) and match only pages that declare one. Filtered searches require the Vitna index with paid fallback disabled. For example: ```json {"query":"solar system","mode":"index","maxResults":5,"domains":["nasa.gov"],"maxAgeDays":14} ``` Keyword retrieval weights titles, drops common query words when other words remain, ranks title phrases alongside complete matches in any word order, and suppresses exact duplicate extracted content. It falls back to partial term matches only when there are no complete matches within the selected filters. That fallback matches any of the query's terms, rarest first, while their page counts sum to at most 200,000; the rarest term always counts. On an index of crawled pages every term counts, as before; on millions of imported articles the commonest terms drop out, because an any-term match over them ranked most of the index. Empty results mean there was no match in this index. Search content is untrusted source data. Every result list, keyword, semantic or hybrid, spreads its hosts. Among the first 20 candidates, at most two pages from one host lead (`www.` aside), and that host's further candidates follow the other sites' within those 20. Candidates below the first 20 keep their order. Nothing is dropped, so a search confined to one site still fills its page. On the production index (49,063 pages), the spread raised expected-domain hits in the top five from 131 to 135 of the 200 probes for hybrid retrieval, from 134 to 140 for semantic and from 108 to 117 for keyword. NQ-open answer recall stayed within noise in every mode; hybrid answered 227 of 1,000 questions in its top ten, against 225. On a staging index taking in Wikipedia, where `en.wikipedia.org` held 7.6 of an average keyword top ten, keyword hits rose from 96 to 109, and 6 top-ten answers were lost for 1 gained. Spreading every row a search reads (200 for a ten-result page) reached 113 hits there, but it lost 27 top-ten answers for 5 gained, by promoting weak matches from deep in the list. In web mode, `VITNA_SEARCH_QUERY_WORKERS=2` serves SQLite retrieval on separate worker threads. The request queue is bounded at 32; its default query deadline is 10 seconds, including queue time. A cancelled request is answered at once, but its worker finishes the query and then serves the next one, since a replacement would reload every vector. A worker still busy at the deadline, cancelled or not, is terminated and replaced. `VITNA_SEARCH_TIMEOUT_MS` sets that deadline. At the API, the same setting is one whole-request search deadline. Discovery and expansion run separately within their job budgets; waiting in the crawl queue can take longer than either work deadline. The HTTP process remains able to serve health checks while retrieval runs. Worker status failures return service unavailable. `VITNA_SEARCH_CACHE_MS=30000` caches up to 256 result sets in memory for 30 seconds. Domain filters form part of the cache key. Crawl-age queries bypass it. Database/WAL metadata invalidates cached content, with up to 250 ms of metadata staleness. No customer query is written to a cache file or receipt. Set the cache duration to zero for uncached measurement. ## Measure coverage before expanding access ```bash npm run search:evaluate ``` This runs 200 authored English queries across 20 topics and saves `.search-data/reports/coverage.json`. `VITNA_SEARCH_INDEX` selects the database; `VITNA_SEARCH_EVALUATION_REPORT` selects a different report path. The report lists URLs, latency, nonempty results, and expected-domain hits in the top five. Expected domains are coverage probes, not human relevance judgments. Partial keyword matches can produce a result for an unrelated intent, so a high nonempty rate alone says little about usefulness. Manually judge the returned URLs and snippets before opening customer access. The first 24-page baseline returned a result for 182/200 queries, but only 69/200 had an expected-domain hit. This small corpus is inadequate for a general search launch. The later 65-page baseline returned 193 nonempty results and 77 expected-domain hits. Neither result establishes relevance. The first expansion milestone is 100,000 real documents with balanced coverage across these topics, measured alongside failed queries, freshness, and an HTTP test on the actual OVH instance. The profile's document cap is a target limit, not indexed coverage. With [semantic retrieval](overview.md#semantic-retrieval) configured, compare the modes side by side: ```bash npm run search:evaluate -- --retrieval keyword,semantic,hybrid npm run search:evaluate -- --fixtures ``` Compare full-workflow accuracy, citation validity, information density, latency, failure rate, and workflow cost across Vitna and external providers under shared result and context budgets: ```bash npm run search:provider-benchmark -- --offline-demo ``` Pass `--providers vitna,tavily,exa,parallel,perplexity,brave` with API keys in the environment (or `VITNA_SEARCH_URL` and `VITNA_SEARCH_KEY`) to measure live providers against `platform/search/provider-benchmark-fixture.json`. The first runs the same 200 probes once per mode through the query pool the API uses, against the index and its vectors, and saves `.search-data/reports/retrieval-comparison.json`. It reports nonempty results, expected-domain hits at one and five, expected-domain reciprocal rank, latency, and every query that fell back to keyword retrieval; any fallback fails the run. The second needs no index: it embeds the three authored judged corpora into temporary vectors and reports nDCG@5, recall, reciprocal rank and negative-query accuracy per mode, in `.search-data/reports/retrieval-fixtures.json`. On 2026-10-06 these ran against a local 816-page index from a focused, robots-respecting crawl of 21 sites, 15 of them evaluation publishers and 6 distractors (Wikipedia, Our World in Data and others). NOAA, the Smithsonian, the Met, the Education and Transportation departments and the USDA refused or failed the crawl, so the weather and art topics had no expected-domain page. Pages were embedded with `bge-small-en-v1.5` on the vitna-anchor engine (three threads, one page of 816 retried shorter) and queries carried the BGE query prefix: | Mode | Hits at 5 | Hits at 1 | Reciprocal rank | Expected-domain results | Median / 95th percentile | |---|---:|---:|---:|---:|---:| | `keyword` | 135 | 108 | 0.598 | 440 | 3.6 / 14.3 ms | | `semantic` | 153 | 112 | 0.636 | 549 | 12.1 / 15.4 ms | | `hybrid` | 155 | 122 | 0.669 | 537 | 15.0 / 25.8 ms | Latency is per search through the query pool, query embedding included, at 816 pages. All 200 queries returned results in every mode and none fell back. Without the query prefix, `semantic` scored 155, 112 and 0.641 and `hybrid` 157, 121 and 0.676. The authored judged suites gave nDCG@5 / negative-query accuracy: | Suite | `keyword` | `semantic` | `hybrid` | |---|---:|---:|---:| | Challenge suite, 33 pages, 24 queries | 0.796 / 0.33 | 0.904 / 0 | 0.918 / 0 | | Held-out relevance, 30 pages, 30 queries | 0.779 / 0.08 | 0.979 / 0 | 0.979 / 0 | | Semantic held-out, 32 pages, 20 queries | 0.628 / 0.13 | 0.925 / 0 | 0.925 / 0 | Hybrid lifts both rank measures on the coverage probes, and semantic and hybrid lead on the judged suites, but neither ever returns an empty result, so every negative query is answered with the nearest pages. These are small local corpora, a domain proxy and authored judgments, measured once; they do not establish production relevance. Hybrid's tie order, embedding first, was chosen from these runs: keyword first scored 154, 117 and 0.657 on the probes and 0.779 on the held-out relevance suite. ### Answer recall on real Google questions The probes above measure domains, not answers. NQ-open's development set holds 3,610 real Google search questions with accepted answer strings, from Google's [Natural Questions](https://github.com/google-research-datasets/natural-questions) (CC BY-SA 3.0). The file is fetched once, checked against its pinned SHA-256, and never committed: ```bash npm run search:answers -- --fetch --retrieval keyword,semantic,hybrid ``` For each question it records the first of the top ten results whose stored text contains an accepted answer as consecutive words (top-k answer recall, as in open-domain QA), and whether any searchable page contains one at all, which tells a ranking miss from a coverage gap. A short answer such as "one" also matches unrelated pages; the metric is standard and noisy in that way. The report, `.search-data/reports/answers.json`, keeps every question's results. The baseline ran on 2026-10-06 against a copy of the production index, 49,063 pages and their vectors; two questions whose answers hold no word were left out: | Mode | Top 1 | Top 5 | Top 10 | Median / 95th percentile | |---|---:|---:|---:|---:| | `keyword` | 7.7% | 13.6% | 16.1% | 115 / 276 ms | | `semantic` | 7.3% | 15.3% | 20.1% | 41 / 47 ms | | `hybrid` | 7.4% | 16.9% | 20.9% | 195 / 389 ms | Some page held an answer for 55.3% of questions. On the 2,980 questions whose answers are all distinctive (two or more words, or one non-numeric word of five or more letters), no page held one for 53.9%, and hybrid missed an indexed answer for another 30.8%: the gap is coverage first. The index's sources are not the reference sites these questions need; `en.wikipedia.org` answered 16 questions in hybrid's top ten, held to 2,000 pages by the per-origin cap (an [import of all of it](#wikipedia-at-full-scale) is measured below). For scale, the [DPR paper](https://arxiv.org/abs/2004.04906) (retrieved 2026-10-06) reports 59.1% (BM25) and 78.4% (dense) top-20 answer recall on these 3,610 questions over all of Wikipedia, split into 21 million 100-word passages: a reference point, not a like-for-like comparison, since those count passages and these count pages. Keyword retrieval now builds excerpts only for the results it returns. It used to run FTS5's `snippet()` on every candidate row, up to twenty times the results kept, and on long pages that was most of a search: over 900 keyword searches from these questions against a 70,000-page staging index that was taking in Wikipedia, median and 95th-percentile time fell from 98 and 788 ms to 32 and 430 ms, with identical results, excerpts included. Most of what remains at the 95th percentile is ranking the fallback match of any query word across tens of thousands of pages. ### Wikipedia at full scale On 2026-10-17 the [importer](#import-english-wikipedia) loaded snapshot 20261001 into a staging index on a laptop (Snapdragon X Elite, 12 cores, 32 GB, NVMe), not on the OVH host. The staging index holds Wikipedia alone: no copy of the production index was at hand, so these figures describe the imported corpus, not the production index with it, where the [host spread](#api-and-production-accounting) lets at most two Wikipedia pages lead each result list. | Import, snapshot 20261001 | | |---|---:| | Dump parts / compressed size | 71 / 26.9 GB | | Pages read | 25,932,219 | | Redirects / other namespaces skipped | 12,065,403 / 6,617,717 | | Disambiguation pages / under 40 words skipped | 375,002 / 843,093 | | Articles imported | 6,030,988 | | Index on disk | 39.9 GB, 6.6 KB per article | | Wall time, 8 decode threads, parts on disk | 110 minutes | | Longest batch commit | 9.8 s | The parts downloaded from dumps.wikimedia.org at 5 to 20 MB/s on one connection, about 25 to 90 minutes for all 71. The wall time includes a 10-minute pass that, on a first import, found nothing to remove (it is now skipped when the source held no pages), and it ran beside other work on the laptop. The JavaScript bzip2 decoder produced the same bytes as the system `bzip2` for a whole 1.13 GB part, at 14 MB/s per thread. In 4,999 sampled articles of this index no template, reference or bold markup was left, and stray link brackets or tags in at most 3; 54 (1.1%) kept the rows of a table that a template had opened (`{{Awards table}}`), which the converter has dropped since this import, as it drops other tables. An earlier converter also leaked tables indented as `: {|` and erased about 0.2% of articles at an unclosed `{{`; both were fixed before this import. The longest commit is an FTS5 segment merge inside one 1,000-page batch: a crawler writing beside the import waits behind it, and a write that waits past its 2-second busy timeout fails and is retried later as a failed fetch. Pausing the crawl timer for the import avoids that. Answer recall on NQ-open's 3,608 questions (keyword retrieval through the query pool, two workers, cache off), against the production baseline above: | Index | Top 1 | Top 5 | Top 10 | Some page holds an answer | Median / 95th percentile | |---|---:|---:|---:|---:|---:| | Production, 49,063 pages, `keyword` (2026-10-06) | 7.7% | 13.6% | 16.1% | 55.3% | 115 / 276 ms | | Production, `hybrid` (2026-10-06) | 7.4% | 16.9% | 20.9% | 55.3% | 195 / 389 ms | | Staging, 6,030,988 Wikipedia articles, `keyword` | 37.2% | 60.8% | 67.7% | 98.1% | 294 / 719 ms | On the 2,980 questions whose answers are all distinctive, no page held one for 2.3% (53.9% on production), and keyword retrieval missed an indexed answer in its top ten for 34.5%: with Wikipedia in, the gap is ranking, which is what semantic and hybrid retrieval address. Every answer in a top ten came from `en.wikipedia.org`. The title weighting can lead with a narrower page than the article a query names: `solar system` led with "List of largest craters in the Solar System". The 200 coverage probes all returned results (median 121 ms, 95th percentile 404 ms), with no expected-domain hit, since the staging index holds none of the probes' publishers. Keyword retrieval needed three changes at this size, each kept identical in its results on small indexes. Status read every row: with the code on `main` it took 167 s on 2.8 million imported pages and 13.9 minutes on 6.03 million, long past any readiness check, and it reads in 0.9 ms now that a bulk source reports its importer's committed count. Ranking read each match's crawl time, hash and provenance, which SQLite reaches through the page text's overflow pages; it now ranks without them, by FTS5 alone when a search has no domain or age filter, and joins only the best rows. On 2.8 million pages that cut "history" (853,675 matches) from 5,652 to 512 ms, and the 189 of 200 questions with a complete match returned identical results and excerpts. The partial-term fallback matched any query word across millions of pages and took 3 to 52 s on 2.8 million (one question took 7.8 minutes on 6.03 million with `main`'s code, while a vector-index build ran beside it); it now keeps the rarest words within 200,000 pages, which kept those searches under 1 s and changed their results, the 11 of 200 questions that used it. At 6.03 million pages, single-threaded on a cold cache, 300 questions took 428 ms median, 1.03 s at the 95th percentile and 1.7 s at most. Semantic and hybrid answer recall were not measured at this size: they need a vector for every article, about 1.6 billion tokens of embedding (see [Vectors for imported Wikipedia](#vectors-for-imported-wikipedia)), which is the owner's decision. The [approximate index](overview.md#semantic-retrieval) that serves them was measured instead, against exact brute force, with real `bge-small-en-v1.5` vectors (title and first 600 characters) of 50,000 articles sampled from the staging index and the 3,610 NQ-open questions as queries (with the BGE query prefix). Its 256 lists hold about 200 vectors each: | Lists scanned / rescored | Recall of the exact top 10 | Median / 95th percentile | |---|---:|---:| | 16 / 100 | 86.7% | 2.1 / 4.0 ms | | 32 / 200 | 93.8% | 3.8 / 6.9 ms | | 64 / 400 (the defaults) | 97.8% | 6.9 / 12.1 ms | | 128 / 800 | 99.6% | 14.9 / 21.9 ms | Exact brute force over the same 50,000 took 24.4 ms median. At the full corpus's size there were no real vectors to test, so the 50,000 were expanded to 6,030,988 by copying each about 120 times with Gaussian noise and normalizing, and 200 of the questions were scored against exact neighbours found by a full pass. The build (4,096 lists) took 7.7 minutes in one thread and 1.19 GB, and wrote a 392 MB index beside a 12.5 GB vectors file; each query worker holds about 372 MB of it (exact search would need 9.3 GB per worker and about 3 s a query). Recall depends on how alike the copies are. With noise of 0.045 per dimension (a copy about 0.75 in cosine from its source): | Lists scanned / rescored | Recall of the exact top 10 | Median / 95th percentile | |---|---:|---:| | 32 / 200 | 74.2% | 18.7 / 29.4 ms | | 64 / 400 (the defaults) | 83.9% | 55.5 / 84.8 ms | | 128 / 400 | 85.6% | 95.5 / 141.4 ms | | 128 / 800 | 90.0% | 107.7 / 157.4 ms | With noise of 0.02 (copies about 0.93 from their source, so the exact top ten are near-duplicates that one-bit codes cannot order), the defaults recalled 74.8% at 31.7 ms median and 128 / 800 recalled 85.3%. The copies stand in for the density of 6 million vectors, not for their structure, so recall on real Wikipedia vectors at this size remains unmeasured until they exist; scanning more lists and rescoring more candidates raise it, at the latency shown, through `VITNA_SEARCH_VECTORS_PROBES` and `VITNA_SEARCH_VECTORS_RESCORE`. Rescoring reads stored floats from the vectors file, so on the search host it depends on how much of that file stays in memory beside the index. ## Domain authority from the web graph Common Crawl's [web graph](https://commoncrawl.org/web-graphs) ranks every registered domain by harmonic centrality, and by PageRank, over three months of crawled hyperlinks; release `cc-main-2026-jul-aug-sep` covers 133.2 million domains and 2.1 billion links. Its domain ranks file is sorted by harmonic centrality, so the top domains are a prefix of one gzip file: ```bash npm run search:authority -- --release cc-main-2026-jul-aug-sep --limit 1000000 ``` This streams the first lines of the release's `domain-ranks.txt.gz`, stops the download at the limit, and replaces the contents of its own SQLite file, `VITNA_SEARCH_AUTHORITY` (default `.search-data/web/authority.sqlite`), in one transaction, so a failed run leaves the previous release in place. The top million domains took 26.6 seconds and 49 MB on 2026-10-06. A host takes the rank of its longest stored suffix (`science.nasa.gov` takes `nasa.gov`'s), so no public suffix list is needed: the release applied one when it grouped hosts. The file is an input for ranking and for choosing what to import; the top of the list is infrastructure (`googleapis.com`, `gstatic.com`, `cloudflare.com`), which serves no searchable pages. Graph data is free to use under Common Crawl's [terms](https://commoncrawl.org/terms-of-use). ## Selective archive bootstrap Common Crawl's [CDXJ index](https://commoncrawl.org/cdxj-index) identifies archived URLs and the byte ranges of their WARC records. It can bootstrap reviewed sources; it does not answer keyword searches. The importer accepts a maximum of 100 captures per plan and only sources from `web-seeds.json`. It never downloads whole crawl archives. ```bash npm run search:archive -- --plan --source web-nasa --collection CC-MAIN-2026-39 --limit 20 ``` This makes one bounded, metered CDX metadata request and saves a new plan under `.search-data/archives/`. Inspect its source scope, URLs, capture dates, and content permissions before running: ```bash npm run search:archive -- --import .search-data/archives/web-nasa-CC-MAIN-2026-39.json ``` Import checks current publisher robots/search policy, verifies the HTTP range, gzip size, WARC target/date, and payload digest, and honors captured noindex. Compressed records are limited to 2 MiB and decompressed records to 3 MiB. Unsupported encodings and chunked captures fail closed. Every network request uses the same durable day/month crawl meter; import shares the crawler lease and document/disk caps. Newer indexed content is retained, repeated captures are skipped, and each imported URL is queued for a live refresh. Current page metadata changes are observed during that live refresh. Archived results carry `contentOrigin: "common_crawl"`, `capturedAt`, `importedAt`, and archive collection/digest. Their `crawledAt` is null. Age filters use the original capture time, and status counts live and archived documents separately. Schema 1 indexes migrate transactionally to schema 2 when opened for writing; existing read-only connections detect that migration. Keep a verified snapshot before upgrading a production database. The parser, import policies, metering, and migration passed fixture tests. A live CDX planning request timed out; no real archive content has been imported in this workspace. Live crawling remains the demonstrated ingestion path. Common Crawl access is free, but content retains its owners' rights; review applicable [terms](https://commoncrawl.org/terms-of-use) and publisher permissions. ## Import English Wikipedia Live crawling adds pages slowly: at the profile's 600,000 requests a month and the first OVH run's yield of 104 pages per 1,000 requests, about 60,000 a month. A bulk import adds all of English Wikipedia, about 7 million articles, from Wikimedia's official dump: ```bash npm run search:wikipedia -- --snapshot 20261001 ``` The snapshot is a `YYYYMMDD` run listed at [dumps.wikimedia.org/enwiki](https://dumps.wikimedia.org/enwiki/) whose `articlesmultistreamdump` job is done. The importer reads that run's `dumpstatus.json` and fetches its `pages-articles-multistream` parts (71 for 20261001, 27 GB in all) one at a time from the fixed `dumps.wikimedia.org` URLs it names, resuming a partial part with a range request. Each part is checked against the size and SHA-1 Wikimedia publishes for it before any row is read from it, and a mismatch deletes the part and stops the import. A part is deleted once imported, so the disk holds at most one (about 650 MB); `--keep-parts` keeps them, `--dir` places them (by default `wikipedia/enwiki-<snapshot>/` beside the index), `--parts N` stops after N parts, and `--workers N` sets the decode threads (by default the CPU count less two, at most 8). Nothing needs installing: the bzip2 decoder is plain JavaScript and checks every block's CRC. Worker threads decode the dump's bzip2 streams in parallel and render each page's wikitext to the plain text the crawler's extractor produces for a page (`extract.mjs`): the article's prose, with links reduced to their words and the inline templates that carry prose (dates, unit conversions, language and formatting wrappers) rendered to their visible text. References, infoboxes, tables, files, categories, comments and the trailing reference, link and "See also" sections are dropped. Only main-namespace articles are imported: redirects, disambiguation pages, other namespaces and pages under the profile's 40-word minimum are skipped and counted. A page's URL is the one Wikipedia itself links to (`https://en.wikipedia.org/wiki/Ender%27s_Game`). Imported articles live in their own source, `wikipedia-enwiki-dump`, with the CC BY-SA 4.0 reuse policy, and each result carries `contentOrigin: "wikipedia_dump"`, the snapshot's capture time (when its stub dump finished; every revision in the dump was current then), the import time, and `archive: {collection, revision}`; its `contentChangedAt` is the revision's edit time. See the [result fields](overview.md#response). Wikipedia's text is reusable under CC BY-SA 4.0 with credit to Wikipedia's contributors and a link to the article; Wikimedia's [reuse guidance](https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content) describes the attribution. The index writes each batch of pages with its resume state in one transaction, so an interrupted import (a signal, a crash, a full disk) runs again with the same command and continues after the last committed batch. Before every batch the importer checks the profile's free-disk floor (`minFreeDiskBytes`, 10 GiB) and the source's cap (`maxDocumentsPerBulkSource`, 8,000,000), and stops cleanly at either. Its downloads are not crawling and are not charged to the crawl's daily or monthly request meters. It holds its own lease, renewed every minute, so the crawler and demand jobs keep running beside it; SQLite serializes their writes, a batch at a time. A batch can hold the write lock for seconds while the full-text index merges (9.8 s at most in the staging import below), longer than a crawler write waits, so stop the crawl timer for the import if failed fetches during it matter. Searches are unaffected. The crawler leaves imported pages to the import: it neither queues nor fetches a URL the import holds, and its caps count crawled pages only (see [Crawl scope and limits](#crawl-scope-and-limits)). A page the crawler fetched after the snapshot's capture keeps its crawled copy; an older crawled copy gives way to the dump's. Importing a newer snapshot rewrites the articles whose text changed, gives unchanged ones the new provenance without touching the full-text index, adds new ones, and, once every part is in, removes the articles the snapshot no longer holds (deleted, now a redirect or a disambiguation page, or under the word floor). An older snapshot than one already imported is refused. The JSON summary reports pages read, articles, each skip reason, inserted, changed, unchanged and removed pages, and `stopReason` (null once the snapshot is complete). Status reports the source's article count from the import's own committed count, so status never reads millions of pages; `importedDocuments` totals them. Vector maintenance passes imported pages over unless `VITNA_SEARCH_EMBED_BULK=true`; see [Semantic retrieval deployment](#semantic-retrieval-deployment). Optional upstream adapters need appropriate commercial rights before resale. Brave's standard terms prohibit redistribution/resale of results and using its API to replace the service; an ordinary subscription does not authorize a public search resale product. See [Brave's terms](https://api-dashboard.search.brave.com/documentation/resources/terms-of-service). ## Hosted beta The general web beta is deployed at [search.vitna.ai](https://search.vitna.ai), with an [API reference](https://search.vitna.ai/docs). Search and status require a separate Vitna Search key. The public page contains no embedded credential. Customer signup and billing are not enabled; coverage is still limited. The first scheduled OVH crawl completed 1,000 requests in about 5.4 minutes, recording 90.3 MB of payload and adding 104 pages for 175 searchable pages. Its 88 publisher failures were recorded for retries while the bounded run completed successfully. The next crawl was scheduled five minutes later. Registered origins and queued URLs are not searchable coverage. Daily private R2 backups are enabled. A real backup download passed SHA-256 and SQLite integrity checks for the index, frontier and quota snapshots. Backups expire after 14 days. HTTPS and API-key protection were verified, and a simulated certificate renewal succeeded through Cloudflare. The crawler and backup timers run on the server while the operator's PC is asleep. ## Virginia cloud deployment The Terraform module under `deploy/search-web` prepares one Ubuntu VM in `US-EAST-VA-1`, OVH's [Virginia region](https://support.us.ovhcloud.com/hc/en-us/articles/360002245164-How-to-create-a-Public-Cloud-instance-and-connect-to-it). Its default is `b3-32` (8 vCPU / 32 GB), with SSH restricted to the supplied administrator IPv4 CIDR. Only ports 80/443 are public; the API listens on loopback. The module installs Node, nginx, and rclone, and stops nginx until configuration is complete. It does not upload application code or credentials. The pilot allocation defaults to $600/month and cannot exceed $800, preserving at least $200 of the supplied $1,000/month OVH budget. An explicit current all-in `quoted_monthly_usd` must fit that allocation before planning succeeds. This is a configuration guard, not a cloud billing enforcement service. At OVH's US hourly rates checked 2026-10-03, `b3-32` compute is approximately $179.68 for 744 hours. An illustrative 200 GB of allocated local NVMe adds $25.61. IPv4, taxes, any extra storage, and other services must be added to the actual project quote. Storage and IPv4 became separate charges on 2026-10-01. Do not use the page's annual commitment price for this hourly pilot. Source: [OVH US pricing](https://us.ovhcloud.com/public-cloud/prices/). For dashboard setup, sign in to the credited OVH project, select Virginia, and inspect its current hourly instance quote and credit eligibility. Use the generated `cloud-init.yaml` below as the instance's startup configuration. Both accounts were accessed through their signed-in dashboards on 2026-10-03. The current beta reuses an existing instance; the Terraform instructions below are an alternative for a future dedicated VM, not the deployed resource state. For Terraform setup, connect the credited OVH project using its private OpenStack RC or cloud configuration through the local environment, rather than chat or Terraform variables. Confirm the project, available image/flavor, SSH public key and administrator CIDR, actual allocated disk/IP charges, and credit eligibility. The [OVH service account guide](https://support.us.ovhcloud.com/hc/en-us/articles/43234629764755-Using-service-accounts-to-connect-to-OpenStack) describes project authentication. Keep Terraform state private as well. Copy `terraform.tfvars.example` to a private `terraform.tfvars`, fill its public-key/CIDR/quote values, then run: ```bash terraform -chdir=deploy/search-web init terraform -chdir=deploy/search-web validate terraform -chdir=deploy/search-web plan -out=beta.tfplan terraform -chdir=deploy/search-web apply beta.tfplan ``` Only apply an inspected plan for that project within the approved allocation. The offline Terraform tests verify one Virginia instance, reject overspending, and reject unrestricted SSH. They do not verify a live account or provision VM resources. Local instance storage can be lost with the instance; the beta has no high-availability replica. R2 backups are part of its recovery plan. Build the deployment package with: ```bash npm run search:release ``` It writes `.search-data/releases/vitna-search-web.tgz`, a SHA-256 checksum, and `cloud-init.yaml` for dashboard-based instance creation. The startup configuration prepares the same runtime/user as Terraform and keeps nginx stopped until application configuration is complete. The Linux Node runtime, SQLite FTS5 support, service installation, API authentication, live crawl, and R2 upload/download restore check have been verified on the actual reused VM. The local release archive `.search-data/releases/vitna-search-web.tgz` contains search code, deployment files, and these docs. It excludes data, environment files, Terraform state, provider caches, and credentials. Transfer it to the prepared VM through SSH, extract into `/opt/vitna-search`, and run: ```bash cd /opt/vitna-search sudo bash deploy/search-web/install-release.sh sudo /opt/vitna-search-node/bin/node platform/search/mint-key.mjs pilot ``` Fill `/etc/vitna-search/web.env` with the client hash and reviewed settings. This environment file is readable only by root and the service group. The install helper preserves existing private settings and does not start services. Initialize and inspect a small live crawl before enabling its timer: ```bash sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/web-run.mjs --max-requests 100 sudo systemctl enable --now vitna-search-web.service curl --fail http://127.0.0.1:8789/readyz ``` Choose the owned search hostname, configure DNS and a valid TLS certificate, adapt `deploy/vitna-search.nginx.conf.example`, and verify nginx's configuration before starting it. The hostname in that template is an example, not a created DNS record. Configure the R2 backup below and perform one successful upload and restore check before enabling the recurring work: ```bash sudo systemctl enable --now vitna-search-web-crawl.timer vitna-search-web-backup.timer ``` The crawl timer advances the durable queue five minutes after each run ends; it does not recrawl every page every five minutes. The backup timer runs daily. These are VM timers; they do not keep the user's PC crawling while it is asleep. ## Semantic retrieval deployment Optional; [semantic retrieval](overview.md#semantic-retrieval) describes the contract. It uses Vitna's own embedding engine, vitna-anchor, which runs on the search host as two units serving `bge-small-en-v1.5`: `vitna-embed.service` at `127.0.0.1:8795` embeds queries, and `vitna-embed-batch.service` at `127.0.0.1:8797` embeds pages for vector maintenance. The engine serves one request at a time, so with two units a query never waits behind a batch: during a backfill at 16 pages a batch, a query's embedding took 17 to 21 ms from the query unit against 0.8 to 1.9 s when it shared the one engine. The weights are mapped from the same file, so the second unit costs little memory. Nothing here makes a request to any third party. The engine binary is built from convexityos/vitna-anchor, gate A14 or later, for Linux x86-64 (for example `zig cc -target x86_64-linux-gnu.2.35 -O3 -Iinclude src/*.c -o vitna-anchor -lm -lpthread` in its `engine/`), and the model fetched with that repository's `node scripts/fetch-model.mjs bge-small-en-v1.5`. The release's installer checks the binary against the digest given and every model file against the pin, installs both units, and starts nothing: ```bash sudo bash deploy/search-web/install-embed.sh /path/to/vitna-anchor <sha256> /path/to/bge-small-en-v1.5 sudo systemctl enable --now vitna-embed.service vitna-embed-batch.service curl -s http://127.0.0.1:8795/health; curl -s http://127.0.0.1:8797/health ``` It should answer with `"embeddings":true`, `"dimensions":384` and `"max_tokens":512`. Use the model identifier it reports. Add the settings to `/etc/vitna-search/web.env`; commented examples are in `deploy/vitna-search-web.env.example`, and the vectors file defaults to `/var/lib/vitna-search/web/vectors.sqlite`: ```dotenv VITNA_SEARCH_EMBED_URL=http://127.0.0.1:8795/v1/embeddings VITNA_SEARCH_EMBED_BATCH_URL=http://127.0.0.1:8797/v1/embeddings VITNA_SEARCH_EMBED_BATCH_SIZE=16 VITNA_SEARCH_EMBED_MODEL=bge-small-en-v1.5 VITNA_SEARCH_EMBED_QUERY_PREFIX="Represent this sentence for searching relevant passages: " VITNA_SEARCH_EMBED_DIMENSIONS=384 ``` The query prefix is the one BGE documents for short queries; in the measurements below, omitting it changed the results by a few queries either way. Fill the vectors file before switching the API over. Each run stops at its budget and the next continues; repeat until the summary reports `"complete":true`. The first pass embeds every page, later passes only new and changed pages: ```bash sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/embed-run.mjs --budget-ms 3600000 ``` On a desktop Ryzen 7 3700X with the engine at the production unit's three threads, embedding averaged about 2,100 tokens per second over pages of about 285 tokens each. At that rate 49,063 pages need roughly two hours of engine time: about two such runs by hand, or four to five hours through the timer alone. The search host's rate will differ. The release installer places `vitna-search-web-embed.service` and `.timer` without enabling them. Enable the timer, which runs the job ten minutes after each run ends within an eight-minute budget, then restart the API so it reads the settings: ```bash sudo systemctl enable --now vitna-search-web-embed.timer sudo systemctl restart vitna-search-web.service ``` Authenticated `GET /v1/status` should then report `retrieval.semantic.state: "ready"` with a `vectors` count near `index.documents`, and a search should report `meta.index.retrieval.used: "hybrid"`. Receipts in `journalctl -u vitna-search-web` show any `fallback` codes. Each query worker holds its own copy of the vectors inside the API unit's 4 GB limit, about 160 MB for two workers at 49,063 pages and 384 dimensions. The engine serves one request at a time, so without `VITNA_SEARCH_EMBED_BATCH_URL` a search's query embedding waits for any maintenance batch in progress on the one engine. At three threads, a query arriving during 32-page batches waited 4.0 s median, and 24 of 26 exceeded the 2 s `VITNA_SEARCH_EMBED_TIMEOUT_MS` default and would have fallen back to keyword retrieval. During the default 4-page batches the wait was 0.35 s median and 0.51 s at the 95th percentile, none over 2 s, with no loss of throughput (2,125 against 2,223 tokens per second). On one engine, raise `VITNA_SEARCH_EMBED_BATCH_SIZE` only if searches can tolerate that wait; with the batch unit, queries never see it, and 16 pages a batch keeps the batch engine busy. To turn the feature off, comment out the settings and restart the API; the vectors file is then ignored. To change the model, point `VITNA_SEARCH_VECTORS` at a new file, fill it, and restart the API. The vectors file is part of the R2 bundle, as `vectors.sqlite.gz`. It is a snapshot of its own, so it can trail or lead the index snapshot slightly; after a restore the job removes vectors whose page changed identity and embeds what is missing. Rebuilding it from the index instead took about 1.2 hours at 4 engine threads for 49,063 pages. The backup reads only `web-backup.env`, so a `VITNA_SEARCH_VECTORS` set for the API must be set there too. ### Vectors for imported Wikipedia Vector maintenance passes [imported](#import-english-wikipedia) articles over unless `VITNA_SEARCH_EMBED_BULK=true`, because embedding them is the expensive part of semantic retrieval over Wikipedia. Each article embeds its title and leading text up to the 1,500-character recipe, 260 tokens on average over 3,000 sampled articles, so the 6.03 million articles of snapshot 20261001 are about 1.6 billion tokens: some 8.6 days at the 2,100 tokens per second measured above for three engine threads, and still days on all eight cores of the search host, during which the batch engine is fully busy. Whether and where to spend that is the owner's decision. With the setting on, crawled pages still go first, the job keeps its memory bounded by comparing pages and vectors in ranges of 100,000 ids, and each run starts with a pass over the whole index and vectors file, so a backfill runs best by hand with a one-hour budget rather than the timer's eight minutes. Seven million 384-dimension vectors are about 11 GB in `vectors.sqlite`, far past what a query worker can hold, so build the approximate index once the backfill is in, with the API's environment, and give each worker room for it: ```bash sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/vector-index-run.mjs ``` ```dotenv VITNA_SEARCH_VECTORS_MAX_MB=1024 ``` The API reads the bound at start, so restart it after raising the bound; it picks up a new or rebuilt index on its own. Vectors embedded after a build are searched exactly beside the index until the next build. ## Private R2 backups and recovery Create a dedicated private R2 Standard bucket and a bucket-scoped Object Read & Write credential. Put it in `/etc/vitna-search/rclone.conf`, using `deploy/vitna-search-rclone.conf.example`. Restrict it to root and the service group, mode 0640. Keep the existing `vitna-search-r2` remote name or update `/etc/vitna-search/web-backup.env` to match it and the actual bucket. Configure a 14-day R2 expiration rule for the `web-backups/` prefix, plus incomplete multipart cleanup. Remote lifecycle settings require cloud account access and are not created by this package. The helper keeps two local bundles after successful upload; failed uploads retain their local files and require attention before they consume the disk. Backups need temporary space for a raw SQLite snapshot plus the compressed file. Since the 2026-10-17 Wikipedia import the index is about 40 GB, so the snapshot is that large and a bundle should be about 16 GB. A run plans for its peak (each raw snapshot beside its compressed file and the compressed files before it, at the compression the last local manifest measured, or one half without one) and leaves `VITNA_SEARCH_BACKUP_MIN_FREE_GB` free at every step, 20 GiB by default, twice the crawler's own stop: - To make room it removes local bundles that R2 already holds, oldest first and only as many as it needs. A bundle is marked as held by `uploaded.json`, written once its remote manifest lands. A bundle that never finished uploading may be the only copy and is never removed for space. - If the room is still short, the run fails before any snapshot and says how much it needed. - It checks again before each snapshot and while compressing, so a file that compresses worse than planned stops the run at the floor, not at a full disk. - A run that stops before its local manifest removes its own partial bundle. One killed outright (the unit's `TimeoutStartSec` is 6 hours) leaves one, and the next run removes it once it is 12 hours old. Each upload may take five minutes plus one second for every 2 MiB, and `manifest.json` records each file's raw size as `sourceBytes`, which is the space a restore needs to decompress it. ```bash sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web-backup.env platform/search/web-backup-run.mjs npm run search:web:backup ``` The npm alias reads the same settings from its environment when used locally. Each bundle contains compressed index, frontier, and API quota databases, the concept catalog when it holds titles, and the semantic vectors file when one holds vectors (beside the index unless `VITNA_SEARCH_VECTORS` names another), plus SHA-256 hashes in `manifest.json`. A file at the vectors path that belongs to another database stops the backup before any snapshot. The run's JSON line lists the snapshots taken. SQLite's snapshot operation and integrity check run before upload. The remote manifest uploads last as a completion marker. A prefix without a manifest is an incomplete remote backup. Discovery contexts are encrypted inside the frontier snapshot. The separate discovery key is not included in the database bundle; keep a private recovery copy if pending jobs must survive restoration, and restore the same key to all three service processes before resuming them. For recovery, stop the API, crawler service/timer, embed timer, and backup service/timer. Download one complete manifest and every file it lists into a new private staging directory. Verify compressed byte counts and SHA-256 hashes, decompress them, and run SQLite integrity checks before replacing any data. Preserve the old files for rollback and set restored ownership/modes for `vitna-search`. A bundle without `vectors.sqlite.gz` predates the vectors backup: leave the vectors file absent and the embed job rebuilds it. The snapshots are individually consistent, not one transaction across databases. Queue leases expire and interrupted work may be retried; reconcile the queue against the index after restoration. Restoring older quota or crawl meters can lose usage after the snapshot. Reconcile those counters from saved status/receipts before resuming; if usage cannot be recovered, keep affected work paused until its day/month limit resets. Verify `/readyz`, an authenticated search/status, and the new usage values before restarting timers. R2 Standard storage is $0.015/GB-month, with a 10 GB-month free storage tier and free Internet egress; operation charges also apply. Source: [R2 pricing](https://developers.cloudflare.com/r2/pricing/). The supplied $10,000 Cloudflare credit award needs its actual eligible products and expiry checked in the account. This implementation assumes no credit balance, award duration, or free unlimited storage. --- <!-- https://vitna.ai/docs/search/grounding.md --> # Grounding check `platform/search/grounding.mjs` decides, segment by segment, whether an answer a model wrote from Vitna Search evidence is supported by the verbatim passages it cites. It is deterministic and lexical: it uses no model and no network, and it checks that what a sentence states occurs in the cited passages, not what the sentence means. ## Where it sits Search supplies the evidence. [Source brief research](overview.md#source-brief-research) returns verbatim passages, each with a citation id, a source, and code-point offsets into the stored text version read for that source. The answer is written outside Search, by a model the caller runs, from those passages only, citing them with inline markers such as `[c1]`. The verifier then reads the evidence and the answer together and says which segments the evidence supports, and returns the supported text alone. It runs where the answer is written, in the caller's process, reads the two values it is handed, and sends nothing anywhere. The file imports nothing and uses no regular-expression lookbehind, so it can be copied whole into code for Node, Deno, Bun, Workers or a browser; it is not yet published in the JavaScript or Python SDK. ```js import { verifyAnswer } from "./grounding.mjs"; const evidence = await response.json(); // the /v1/research response, as returned const result = verifyAnswer(evidence, answer, { minOverlap: 0.45 }); ``` `minOverlap` is optional and defaults to `GROUNDING_MIN_OVERLAP`, the measured `0.45`. The module also exports `parseCitedAnswer(text)`, which returns the answer's segments as `{text, citationIds, kind}` with markers removed, and the `INSUFFICIENT_EVIDENCE` sentinel. `supported` means the cited passages hold the segment's figures, quotations and names and most of its words. It does not mean the segment is true: passages are publisher text, and the checks below cannot see a reversed meaning. ## Evidence The verifier reads `sources`, `citations`, and the passage texts quoted in `overview` and `sections[].passages`. A top-level `passages` list, or a `text` on the citation itself, is read too. Several research responses merged into one pack work the same way, provided their source and citation ids are renumbered to stay unique. A citation is usable only when its id is unique, `offsetUnit` is `unicode_code_points`, `start` and `end` are integers with `start` below `end`, `sourceId` names a listed source, exactly one passage text is quoted for it, and that text is `end - start` code points long. When a source carries its `text`, the passage must also be that slice of it. A citation that fails is listed in `evidence.invalid` with its reason: `duplicate_id`, `offset_unit`, `offsets`, `unknown_source`, `missing_text`, `conflicting_text`, `too_long`, `length_mismatch` or `source_text_mismatch`. A segment citing it is refused with `invalid_citation` instead of being checked against it. ## Answer The answer is Markdown text with inline markers, or a list of segments, `[{ text, citationIds }]`. ```markdown ## Opening It opened to pedestrians on May 27, 1937 [c1]. Cars crossed on May 29, 1937 [c1]. ``` Text is parsed into segments: each heading (a `#` line, or a line that is only bold text), and each sentence of a paragraph or list item. Sentences end at `.`, `!` or `?` followed by a capital, a digit or a quotation mark, except after common abbreviations such as `Dr.`, `e.g.` and `U.S.`, and after a single initial. Markers just after a sentence's full stop belong to that sentence, and a line holding only markers belongs to the sentence before it. A bracket is a marker when each comma-separated part is an id in the evidence or has the shape of one, letters then digits or digits alone, so `[1]` is refused as an unknown id rather than read as the figure 1. A link's bracketed text is not a marker. Fences and horizontal rules are markup; the text inside a fence is read as a paragraph. A segment list is used as given, with any inline markers in a segment's text added to its ids. A segment whose text opens with `#`, or whose `kind` is `heading`, is a heading; one opening with a list marker is a list item. When the passages do not answer the question, the model should reply with `INSUFFICIENT_EVIDENCE` alone. An answer that is only that word, allowing surrounding whitespace, quotes, backticks or a full stop, has no segments: `coverage.status` is `insufficient` and `coverage.abstained` is `true`. Beside other text the word is an uncited sentence and is refused. ## Output ```json { "segments": [ {"index": 0, "kind": "heading", "text": "Opening", "citationIds": [], "claim": false, "supported": true, "reasons": [], "details": [], "overlap": null}, {"index": 1, "kind": "sentence", "text": "It opened to pedestrians on May 27, 1937.", "citationIds": ["c1"], "claim": true, "supported": true, "reasons": [], "details": [], "overlap": 1}, {"index": 2, "kind": "sentence", "text": "Cars crossed on May 29, 1937.", "citationIds": ["c1"], "claim": true, "supported": false, "reasons": ["unsupported_number", "low_overlap"], "details": [{"reason": "unsupported_number", "text": "May 29, 1937"}, {"reason": "low_overlap", "text": "cars crossed"}], "overlap": 0} ], "supportedText": "## Opening\n\nIt opened to pedestrians on May 27, 1937.", "coverage": {"status": "partial", "segments": 3, "supported": 2, "unsupported": 1, "claims": 2, "supportedClaims": 1, "abstained": false}, "evidence": {"citations": 1, "usable": 1, "invalid": []}, "minOverlap": 0.45 } ``` | Field | Meaning | |---|---| | `segments[].text` | The segment with its markers, list marker and emphasis removed | | `segments[].kind` | `heading`, `lead_in`, `connective`, `closing`, `item` or `sentence` | | `segments[].claim` | `false` for the four structural kinds, which need no citation | | `segments[].supported` | Every check passed | | `segments[].reasons` | Reason codes, in the order of the table below | | `segments[].details` | Each failing token as `{reason, text}`; for `low_overlap`, the content words no cited passage holds | | `segments[].overlap` | Share of the segment's content words found in its cited passages, or `null` | | `supportedText` | The supported segments with their headings and list markers, citation markers removed | | `coverage.status` | `matched`, `partial` or `insufficient`, as research uses them | | `coverage.supported` / `unsupported` | Segments supported and refused, structural ones included | | `coverage.claims` / `supportedClaims` | Segments that are claims, and those supported | | `coverage.abstained` | The answer was only `INSUFFICIENT_EVIDENCE` | `coverage.status` is `matched` when at least one claim is supported and no segment is refused, `partial` when some claim is supported and some segment is refused, and `insufficient` when no claim is supported. A refused heading or lead-in counts as refused. In `supportedText` a heading is kept only over a supported claim, a lead-in only before one, and a connective or closing sentence only when some claim survived. The segments keep their `citationIds`, so a reader can show the supported text with its sources. ## Checks A claim is any segment that is not structural. Each claim is checked against the usable passages it cites, and every failure is reported, not only the first. | Reason | When | |---|---| | `missing_citation` | The claim cites nothing | | `unknown_citation` | It cites an id the evidence does not hold, such as an invented `c9` or a source id like `s3` | | `invalid_citation` | It cites a citation that failed the evidence checks | | `unsupported_number` | A number, percentage, money amount, year or date is in no cited passage | | `unsupported_quote` | A quoted string is in no cited passage | | `unsupported_name` | A capitalised name is in no cited passage | | `low_overlap` | Fewer than `0.45` of its content words occur in the cited passages | One unknown or invalid id refuses a claim even beside a good one. The content checks still run against the usable ids, so every problem is listed. Numbers are compared as values. `1,200,000`, `1.2 million` and `$1.2m` are the same value; a currency symbol or word must match; a percentage must meet a percentage; a bare number may meet a passage number of any kind. Spelled numbers count (`eleven`, `twenty-five`, `a thousand`), and a lone `one` counts only as a quantity, not in `one of` or `no one`. A figure hedged with `about`, `roughly`, `nearly` or the like may round the passage's figure to its own precision, so `about 40%` meets `38.7%` but `about 99%` does not meet `97%`. An unhedged figure must match exactly. Dates are read in their written forms (`May 27, 1937`, `27 May 1937`, `2022-08-18`, `August 2022`, `May 27`) and compared as dates. A date is supported by one at least as specific, so `May 1937` meets `May 27, 1937`, and a passage's `12 March` meets `12 March 2021` when the passage holds 2021 elsewhere. Quotations in straight or curly double quotes, guillemets, and single quotes that are not apostrophes must occur as a word sequence in one cited passage, ignoring case, punctuation and diacritics. Names are runs of capitalised words, joined across `of`, `de`, `van` and the like, matched as a word sequence in one cited passage, ignoring case and inflection. Month and weekday names are left to the date check. A sentence's first word joins a name only when the evidence writes it capitalised mid-sentence, or when it starts a longer run and is not an ordinary word of the cited passages, so `Brooklyn Bridge` is checked whole while `Typically` is not a name. Headings are title case, so their capitals are not read as names. Content words are the segment's words less function words, discourse words such as `overall` and `according`, numbers, and month names, after a light suffix stripper on both sides (`spans` meets `spanning`, `released` meets `releases`). Diacritics and letters such as `ł` fold, so `Sklodowska` meets `Skłodowska`. ## Structural text Four kinds of segment make no claim of their own and need no citation, each only while it is 20 words or shorter. | Kind | What it is | |---|---| | `heading` | A Markdown heading, or a bold-only line | | `lead_in` | An uncited sentence ending in a colon, such as `Bakers watch a few signals:` | | `connective` | An uncited segment with no content word, figure or quotation, such as `Hope this helps!` | | `closing` | The answer's last segment, uncited, that passes every check, overlap included, against the passages the answer's supported claims cited | Structural text still answers for what it states. A number, quotation or name in a heading, lead-in or connective must occur in the passages it cites or, citing none, somewhere in the evidence; so `# Opened in 1938` is refused when no passage says 1938. A closing sentence that fails any check is an uncited claim and is refused with `missing_citation`. A list item is a claim like any sentence. ## Measured precision and recall ```bash npm run search:grounding-evaluate ``` The command verifies twelve labelled answers in `platform/search/grounding-fixture.mjs` and writes `.search-data/reports/grounding.json` (or `--output path`) with the threshold sweep, the results per split and per kind, and every misjudged segment. Each answer is written out as the Markdown a model would return and verified whole, so the parser is measured with the checks; a pack that does not parse into one segment per labelled row fails the run. The fixture holds 230 segments over twelve evidence packs shaped like research responses: 115 supported and 115 adversarial. Six packs choose the threshold and six held-out packs only measure it. The threshold is the lowest with the best F0.5 of predicting `supported` on the development packs, precision weighted over recall because a passed unsupported sentence costs a reader more than a refused one. On the development packs F0.5 is 0.866 at both 0.45 and 0.5. Recall starts to fall at 0.55, and precision rises again only from 0.9, where the overlap refuses some named limits along with a third of the supported segments. The module's `0.45` is that choice, and a test holds the two equal. | Packs | Segments | Precision | Recall | |---|---|---|---| | Development | 114 | 0.838 | 1.000 | | Held out | 116 | 0.871 | 0.931 | | All packs | 230 | 0.854 | 0.965 | | All packs, named limits left out | 211 | 1.000 | 0.965 | Precision is held below 1 by the named limits below, which the adversarial half includes on purpose. Every other adversarial segment is refused. | Kind | Label | Segments | Judged as labelled | |---|---|---|---| | `verbatim` | Supported | 31 | 31 | | `paraphrase` | Supported | 34 | 33 | | `synthesis` | Supported | 6 | 6 | | `heading` | Supported | 12 | 12 | | `lead_in` | Supported | 7 | 7 | | `list_item` | Supported | 13 | 13 | | `closing` | Supported | 12 | 9 | | `changed_number` | Adversarial | 30 | 30 | | `changed_date` | Adversarial | 8 | 8 | | `swapped_name` | Adversarial | 12 | 12 | | `uncited` | Adversarial | 12 | 12 | | `invented_id` | Adversarial | 12 | 12 | | `merged` | Adversarial | 12 | 12 | | `wrong_quote` | Adversarial | 7 | 7 | | `embellished` | Adversarial | 3 | 3 | | `negation` | Named limit | 6 | 0 | | `comparison` | Named limit | 5 | 0 | | `role_swap` | Named limit | 5 | 0 | | `substitution` | Named limit | 3 | 0 | The four supported segments refused are all held out: three closing sentences that add an evaluation the evidence does not make (`the budget grows modestly`, `two doses give strong protection`, `pressure, dose and time decide the shot`), and one paraphrase sharing two of five content words with its passage (`Its net loss shrank to $6.1 million, compared with $9.8 million a year before`). The fixture is synthetic and was written by one author who knew the checks. The two Calder County and Northwind Robotics packs are invented; the rest follow well-known public facts, but each pack asserts only what its own passages say. It measures the checks on the failure kinds named here, not a model's real error rate, and a model's errors will include subtler ones. ## Limits A lexical check confirms that tokens occur, not what a sentence does with them. These pass, and the fixture counts them as passed unsupported segments: - Negation. `The bridge is not painted International Orange` keeps every word of its passage; `not`, `no` and `never` are function words to the overlap, and no other check reads polarity. - Reversed comparison or change. `falls` for `rises`, `cooler` for `warmer`, `widened` for `narrowed`: one changed word in a sentence otherwise drawn from the passage leaves the overlap high. - Swapped roles. Two entities or two figures of the same passage exchanged, as in `Aldrin became the first person to step onto the lunar surface`, or `$15.9 million, up from $18.2 million` for the reverse. That includes a figure moved to another unit or quantity, such as `21.5 pounds` where the passage has `47.5 pounds (21.5 kg)`: units are not compared. - Substitution. One ordinary word replaced in a long sentence, such as `stroma` for `thylakoid membranes` or `second` for `first`. Spelled ordinals are words, not numbers. Other edges, measured or known: - A count in a lead-in, such as `Two changes matter most:`, must occur in the evidence, and can be met by coincidence. Both of the fixture's passed that way, one on `The two astronauts` and the other on the day of `October 2`. - A paraphrase that shares too few words with its passage is refused, and so is a closing sentence that evaluates rather than restates. - Only hedged figures round. `1.2 million` without `about` must match exactly. - Stopwords, number words, month names and the suffix stripper are English. - The verifier trusts a passage as far as its length, and its slice when the source text is supplied. It fetches nothing, so it cannot tell whether the passage is still what the page says; a later crawl can change the offsets. --- <!-- https://vitna.ai/docs/search/indexes.md --> # Vitna Search Vertical Indexes Vitna Search exposes specialized vertical corpora alongside the default `web` index through a single `index` field on `/v1/search`, `/v1/similar`, and `/v1/extract`: - `web`: Vitna's general web index (default when `index` is omitted). - `research`: Scholarly paper abstracts and metadata (PubMed / MEDLINE, arXiv, bioRxiv, medRxiv, and CC0 OpenAlex non-overlapping works). - `government`: US federal and public-sector records (eCFR, Federal Register, CourtListener US case law, and Common Crawl `.gov` / `.mil` extracts). - `developer`: Open-source package registries, OpenAPI specifications, and public GitHub issues, merged pull requests, and releases (npm, PyPI, crates.io, APIs.guru, GH Archive). - `filings`: US public company filings and XBRL financial statements (SEC EDGAR submissions, company facts, 10-K, 10-Q, 8-K, DEF 14A, S-1, and 20-F filings). - `security`: Vulnerability and threat intelligence (CVE v5, OSV across 48 package/OS ecosystems, CISA KEV, and MITRE ATT&CK). - `clinical`: Clinical research and regulatory drug/device records (ClinicalTrials.gov v2 studies, openFDA drug labels, 510(k) and PMA device clearances, Orange Book, and drug/device recalls). - `patents`: US utility and design patents (USPTO bulk text corpus). - `knowledge`: Clean encyclopedic reference articles (English Wikipedia). - `finance`: Financial filings, insider/institutional holdings, macroeconomic series, and corporate entity registries (Wave 3 scaffold: SEC Form 4/13F/Form D, FRED, IAPD, Companies House, GLEIF LEI, CFPB). - `public-records`: Federal contracts, grants, nonprofit 990s, campaign finance, lobbying, trademarks, and regulatory registries (Wave 3 scaffold). - `health`: Open-access biomedical full text, DailyMed SPL inserts, MedlinePlus, StatPearls, CMS provider data, ICD classifications, and FAERS (Wave 3 scaffold). - `ai`: AI model cards, open datasets, demo spaces, peer-reviewed ML papers, and open AI tool registries (Wave 3 scaffold). - `media`: News wire extracts, books, music releases, podcasts, and media catalog records with 365-day Quickwit retention (Wave 3 scaffold). - `gaming`: Steam catalog/patch notes, Modrinth mods, IGDB/Wikidata game catalogs, and CC-BY-SA gaming wikis (Wave 3 scaffold). - `sports`: Historical play-by-play/box scores and schedules across Retrosheet, nflverse, hoopR, hockeyR, fastf1, and OpenFootball (Wave 3 scaffold). - `listings`: Local places/POIs, public job openings, events, and public property records with 90-day Quickwit retention (Wave 3 scaffold). - `ecommerce`: Schema.org Product/Offer web extracts and Open Food/Beauty/Products Facts with 90-day Quickwit retention (Wave 3 scaffold). - `social`: Public discussions and open federated streams across Bluesky Jetstream, Mastodon, Hacker News, Stack Exchange, and Lemmy with 180-day Quickwit retention (Wave 3 scaffold). - `real-estate`: Zillow Research ZHVI/ZORI/market metrics, Realtor.com Core Inventory, FHFA House Price Index, NYC Property Sales, and UK HM Land Registry Price Paid sales (Wave 4 scaffold). - `jobs`: O*NET occupational profiles and live public ATS job boards across Greenhouse, Lever, and Ashby with 90-day Quickwit retention (Wave 4 scaffold). - `legal`: CourtListener bulk court directories, judicial biographies, opinion clusters, and federal/state dockets (Wave 4 scaffold). - `travel`: OurAirports global directory, OpenFlights airlines/routes, and English Wikivoyage destination guides (Wave 4 scaffold). - `energy-climate`: WRI Global Power Plant Database, Our World in Data energy transition profiles, NOAA Storm Events, and USGS Earthquakes (Wave 4 scaffold). ## API Contract Pass `index: "auto" | "web" | "research" | "developer" | "government" | "filings" | "security" | "clinical" | "patents" | "knowledge" | "finance" | "public-records" | "health" | "ai" | "media" | "gaming" | "sports" | "listings" | "ecommerce" | "social" | "real-estate" | "jobs" | "legal" | "travel" | "energy-climate"` on `POST /v1/search` (`"auto"` classifies the query, fans out to `"web"` plus up to 2 matching configured verticals, and fuses ranked hits with Reciprocal Rank Fusion; omit or pass `"web"` for general search). `/v1/similar` and `/v1/extract` accept `"web"` or a single named vertical index: - `POST /v1/search`: Hybrid BM25 (Quickwit) + dense vector ANN (`vitna-vec`) retrieval fused with Reciprocal Rank Fusion. Supports `domains`, `excludeDomains`, `startPublishedDate`, `endPublishedDate`, `maxResults`, `retrieval` (`keyword`, `semantic`, `hybrid`), and inline `contents`. Query-guided web expansion (`expand: true`) and third-party fallback (`fallback: true`) are rejected on vertical indexes. - `POST /v1/similar`: Resolves the seed document from Quickwit by URL, queries `vitna-vec` `/v1/similar` by `doc_id`, and hydrates ranked hits from Quickwit. - `POST /v1/extract`: Fetches stored full text directly from Quickwit / R2 without touching the origin website. - `GET /v1/indexes`: Lists every registered vertical corpus with `wave`, `status` (`"serving"`, `"backfilling"`, or `"scaffold"`), `documents`, `vectors`, and `snapshotDate`. - `GET /v1/status`: Reports `indexes.<name>` with `wave`, `status` (`"serving"`, `"backfilling"`, or `"scaffold"`), `ready`, `documents`, `vectors`, `snapshotDate`, and `semantic` for each configured vertical. When a requested vertical is unconfigured (`"scaffold"`), unavailable, or still `"backfilling"` (`0` published documents) on the serving node, `/v1/search`, `/v1/similar`, and `/v1/extract` return HTTP `503` with `error.code = "index_unavailable"` before charging client quota. ## Serving Architecture Vertical indexes separate storage from compute so all 140M+ documents serve from a single OVH `b3-16` (4 vCPU, 15 GiB RAM, 100 GB NVMe) behind the private vRack (`10.1.0.0/16`): 1. **Cloudflare R2 (`vitna-index`)**: Source of truth for canonical `.jsonl.gz` source shards (`sources/<index>/<snapshot>/`), Quickwit BM25 + zstd doc-store splits (`quickwit/<index>/`), and versioned `vitna-vec` binary artifacts (`vectors/<index>/<snapshot>/`). 2. **Quickwit (`10.1.2.86:7280`)**: Stateless Rust/Tantivy lexical search and document hydration reading split metadata and posting lists directly from R2 with bounded local RAM/NVMe split caches. 3. **`vitna-vec` (`10.1.2.86:8798`)**: Compact Rust ANN server (`vec/`) using shared-mmap 1-bit binary IVF lists for RAM-efficient candidate generation, slot-ordered `int8` or `sq4` NVMe vector rescoring, and a binary-searchable `idmap.bin` for `/v1/similar` lookups. 4. **Vitna Search Web (`40.160.7.54` / `10.1.0.94`)**: Terminates public TLS, embeds queries with local `vitna-embed` (`bge-small-en-v1.5`), fans out over private vRack (`<1 ms` RTT) to `10.1.2.86:7280` and `10.1.2.86:8798`, fuses results, and applies Vitna's standard response envelope. Firewall rules on `10.1.2.86` restrict TCP `7280` and `8798` exclusively to `10.1.0.94` on the private vRack interface (`ens4`). ## Deduplication and Retention 1. **Unique-Upload Guard (`DedupStore`)**: Both Node (`dedup-store.mjs`) and Python (`dedup_store.py`) shard writers, Quickwit R2 watchers, and `embed_shards.py` back every corpus with a SQLite `WITHOUT ROWID` `seen_docs` index (plus optional 128-bit `content_hash` deduplication). Candidate records are checked read-only first and committed atomically only after R2 upload, Quickwit ingestion, or atomic `.ids`/`.f16` writes succeed. 2. **Wave-3/4 Quickwit Retention**: High-churn indexes apply automatic Quickwit retention policies in `deploy/vitna-indexes/index-*.yaml` (`90 days` on `listings`, `ecommerce`, and `jobs`, `180 days` on `social`, and `365 days` on `media`). ## Operator Commands Credentials are split by role: `/etc/vitna-indexes/reader.env` (read-only R2 token for serving/sync) and `/etc/vitna-indexes/writer.env` (write R2 token for ingestion/GPU build). Never place write credentials on public-facing web nodes. ### Backfills ```bash # Research (PubMed baseline + updates + arXiv + bioRxiv + medRxiv, plus optional OpenAlex top-up) npm run index:research:backfill -- --snapshot 20261120 --concurrency 3 npm run index:research:backfill -- --snapshot 20261120 --stages openalex --openalex-limit 15000000 npm run index:research:ingest -- --snapshot 20261120 --source pubmed-updates # Government (eCFR + Federal Register) npm run index:gov:backfill -- --snapshot 20261120 --stages ecfr,federal-register # Developer (APIs.guru OpenAPI, bulk PyPI/npm/crates.io registries, and GH Archive issues/PRs/releases) npm run index:dev:backfill -- --snapshot 20261120 --stages openapi,gh-archive --gh-from 2025-01-01 npm run index:dev:packages -- --snapshot 20261120 --stages crates,pypi,npm # Resumable R2-to-Quickwit watch ingest for AWS bulk shards npm run index:quickwit:ingest -- --index vitna-research --prefix sources/research/20261120/openalex/ --watch # Build the native Rust vitna-vec binary (release mode) npm run index:vec:build ``` ### AWS `us-east-1` Bulk Processing & GPU Vector Builds In-region AWS Spot instances read `s3://commoncrawl`, `s3://openalex`, and `s3://com-courtlistener-storage` at zero egress cost and build `vitna-vec` artifacts from R2 canonical shards: ```bash # 1. Verify FP16 GPU embedding parity against vitna-embed (cosine >= 0.999) python3 platform/search/indexes/gpu/verify_parity.py --shard /mnt/nvme/sample.jsonl.gz --anchor-url http://127.0.0.1:8795/v1/embeddings # 2. Embed canonical shards on GPU, build vitna-vec (int8 or sq4), and upload to R2 bash platform/search/indexes/gpu/aws_spot_run.sh research 20261120 int8 # 3. Pull verified vectors onto 10.1.2.86 and hot-reload vitna-vec npm run index:vectors:sync -- pull --corpus research --snapshot 20261120 --dir /var/lib/vitna-indexes/vec/research --reload-url http://10.1.2.86:8798 ``` ### Enabling Verticals on `vitna-search-web` Configure these environment variables on `vitna-search-web`: ```bash VITNA_SEARCH_QUICKWIT_URL=http://10.1.2.86:7280 VITNA_SEARCH_VEC_URL=http://10.1.2.86:8798 VITNA_SEARCH_VERTICAL_INDEXES=research,government,developer ``` --- <!-- https://vitna.ai/docs/search/live-engine.md --> # Migrate Bright Data and Live/Hybrid clients Vitna retrieves actual Google and Bing organic SERPs through its own transport and parser at `POST /v1/serp`, with a URL-based compatibility path at `POST /request`. The former Bright Data search adapter has been removed. Native indexed search and cited research remain separate capabilities. This page keeps its existing URL to help older clients migrate. ## Actual Google/Bing results Change the retrieval URL and bearer key to the Vitna search service. For a new client, use the explicit SERP contract: ```bash curl http://127.0.0.1:8789/v1/serp \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"engine":"google","query":"lunar science","country":"us","language":"en","maxResults":5,"format":"json"}' ``` For a supported URL-based client, use the compatibility alias: ```bash curl http://127.0.0.1:8789/request \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"url":"https://www.google.com/search?q=lunar+science&hl=en&gl=us","format":"json"}' ``` Both return actual search-engine organic listings in `organic`, with `rank`, `title`, and `link`, plus optional snippets and display links. The independent SERP route needs no Bright Data key, zone, or API call. It does not require a local index or native crawl, and it does not use indexed pages as an empty-result fallback. This alpha supports a bounded organic text-search contract. Ads, shopping, maps, images, news, knowledge panels, asynchronous vendor jobs, and the full vendor output are not implemented. Country and language parameters are localization hints; they do not prove the geographic location of egress. Challenges and unrecognized pages return explicit failures. Review the [SERP API contract](serp.md) for supported URL parameters, formats, and operator transport setup. Former `POST /v1/search` requests with `mode: "live"` or `mode: "hybrid"` still return HTTP 400 with `error.code: "invalid_request"`. Move those actual-SERP requests to `POST /v1/serp` rather than changing them to an indexed query. ## Vitna's indexed corpus Send `mode: "index"` for native retrieval and omit legacy `provider` and `fallback`. Search returns ranked indexed pages with source identity, observed crawl timestamps, and index provenance. It does not return a Google result list or fetch pages inline. For bounded coverage expansion, add `expand: true`. Add `discover: true` to examine selected sites' homepages and sitemaps; domains are optional guidance. Use `discover: false` to refresh known or supplied targets only. Name and topic searches can also admit automatic discovery when configured. ```bash curl https://search.vitna.ai/v1/search \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"query":"lunar science","mode":"index","expand":true,"discover":true,"domains":["nasa.gov"]}' ``` Expansion returns a separate private job. Poll `GET /v1/search/jobs/<id>` with the same client's key, then repeat the query after completion to use newly indexed content. Completion means the bounded pass finished; inspect indexed, unchanged, failed, and unmet target counts for actual coverage. `POST /v1/research` also accepts `expand` and `discover`. It assembles cited verbatim passages from matched indexed pages. It accepts no mode or paid provider fields. See the [API contract](overview.md) for search filters, passage budgets, quotas, and response fields. Research now selects from bounded candidate metadata before reading and rechecks each extracted page's URL and title. A small PostgreSQL, Python, and Node.js registry identifies known primary domains and canonical version paths. When the query identifies one requested version, explicit incompatible versions are excluded; unknown or mixed versions remain eligible. Version evidence comes from recognized paths or subject-specific titles, not snippet numbers. Documentation queries prefer documentation metadata, known primary domains, and compatible versions, while equal-priority candidates retain their input order. This selection policy does not verify content facts or establish complete coverage. ## Update the deployment Remove former Bright Data search credentials and search adapter settings from the service's private environment. Enable independent SERP retrieval with `VITNA_SEARCH_SERP_ENABLED=true` and `VITNA_SERP_RESALE_APPROVED=true`. The second setting now records the operator's decision that collecting Google and Bing results and redistributing them through Vitna's API is permitted. A value kept from the Bright Data adapter confirmed a reseller/OEM agreement for that vendor's service, not this decision, so decide again before keeping it. Startup refuses SERP retrieval enabled without this confirmation. `auto` is the default transport. It can try direct HTTPS followed by browser retrieval, while configured proxies restrict attempts to pinned browser routes. The browser transport uses Vitna Unblocker's separately installed runtime. Distinct attempts, route concurrency, cooldowns, and a short JSON cache bound retrieval work. Send `fresh: true` for a new capture. It has no CAPTCHA solver. The same search client registry and request quotas apply to SERP and native requests. See [SERP setup](serp.md#run-without-an-index). Native retrieval needs an initialized `VITNA_SEARCH_INDEX` and `VITNA_SEARCH_PROVIDERS=vitna`, with dedicated client hashes. Query-guided expansion also needs the separate frontier and demand worker. Site discovery needs a shared private discovery key and the relevant local catalogs. Starting the API does not start those workers. Follow [focused index setup](self-hosted.md) or [general web coverage expansion](general-web.md#query-guided-coverage-expansion) for crawler setup, budgets, and timers. The optional Brave, Exa, and Tavily gateway remains available only through separately configured provider keys and routing; it is separate from native crawling and can incur provider charges. --- <!-- https://vitna.ai/docs/search/self-hosted.md --> # Run a focused search index Vitna can crawl selected public AI documentation, keep an independent keyword index, and serve it through the search API. An OVHcloud Public Cloud instance can run the service and crawler; Cloudflare R2 can hold standalone snapshots. The deployment files are prepared locally. No cloud resources, DNS records, R2 uploads, or credit spending have been created by this implementation. For cross-site discovery, persistent crawl budgets, API quotas, and the Virginia deployment package, use the [general web beta](general-web.md). ## Try it locally ```bash node platform/search/demo.mjs ``` Open `http://127.0.0.1:8789`. The page connects automatically using a temporary demo-only key. The four illustrative pages pass through the real crawler, SQLite index, and HTTP API using an in-memory fetch fixture. Nothing is downloaded, no provider credential is required, and no price claim is implied. The demo binds to loopback and loses its index and key when stopped. The repository also provides these script aliases. The crawl, snapshot, and backup commands read the settings from their process environment: ```bash npm run search:demo npm run search:crawl -- --source exa npm run search:snapshot npm run search:backup ``` For a command-line check that exits after verifying a search: ```bash node platform/search/demo.mjs --check ``` ## Crawl and serve real pages Review `platform/search/sources.json` or create a private catalog and set `VITNA_SEARCH_CATALOG` to its path. The starter catalog contains the official Cloudflare Workers AI/R2, Exa, and Tavily documentation. It is a crawl scope, not a downloaded corpus or a promise that those sites permit this crawler. Generate a key with `node platform/search/mint-key.mjs pilot`. Keep the printed token for the API client and put only its hash into a private `.env.search`: ```dotenv VITNA_SEARCH_INDEX=.search-data/index.sqlite VITNA_SEARCH_PROVIDERS=vitna VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"} ``` ```bash node --env-file=.env.search platform/search/crawl-run.mjs node --env-file=.env.search platform/search/start.mjs ``` The first command performs a live, bounded crawl. The second starts the browser interface and authenticated API on `127.0.0.1:8789`. Paste the token into the interface, connect, and search. No paid search-engine API account is needed. New clients should send `mode: "index"` explicitly; the browser selects Index by default. A valid empty Index search does not call an external engine. Cloud credits cover eligible infrastructure according to the account's offer. This focused catalog controls its own crawl scope. For broader query-guided coverage, the [general web beta](general-web.md#query-guided-coverage-expansion) can automatically queue discovery after an empty name/topic query or weak partial-term topic matches, while preserving current search results. A plain `{"query":"examplebrand","mode":"index"}` needs no domain input; domains are optional guidance. Automatic discovery uses observed names; explicit expansion can also use matching known URLs. Narrow ASCII brand candidates can use `.com`, `.ai`, and `.org`. Candidate hosts are not verified official identities. Name detection is heuristic; common topic words and multiword topic queries generate no guessed hosts. Broad queries can instead select known publisher domains by matching catalog descriptions. Those domains are discovery candidates, not indexed coverage. An optional separate local English Wikipedia title catalog can suggest concept article URLs; titles, including redirect titles, are metadata without canonical-target mappings. Article text becomes usable only after a permitted page is fetched and indexed. Queries without a suitable name, topic, or concept match can return `no_candidates`. Explicit `expand: false` or `discover: false` disables automatic discovery; `expand: true, discover: false` refreshes known targets only. Explicit `expand: true, discover: true` requests site discovery. It examines up to three relevant sites and 2,000 sitemap candidates, then attempts at most eight target pages within 40 HTTP requests and 60 seconds of crawler work. It requires a private discovery key and does not promise whole-web coverage. Current results return first; poll the private job, then search again after completion to retrieve any newly indexed pages. Use `POST /v1/extract` with a returned URL to read its already indexed cleaned text. It shares the search key and quotas, returns source/crawl metadata, and performs no live fetch or JavaScript execution. The default text limit is 20,000 characters, up to 100,000; Markdown structure is not guaranteed. An absent or policy-unavailable page returns HTTP 404. See [indexed text extraction](overview.md#indexed-text-extraction). `POST /v1/research` assembles Source brief from cited verbatim passages in matched indexed pages. It defaults to three sources and an 8,000-code-point quoted passage budget, capped at five sources and 16,000 code points. It uses actual page titles and reports bounded evidence coverage; it does not generate an explanation or claim an exhaustive review. See [Source brief research](overview.md#source-brief-research). For the general web deployment, initialize the topic catalog explicitly with the existing web `VITNA_SEARCH_FRONTIER` and `VITNA_SEARCH_INDEX` settings in the process environment: ```bash npm run search:topics -- --init npm run search:topics -- --review-seeds npm run search:topics -- --backfill ``` These commands make no network requests. They initialize storage, record 28 curated publisher descriptors, and inspect at most 1,000 eligible already indexed homepages. The optional dataset download is a separate command: ```bash npm run search:topics -- --import-majestic --limit 10000 ``` Imported domain candidates remain unverified metadata, separate from publisher descriptions and indexed pages. None of these commands register crawl origins, enqueue pages, or start crawling. See [topic catalog setup](general-web.md#initialize-the-topic-catalog) for default files, download bounds, the 24-hour cooldown, and source attribution. For optional concept discovery, point `VITNA_SEARCH_CONCEPTS` to a separate SQLite file shared by the API and demand worker. Web mode defaults to `.search-data/web/concepts.sqlite`. Import a selected published title snapshot explicitly; startup does not download it: ```bash npm run search:concepts -- --import-wikipedia --snapshot 20261003 ``` This is title metadata, including redirect names without target mappings, not article content. See [concept snapshot import](general-web.md#import-a-concept-title-snapshot) for source selection, disk and download bounds, and attribution. `/healthz` checks the HTTP process. `/readyz` returns 200 only when the default local provider has indexed content. `/v1/status` requires a search key and reports source counts, crawl freshness, and the last run. See the [API contract](overview.md) for request limits, authentication, and fallback. ## Catalog and crawl boundaries The operator controls the catalog. There is no public crawl/import endpoint. Each source specifies a unique `id`, display `name`, HTTPS `origin`, directory `pathPrefixes` ending in `/`, and `seeds` inside those prefixes. Sources can be disabled with `enabled: false`; dropping, disabling, or narrowing a source removes its excluded documents the next time the crawler synchronizes it. | Source setting | Default | Limit | |---|---:|---| | `maxPages` | 50 | Page attempts per source per run; 1–10,000 | | `maxDocuments` | 5,000 | Indexed documents per source; 1–100,000 | | `maxDepth` | 2 | Discovered-link depth, 0–5 | | `delayMs` | 1,000 | Minimum time between request starts to an origin, 1,000–60,000 | | `refreshHours` | 24 | Minimum interval before downloading a known page again, 1–8,760 | Run only one source with `search:crawl -- --source exa`, or add `--force` to refresh known pages before their interval. Force does not override robots, scope, byte limits, document caps, or the global request budget. | Crawler environment | Default | Meaning | |---|---:|---| | `VITNA_SEARCH_INDEX` | `.search-data/index.sqlite` | Writable standalone search file | | `VITNA_SEARCH_CATALOG` | Bundled `sources.json` | Reviewed operator catalog | | `VITNA_SEARCH_CRAWL_REQUESTS` | 200 | All HTTP attempts, including robots and redirects | | `VITNA_SEARCH_CRAWL_DURATION_MS` | 900,000 | Maximum run duration; up to one hour | | `VITNA_SEARCH_CRAWL_TIMEOUT_MS` | 10,000 | Timeout per request; up to 60 seconds | The crawler identifies itself as `VitnaSearch`, checks robots.txt each run, merges matching agent groups, honors allow/disallow wildcard and end rules, and uses the larger of the source delay and robots `Crawl-delay`. A delay above 60 seconds skips that source. A missing robots file (404/410) permits the configured scope; failed, forbidden, oversized, or redirected-to-other-origin robots responses skip the source. Newly disallowed indexed pages are removed even if they are beyond the page budget. SERP page contents are read by a separate agent, [`VitnaSearch-User`](serp.md#vitnasearch-user), only when a request asks, and nothing it reads enters the index. Connections require public HTTPS hostnames on the standard port. DNS answers must all be public addresses; the validated address is pinned into the actual TLS connection. Redirects are checked again before a connection. Page redirects remain inside the source's origin and path prefixes. Credentials, IP literals, internal destinations, query strings, encoded path separators, and non-text/download links are refused. Pages are capped at 2 MiB and robots.txt at 512 KiB. Excessively nested HTML is refused. An instance that must send no Google traffic from its own addresses sets `VITNA_SEARCH_REFUSE_GOOGLE=true` (default `false`) for every search process: the API, the web crawl and the demand runs. Each public fetch then refuses Google's own hosts by name before any lookup (`crawl_google_host`), and any name whose DNS answer falls in Google's own serving ranges before any connection (`crawl_google_address`), which also covers a site on Google's front ends under its own domain. The ranges are Google's published `goog.json` less the Google Cloud customer ranges of `cloud.json`, so a third-party server on Google Cloud is still read; refresh them with `node scripts/google-ranges.mjs` on a workstation, never on the instance. A crawl source on Google then fails its jobs with those codes, and pages already indexed from it stay until removed. Any other value refuses to start. HTML, plain text, and Markdown are supported. Extraction prefers main/article content, removes scripts/navigation/hidden content, and indexes bounded plain text. Noindex pages and observed 404/410 pages are removed; nofollow suppresses link discovery. Canonical tags cannot redirect the crawler. ETag and Last-Modified validators support conditional refreshes. A 304 or unchanged extracted text advances crawl freshness without inventing a content-change date. A failed refresh retains the previous content and its old timestamp. The index ranks lexical matches with SQLite FTS5 BM25, weighting titles more than body text. It returns excerpts, source identity, `crawledAt`, and `contentChangedAt`, and a publication date only where the page declares one ([declared publication dates](overview.md#declared-publication-dates)); it never infers one. It does not execute JavaScript, parse PDFs, ingest arbitrary sitemaps, rerank with a model, or generate answers. Embedding pages is optional: with Vitna's own embedding engine configured, [semantic retrieval](overview.md#semantic-retrieval) adds semantic and hybrid ranking from a separate vectors file. In this single-process mode the vector scan runs on the API's own thread rather than in query workers. Raw HTML is not archived by this version; the index stores extracted text, validators, and discovered in-scope links. One index lease prevents overlapping crawler runs, with expiry to recover after an interruption. Writes and FTS updates are atomic; WAL permits the read-only API connection to coexist with a crawler. Recent run summaries are kept in the index, with source counts and error codes. No customer search queries, client tokens, signing keys, or site database data enter this file. ## OVHcloud deployment package Use a single Linux Public Cloud instance with Node 24, persistent local/block storage, nginx, and rclone if R2 backups are enabled. A 2-vCPU instance with 4–8 GB RAM is a starting configuration to benchmark, not a throughput guarantee. Credit coverage and expiry come from the account's actual offer; no specific OVH instance price or credit balance is assumed here. Keep the checkout under `/opt/vitna-search`, create an unprivileged `vitna-search` account, and give that account ownership of `/var/lib/vitna-search`. Code stays owned by the deployment administrator. There is no web build or npm install needed for these standalone commands. Copy and fill the three environment templates into `/etc/vitna-search`: - `deploy/vitna-search.env.example` → `api.env`: hashed clients and local-index mode. - `deploy/vitna-search-crawl.env.example` → `crawl.env`: reviewed scope and request budgets. - `deploy/vitna-search-backup.env.example` → `backup.env`: snapshot directory and dedicated R2 remote. Use root ownership, the `vitna-search` group, and mode 0640 for those files. The API receives client hashes; the crawler receives no API keys; the backup process alone receives the path to its R2 config. None needs the website's database environment or any signing credential. Install `deploy/vitna-search.service`, `vitna-search-crawl.service`, `vitna-search-crawl.timer`, `vitna-search-backup.service`, and `vitna-search-backup.timer` under `/etc/systemd/system`. After reviewing the scope and activating credit-covered resources, the operator can run: ```bash sudo systemctl daemon-reload sudo systemctl start vitna-search-crawl.service sudo systemctl enable --now vitna-search.service vitna-search-crawl.timer ``` Check readiness and the first crawl summary before exposing the service: ```bash curl --fail http://127.0.0.1:8789/readyz sudo journalctl -u vitna-search-crawl -u vitna-search --since today ``` Configure DNS, a valid TLS certificate, and nginx using `deploy/vitna-search.nginx.conf.example`. The example host is `search.vitna.ai`; it is not a provisioned hostname. The API remains bound to loopback. The hourly timer checks freshness; the catalog's default refresh interval is daily. Units bound CPU/memory and leave only the search state directory writable. An alternative standalone image is available with: ```bash docker build -f deploy/vitna-search.Dockerfile -t vitna-search . docker volume create vitna-search-data docker run --rm -v vitna-search-data:/data vitna-search node platform/search/crawl-run.mjs docker run --rm --env-file .env.search -p 127.0.0.1:8789:8789 -v vitna-search-data:/data vitna-search ``` For the container environment file, use `VITNA_SEARCH_INDEX=/data/index.sqlite` and `VITNA_SEARCH_PROVIDERS=vitna`. API and crawler share the writable volume for SQLite WAL coordination; the API opens the database read-only. The image contains the API, crawler, and local snapshot command. Rclone uploads use the host backup unit, or a separate backup image with rclone installed. Linux systemd/nginx/Docker execution has not been verified by the local Windows run. ## R2 snapshots and restore Create a dedicated private `vitna-search` bucket and a bucket-scoped Object Read & Write credential. Fill `deploy/vitna-search-rclone.conf.example` into `/etc/vitna-search/rclone.conf`, readable only by root and `vitna-search`. Use the account endpoint without appending the bucket name. Actual keys stay outside the checkout and outside the API environment. `search:snapshot` creates a consistent standalone SQLite file using `VACUUM INTO`. It works while the service is up and refuses to overwrite an existing destination. Copying the live file directly can miss WAL changes. ```bash node --env-file=.env.search platform/search/snapshot-run.mjs sudo systemctl start vitna-search-backup.service sudo systemctl enable --now vitna-search-backup.timer ``` The R2 command writes to a unique `.partial` name, finalizes the snapshot, then uses `rclone copyto`. A failed upload keeps the complete local snapshot. After successful uploads, 14 completed snapshots are retained locally by default; only generated snapshot names inside the configured directory are eligible for pruning. Remote objects are never deleted by the command. Set an R2 lifecycle policy for the dedicated backup prefix to bound remote storage as the index grows. Failed uploads require monitoring because local retention does not run until an upload succeeds. For restore, stop the API and crawl timer, download a selected snapshot with `rclone copyto` to a new filename, and check it with `openSearchIndex` in read-only mode. Point the API and crawler to that file, restore the search account's ownership, and restart. Keep the old index for rollback. Do not replace a live SQLite file or mix a restored file with old WAL/SHM sidecars. If a snapshot contains an active crawl lease, it expires within the configured run duration plus timeout and one minute before a new crawl can acquire it. ## Validation and remaining scope The offline tests cover persistence, ranking, freshness, source removal, robots behavior, scope/DNS/redirect restrictions, crawl caps, conditional GET, deletion/noindex, cancellation, local authentication, and snapshot restore. The backup test uses an injected upload operation; it does not contact R2. Live source accessibility, account credit eligibility, real uploads, Linux service installation, corpus size, and query throughput require validation on the configured cloud instance. In-memory client quotas remain an alpha limit; public signup, durable billing metering, and multi-instance serving are future work described in the [API overview](overview.md). Protocol references: [Node SQLite](https://nodejs.org/api/sqlite.html), [SQLite FTS5](https://www.sqlite.org/fts5.html), [robots exclusion](https://www.rfc-editor.org/rfc/rfc9309.html), and [Cloudflare R2 with rclone](https://developers.cloudflare.com/r2/examples/rclone/). --- <!-- https://vitna.ai/docs/search/serp.md --> # Google and Bing SERP retrieval Vitna's SERP API retrieves actual Google or Bing search result pages through Vitna's own transport and parser. It returns parsed organic and rich SERP features (or the fetched HTML) without calling Bright Data, supporting web, news, images, videos, and shopping verticals alongside SerpApi-compatible query aliases and output shapes. ## Request ```text POST /v1/serp GET /v1/serp POST /v1/serp/bing POST /request ``` `POST /v1/serp` and `POST /request` accept the same JSON request; `GET /v1/serp` accepts the scalar fields below as query-string parameters for drop-in SerpApi / Serper compatibility; `POST /v1/serp/bing` defaults `engine` to `bing` and serves Bing SERP requests only. Send a dedicated `vitna_search_` bearer key, or your org's Vitna key, `vitna_key_` (see [one key](../one-key.md)). `/request` supports the URL-based request shape used by Bright Data's SERP API without forwarding the request to Bright Data. ```bash curl http://127.0.0.1:8789/v1/serp \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"engine":"google","query":"lunar science","country":"us","language":"en","page":1,"maxResults":5,"format":"json"}' ``` | Field | Default | Allowed values | |---|---|---| | `engine` | `google` (`bing` on `/v1/serp/bing`) | `google` or `bing` (`bing` only on `/v1/serp/bing`) | | `query` (or `q`) | Required | 1 to 1,000 characters, with no control characters | | `country` (or `gl`) | Unset; search-engine default | Two-letter country hint; normalized to lowercase | | `language` (or `hl`) | Unset; search-engine default | Language tag, such as `en` or `en-US`, at most 32 characters | | `page` | 1 when `offset` is omitted | Integer from 1 to 20; selects ten-result offsets; cannot accompany `offset` | | `offset` (or `start`) | Derived from `page` | Integer from 0 to 190; zero-based engine result offset; cannot accompany `page` | | `maxResults` (or `num`) | 5 (10 when `url` omits `num`/`count`) | Integer from 1 to 30 | | `type` / `tbm` | `web` | `web`, `news` (`nws`), `images` (`isch`), `videos` (`vid`), or `shopping` (`shop`) | | `timeRange` / `tbs` | Unset | `hour`, `day`, `week`, `month`, or `year` (or raw Google `tbs` filter such as `qdr:w`) | | `location` / `uule` | Unset | Canonical location string (encoded deterministically to UULE) or raw Google `uule` token; mutually exclusive, Google only | | `googleDomain` (or `google_domain`) | `google.com` | One of 25 supported regional Google domains (such as `google.co.uk`, `google.de`, `google.co.jp`), Google only | | `lr`, `cr` | Unset | Google language (`lang_en`) and country (`countryUS`) restrictions, up to five pipe-separated values | | `device` | `desktop` | `desktop` or `mobile` | | `safe` | Unset | `active` or `off` (maps to `safe` on Google and `adlt` on Bing) | | `nfpr` | Unset | Boolean, `"0"`, `"1"`, `0`, or `1`, Google only; disables query spell-correction when truthy | | `udm` | Unset | `"2"`, `"7"`, `"12"`, `"14"`, or `"28"`, Google only | | `linkResolution` | `complete` | `complete` or `fast` (skips interactive `/goto` browser unlocking for low-latency organic runs), Google only | | `format` | `json` | `json` for parsed results or `html` (or `raw`) for the fetched HTML | | `fresh` | `false` | Boolean; bypass a stored JSON response and retrieve a new engine page | | `extras` | `false` | Boolean; when true with `format: "json"`, extracts People Also Ask (plus `people_also_ask_details`), related searches, ads, answer box, knowledge graph, AI overview, top stories, local results, inline videos/images, and forum discussions | | `includeOmitted` | `false` | Boolean, Google only; when Google says it left out entries very similar to those shown, the search runs again with Google's `filter=0` and answers with the fuller page (see [Google's notices](#googles-notices)) | | `filter` | Unset | `"0"`, Google only; asks Google for every entry, the very similar ones included, in the one capture | | `keepDoorways` | `false` | Boolean; when true, `organic` keeps the doorway pages Vitna otherwise leaves out (see [Doorway pages](#doorway-pages)) | | `googleShareMs` | Operator default | Integer from 1,000 to 60,000, Google only; caps how long owned Google may spend on this request before timing out or handing off to Bing, never exceeding the operator's `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` | | `fallbackEngine` | Operator default | `"bing"`, Google only; opts this request into Bing fallback when `VITNA_SEARCH_SERP_FALLBACK_ENGINE` is unset | | `shape` | `vitna` | `vitna` for the fields under [Parsed response](#parsed-response), or `serpapi` for SerpApi's names, built from the same result (see [SerpApi-shaped response](#serpapi-shaped-response)); JSON only, so not with `format: "html"`, and `serpapi` not with `data_format` or `brd_json` | | `contents` | Unset | `{ "highlights": true }`, `{ "excerpts": true }` and/or `{ "text": true }`, each `true` or an object with `maxCharacters` (highlights and excerpts also take `query`): read each result page and attach verbatim highlights, excerpts, or its text (see [Page contents](#page-contents)); Vitna's JSON shape only, and only where the operator enables it | The country and language settings become search-engine URL parameters. `country` is a requested localization hint, not a verified proxy location. The operator controls the outbound network and optional proxies. The caller cannot select a proxy, upstream endpoint, headers, browser profile, or credentials. Page numbers or offsets select one result page per request; `maxResults` limits parsed organic entries, not the search engine's total result count. When `extras: true` is requested, the parsed JSON response also supplies `people_also_ask` (the questions listed under People Also Ask, without their answers), `related_searches` (the related queries), and `ads` (sponsored listings with `position`, `title` and `link`, and `snippet` and `display_link` where shown), without altering organic ranking. Each is omitted when the page has none. Google lists up to eight related searches, fewer or none for some queries, and the list holds what the page showed. Unknown fields are refused, and request bodies are limited to 8,192 bytes. Whitespace around the query is removed before retrieval, and a run of whitespace inside it becomes one space, so queries that differ only in spacing share a cached answer and a capture in progress. Letter case is kept, since Google reads `OR` and `AND` as operators only in capitals. ## Page contents With `contents`, each organic result gains a `contents` object read from the result's page, so an agent can check an answer without fetching every link. `highlights` are up to three sentences from the page that hold the query's terms, best first, quoted exactly, with their offsets in Unicode code points; search operators (`site:`, `-term`, `OR`, quotes) never choose one, and `contents.highlights.query` replaces the query for choosing them. `excerpts` are passages of whole sentences, up to 500 characters each, that best answer the query: as many as `maxCharacters` holds (1,000, two passages, by default; at most 4,000), best first, quoted exactly with their offsets, and `contents.excerpts.query` replaces the query for choosing them (see [Excerpts](#excerpts)). `text` is the page's main text, clipped to `maxCharacters`, with `characters`, `truncated` and `totalCharacters`. Limits match [indexed search's inline contents](overview.md#inline-contents): highlights default to 1,000 characters (at most 4,000), text to 20,000 (at most 100,000). ```json { "query": "\"Lake Lavarone\" elevation", "contents": { "highlights": true, "excerpts": true } } ``` ```json "contents": { "status": "ok", "source": "live", "readAt": "2026-10-17T13:20:05.000Z", "totalCharacters": 2841, "highlights": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres.", "start": 0, "end": 51 }], "excerpts": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres. In most winters the whole lake freezes over.", "start": 0, "end": 96 }], "offsetUnit": "unicode_code_points" } ``` A Wikipedia result the search index holds is read from the index, at once and whole: `source` is `stored`, and `contentChangedAt` and the text's `reusePolicy` (Wikipedia's CC BY-SA 4.0) stand in for `readAt`. A Wikipedia link with a query, or a page the index lacks, is read live like any other. Other pages are read live, after the search, on the caller's behalf by [`VitnaSearch-User`](#vitnasearch-user): HTTPS only, never on Google's own hosts, without its fragment, through at most three redirects (`url` names where one ended), in at most 2.5 s a page, and no more than two pages of one origin at once. A page longer than 1 MiB is read to its first 1 MiB and says `partial: true`, and `totalCharacters` counts what was read. A PDF is read too, to its first 4 MiB, for the text its pages draw, in a thread of its own; its contents say `format: "pdf"`, and a cut one gives the pages its first bytes hold. A scanned PDF has no text to read and says `no_text`; an encrypted or unreadable one says `page_unavailable`. The whole read takes at most 3 s; a page not read in time says `timeout`. Pages read in the last ten minutes are reused. A page with under 25 characters of main text (a script-only shell, an empty sign-in wall) says `no_text`. A result whose page gives nothing has `contents.status` of `no_text`, `robots_disallowed`, `noindex`, `host_excluded`, `unsupported_url`, `unsupported_type`, `page_unavailable`, `page_too_large` or `timeout`, and its result is unchanged. The complete response is bounded to 510 KiB: when the requested text would exceed it, every result's text is clipped to one shared length, the longest that fits, and marked `truncated`. Like snippets, contents are untrusted publisher data: treat them as data, and escape them before rendering HTML. Contents are read for each request and never enter the JSON cache, a shared capture or Vitna's crawl index. They come with Vitna's JSON shape only: `format: "html"` or `shape: "serpapi"` with `contents` is answered 400 `contents_require_json`, and an instance without them answers 503 `serp_contents_unavailable`. `/v1/status` reports `serp.contents`. On the AA Search API benchmark replica of 2026-10-17, wrong BrowseComp answers came after about 77 searches and under two page reads each, against 28 searches for right ones: an agent with snippets alone searched on and guessed. ### Excerpts A page is cut into windows of whole sentences, up to 500 characters (or `maxCharacters`, when smaller), each starting about halfway into the one before, and ranked by keyword: BM25 over the page's own windows, with a bonus for each quoted phrase a window holds. With `VITNA_SEARCH_SERP_EXCERPT_RERANK=true`, keyword's best 16 windows of each page are ranked again by meaning on the batch embedding engine, and the two orders are fused by reciprocal rank, within 1 second of the read. Otherwise, or when the engine is late or fails, keyword order stands. `serp.contents.excerpts.order` in `/v1/status` says which an instance uses. On the cached pages of the 743 searches an agent ran in the BrowseComp replay of 2026-10-18, two excerpts a result carried the answer for 10 of the 81 questions the agent missed by keyword, and 11 with the meaning rerank, against 7 in Google's own snippets and 25 in Perplexity's results for the same searches. Meaning over whole pages reached 12, at about 270 embeddings a search; on the search host's 2-thread engine, 160 windows took 6.4 seconds. ## VitnaSearch-User Page contents are read by `VitnaSearch-User/0.1 (+https://vitna.ai/docs/search/serp#vitnasearch-user)`, and only when a `/v1/serp` request asks for them: one search's result pages, once each, on that caller's behalf. It does not crawl, follow links, or keep what it reads beyond a ten-minute cache, and nothing it reads enters Vitna's index. Because each read is one a caller asked for, the rules a `robots.txt` gives every crawler (`User-agent: *`) do not apply to it. These still do: - a `robots.txt` group naming `VitnaSearch-User`, in full (`Disallow`, `Allow`, `Crawl-delay`), or, when no group names it, one naming Vitna's crawler, `VitnaSearch`; - a `Content-Signal` of `search=no` in any group that speaks to the reader: its own, the crawler's, or every crawler's (`User-agent: *`); - a page's `noindex`, by `X-Robots-Tag` or by a meta tag named `robots`, `VitnaSearch` or `VitnaSearch-User`. The rules are checked again before a page read in the last ten minutes is reused. A `robots.txt` that cannot be read shows no group naming the reader, so only a page's own signals apply, and it is asked again after five minutes. One that redirects is followed to any public HTTPS host, and its rules speak for the origin asked. Google's own hosts are never read, at any hop, robots.txt included: not by name, and not at an address Google serves its own services from, which also covers a site on Google's front ends under its own domain. To keep the reader off a site: ```text User-agent: VitnaSearch-User Disallow: / ``` ## URL-based compatibility A URL-based client can change its API base URL and bearer key while retaining a supported search URL: ```bash curl http://127.0.0.1:8789/request \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"url":"https://www.google.com/search?q=lunar+science&hl=en&gl=us","format":"json"}' ``` The URL must use a supported Google host (`https://www.google.com/search` or one of the 25 supported regional `www.google.*` domains) or Bing vertical path (`https://www.bing.com/search`, `/news/search`, `/images/search`, `/videos/search`, or `/shop`), without credentials, a custom port, or a fragment. It is limited to 4,096 characters. Vitna validates the parameters and builds a normalized engine request itself. Arbitrary URLs and unsupported search verticals are refused. A URL-based request must not also supply any structured search field: `engine`, `query`, `country`, `language`, `page`, `offset`, `maxResults`, or `googleDomain`. | Engine | Supported URL parameters | |---|---| | Google | `q`, `hl`, `gl`, `start`, `num`, `udm`, `filter`, `tbs`, `tbm`, `uule`, `lr`, `cr`, `safe`, `nfpr`, `brd_json` | | Bing | `q`, `setlang`, `cc`, `first`, `count`, `filters`, `adlt`, `brd_json` | Unknown or repeated URL parameters are refused. Google `start` accepts integers from 0 to 190; Bing `first` accepts integers from 1 to 191. Google `udm` accepts `2`, `7`, `12`, `14`, or `28`, and `filter` accepts `0`. Google `num` and Bing `count` must be from 1 to 30. Defaults select the first page with at most ten parsed results. The optional legacy `zone` field accepts 1 to 128 ASCII letters, numbers, underscores, or hyphens and is ignored. It does not select a vendor zone or proxy. `format: "raw"` selects HTML unless `data_format: "parsed_light"` or the URL parameter `brd_json=1` requests parsed JSON. `parsed_light` is the only supported `data_format`; `brd_json` accepts only `1`. Explicit `format: "html"` cannot accompany either parsed-JSON selector. For example, this supported migration request returns parsed organic JSON: ```json { "zone": "former_serp_zone", "url": "https://www.google.com/search?q=lunar+science&hl=en&gl=us", "format": "raw", "data_format": "parsed_light" } ``` This compatibility surface is a migration aid, not full compatibility with every vendor request or output field. Bright Data documents its broader `/request` shape and parsed response in its [SERP API reference](https://docs.brightdata.com/api-reference/rest-api/serp/serp-api). Clients that use additional fields or depend on ads, maps, shopping, news, images, knowledge panels, or vendor-specific ranks must adapt to Vitna's organic-only contract. Google's Custom Search JSON API is closed to new customers and is scheduled to end on January 1, 2027. Existing customers need a migration plan; it is not an available new-customer provider for this replacement. See Google's [Custom Search JSON API notice](https://developers.google.com/custom-search/v1/overview). ## Parsed response An illustrative JSON response: ```json { "general": { "search_engine": "google", "query": "lunar science", "page_title": "lunar science - Google Search", "timestamp": "2026-10-04T12:00:00.000Z", "requested_country": "us", "requested_language": "en", "search_type": "text", "mobile": false }, "input": { "original_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5", "final_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5" }, "organic": [ { "rank": 1, "title": "An example lunar science page", "link": "https://example.com/lunar-science", "description": "An example excerpt from a search result.", "display_link": "example.com" } ], "meta": { "provider": "vitna_serp", "engine": "google", "requestId": "example-request", "latencyMs": 500, "transport": "direct", "coverage": "organic_only", "parserVersion": "1", "provenance": "search_engine", "offset": 0, "page": 1, "count": 1, "retrieval": { "route": "direct", "attempts": [ { "route": "direct", "transport": "direct", "outcome": "success", "latencyMs": 500 } ], "cache": { "hit": false, "ageMs": 0 } } } } ``` `organic` follows the recognized organic listing order on the fetched page. `rank` is the entry's position among recognized organic listings on that page. Rejected unsafe URLs and removed [doorway pages](#doorway-pages) can leave rank gaps, and separate listings that link to the same URL retain their observed positions. Vitna does not invent a whole-page `global_rank`, total result count, publication date, or verified location. `description` is search-engine snippet text. It is not a quote retrieved from the linked page. Missing descriptions and display links are omitted. `display_link` selects an observed publisher hostname or breadcrumb. Where Google's citation for a result holds only counts and an age, as on forum, video and community results ("20+ comments · 4 months ago"), it is the source line Google shows above the title instead, such as `Reddit · r/buildapc` or `YouTube · Valve`. Dates, ages and view or comment counts are never display links. A listing can include observed `sitelinks`; they are associated with the parent result and do not receive invented organic ranks. `timestamp` is the transport's fetched-at time when available, otherwise the response assembly time. `page_title` comes from the fetched page or is null when unavailable. `requested_country` and `requested_language` appear only when requested. `input.original_url` is the normalized outbound search URL, including generated pagination and result-limit parameters. It is not a verbatim echo of a supplied URL. `input.final_url` is the observed URL after retrieval and permitted redirects. The observed final URL must retain the requested engine, query, and offset. A changed query or offset returns an explicit `serp_query_mismatch` error. `meta.transport` reports the transport that captured the result; `meta.parserVersion` identifies the parser implementation. `meta.coverage: "organic_only"` remains present even when the page contains other features. `meta.offset` records the zero-based requested offset. `meta.page` appears only when that offset is a multiple of ten and identifies its ten-result window. `meta.count` is the returned organic entry count. `meta.doorwaysRemoved` appears when Vitna left doorway pages out, and counts them. When the page contains a recognized forward navigation control, `pagination.next_page_link` carries a normalized continuation URL derived from that control. It preserves the observed Google `start` or Bing `first` offset, validates supported search fields, and removes engine tracking parameters. If the control omits `num` or `count`, the continuation defaults to at most ten parsed results. `pagination.current_page` appears only when the requested offset maps to a ten-result window. Vitna does not invent a next-page URL when no control is observed. Submit that continuation as `url` to `POST /v1/serp` or `POST /request`. A structured client can instead use the observed Google `start` as `offset`, or Bing `first` minus one. A recognized no-results page may return an empty `organic` list. A challenge, consent interstitial, unknown page layout, or invalid upstream response is an explicit failure. It is never represented as a successful empty search. A page with accepted organic results, or a recognized no-results page, is a results page. Query text that the engine echoes in its title, suggestions, related searches, or no-results notice does not make it a challenge. Challenge elements, consent or sign-in forms, and consent, sign-in, or `/sorry` landings still fail as blocks. Google's toolbar zero count is read in any interface language, and its no-results notice is recognized in English, German, and French. Bing's no-results notice is recognized in English only. Google descriptions support known snippet classes and the `data-sncf` or `data-content-feature="1"` markers inside recognized organic wrappers. These layout checks improve extraction after retrieval; they do not establish that Google will accept an outbound route. For `format: "html"`, the successful response body is the fetched HTML instead of the parsed envelope, with `Content-Type: text/html` and an `X-Request-Id` header. It uses the same page recognition and challenge checks as JSON output. Treat HTML, titles, and descriptions as untrusted external content. Escape text in a UI and keep raw HTML out of an application document's trusted execution context. ### Google's notices `search_information` appears when Google's page carries one of two notices about its own result list, and is left out otherwise. Neither notice changes `organic`. ```json "search_information": { "omitted_results": { "shown": 1, "included": true, "before": 1 }, "quotes_dropped": { "no_results_for": "\"acme widget\" site:example.com", "results_for": "acme widget site:example.com" } } ``` `omitted_results` is Google's note under the results that it "omitted some entries very similar to the 1 already displayed". It is recognized by its `filter=0` link, so it reads alike in any interface language. `shown` is the count Google states, or null when the notice holds no single number. Without `includeOmitted` the notice is reported and nothing more is fetched. With `includeOmitted: true`, Vitna fetches the page again with Google's `filter=0` inside the same request's time and answers with it when it holds more results, doorway pages aside: `included: true`, `before` the first page's count, and an `input.final_url` that carries `filter=0` while `input.original_url` stays the first request. Otherwise `included: false` and `reason` says why: `no_more_results` when the unfiltered page held no more, `serp_timeout` when the repeat ran out of time (it runs for at most 17 seconds, ends 2 seconds before the attempt's deadline, and is skipped when that would leave it under 3 seconds), or the repeat's own error code, such as `serp_blocked`. The first page is answered in each of these cases, at the latest when the repeat's time is up; only the caller's cancellation or the attempt's own deadline ends the search. The repeat is a second engine capture within one request, and its answer is cached apart from a request without `includeOmitted`. Google showed this notice on 3 of 30 one-result `site:` searches measured on 2026-10-17. `quotes_dropped` is Google's note above the results that a quoted query found nothing, so it searched the same words without their quotes: `no_results_for` is the query as Google echoed it and `results_for` the query it ran. The organic results are for `results_for`, not for the exact phrase. It is read on English result pages only. ### Doorway pages Some sites, most of them hacked small-business sites, publish a page for each query searched, titled with the query's own words in lowercase at an address that carries none of them. Google ranks them for long, specific queries, the kind research agents write. Vitna leaves an entry out of `organic` when its title has four or more words, every letter a lowercase English letter, and no word of four letters or more that its link repeats. A PDF, or a page on a `www.` host, is always kept. Entries are removed before `maxResults` applies, the rest keep the engine's `rank`, and `meta.doorwaysRemoved` counts the removed entries. Multi-page jobs leave them out too. `keepDoorways: true` answers the page as the engine listed it, and that answer is cached apart. On 6,575 Google results for 743 searches a research agent ran on 2026-10-18, the rule removed 1,079 (16%). One was a real page, a journal article with a lowercase title. Each of the 33 searches whose results carried the answer the agent was after still had it in a result kept. ## SerpApi-shaped response Code written against SerpApi's Google results can ask for SerpApi's field names with `shape: "serpapi"`, on `POST /v1/serp` or `POST /request`, with structured fields or a search URL: ```bash curl http://127.0.0.1:8789/v1/serp \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"query":"lunar science","country":"us","language":"en","maxResults":5,"extras":true,"shape":"serpapi"}' ``` This shape is built from the same parsed result as the default one, after the cache and any shared capture, so the two shapes share cached answers and captures in progress, and authentication, quota, the deadline and `fresh` work the same for both. It renames and regroups fields and adds no value the parser did not observe: ```json { "search_metadata": { "id": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01", "status": "Success", "processed_at": "2026-10-13 09:41:07 UTC", "total_time_taken": 3.19, "google_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5" }, "search_parameters": { "engine": "google", "q": "lunar science", "google_domain": "google.com", "gl": "us", "hl": "en", "start": "0", "num": "5", "device": "desktop" }, "related_questions": [{ "question": "What is lunar science?" }], "organic_results": [ { "position": 1, "title": "An example lunar science page", "link": "https://example.com/lunar-science", "displayed_link": "https://example.com › lunar-science", "snippet": "An example excerpt from a search result.", "sitelinks": { "inline": [{ "title": "Missions", "link": "https://example.com/lunar-science/missions" }] } } ], "related_searches": [{ "query": "lunar science missions" }], "pagination": { "current": 1, "next": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=10&num=10" }, "meta": { "provider": "vitna_serp", "engine": "google", "requestId": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01", "latencyMs": 3187, "transport": "browser", "coverage": "full", "parserVersion": "1", "provenance": "search_engine", "offset": 0, "page": 1, "count": 1 } } ``` | SerpApi field | From | Notes | |---|---|---| | `search_metadata.id` | `meta.requestId` | This request's identifier; a cache hit has its own | | `search_metadata.status` | | `Success`, or `Error` for a search that failed after admission | | `search_metadata.processed_at` | `general.timestamp` | When the page was captured, written `YYYY-MM-DD HH:MM:SS UTC`; a cache hit keeps its capture's time | | `search_metadata.total_time_taken` | `meta.latencyMs` | Seconds, to two places | | `search_metadata.google_url` | `input.final_url` | `bing_url` instead when Bing answered | | `search_parameters` | The request | `engine`, `q`, `google_domain` (always `google.com`), `gl`, `hl`, `start`, `num`, `device` (always `desktop`) and `udm` when sent, as strings; a Bing request gives `cc`, `setlang`, `first` and `count` | | `organic_results[].position` | `rank` | The position on the page; a rejected listing leaves a gap | | `organic_results[].title`, `.link` | `title`, `link` | | | `organic_results[].displayed_link` | `display_link` | The breadcrumb, or on a forum, video or community result the source line (`Reddit · r/buildapc`); omitted when the page shows neither | | `organic_results[].snippet` | `description` | Omitted when the page shows none | | `organic_results[].sitelinks.inline` | `sitelinks` | `title` and `link` each; the parser reads no sitelink snippets and does not tell Google's expanded block from its inline row | | `related_questions[].question` | `people_also_ask` | With `extras: true`; the question only | | `related_searches[].query` | `related_searches` | With `extras: true` | | `ads[]` | `ads` | With `extras: true`: `position`, `title`, `link`, `displayed_link`, and `description` from the ad's `snippet` | | `pagination.current`, `.next` | `pagination.current_page`, `.next_page_link` | `current` only for an offset on a ten-result page | | `meta` | `meta` | Vitna's own, unchanged | SerpApi fields Vitna does not parse are left out rather than sent empty: an organic result's `date`, `source`, `favicon`, `thumbnail`, `snippet_highlighted_words`, `redirect_link` and `rich_snippet`; the answers to People Also Ask and the links of related searches; an ad's `tracking_link`, `block_position` and sitelinks; `search_information`, which carries result counts (Vitna's own `search_information`, [Google's notices](#googles-notices), is not mapped into this shape); `pagination.other_pages` and `serpapi_pagination`; and page features such as the AI Overview, perspectives, the knowledge graph, the answer box, local results, top stories and images. A recognized page with no results has an empty `organic_results` and status `Success`, where SerpApi adds an `error` string. Where the operator sets `VITNA_SEARCH_SERP_FALLBACK_ENGINE=bing`, a Google search that Google does not answer can be answered by Bing. Both shapes say so in `meta.engine`, which is then `bing`. In this shape `search_metadata` gives `bing_url` in place of `google_url`, while `search_parameters` still records the Google request. A search that fails after admission keeps its HTTP status, `Retry-After` and Vitna's error, and adds `search_metadata` and `search_parameters`: ```json { "search_metadata": { "id": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "status": "Error", "total_time_taken": 0.01 }, "search_parameters": { "engine": "google", "q": "lunar science", "google_domain": "google.com", "start": "0", "num": "10", "device": "desktop" }, "error": { "code": "serp_routes_busy", "message": "all configured routes for this search engine are in use" }, "meta": { "requestId": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "engine": "google", "latencyMs": 8, "retrieval": { "attempts": [] } } } ``` `error` stays an object with `code` and `message` rather than SerpApi's string, because the code says whether and when to retry. A request refused before a search begins, for invalid input, authentication or quota, keeps the shared error envelope in either shape. The Web API's MCP server offers this route to agents as the `google_search` tool, with SerpApi's argument names; see [MCP](../web/overview.md#mcp). ## Measured delivery Three runs of live Google searches on the production search host, after its Google retrieval moved to residential exits alone on the evening of 2026-11-03 (Eastern time). Each run was a separate process on that host running the deployed SERP code as the service's own user, with the service's configuration and browser daemon: `engine: "google"`, `maxResults` unset (so 5), `fresh: true` to skip the JSON cache, and one call a query. The first two kept three searches in flight and the third twelve. The queries were technical questions (configuration keys, flags, command lines and named techniques), and none appears in more than one run. A search counts as answered when it returned without an error. Every search, with its query, result count and time, is in [serp-runs-2026-11.json](https://vitna.ai/search/serp-runs-2026-11.json). | Finished (ET) | In flight | Searches | Answered | 95% CI | All five results | Median | p90 | p95 | Slowest | |---|---|---|---|---|---|---|---|---|---| | 2026-11-03 21:57 | 3 | 100 | 97 | 91.5 to 99.0% | 89 | 4.01 s | 4.43 s | 9.30 s | 20.44 s | | 2026-11-04 00:23 | 3 | 200 | 199 | 97.2 to 99.9% | 186 | 3.89 s | 4.23 s | 4.87 s | 20.86 s | | 2026-11-04 15:42 | 12 | 2,500 | 2,273 | 89.7 to 92.0% | 2,256 | 4.19 s | 9.66 s | 15.96 s | 22.37 s | Times are over every search, refusals included, from the call to the parsed result on the search host. They leave out a caller's network round trip, the HTTP layer, authentication and quotas, so a request from elsewhere takes longer. The intervals are Wilson's; across the three runs 2,569 of 2,800 answered (90.7 to 92.7%), and across the two at three in flight 296 of 300 (96.6 to 99.5%). The first run followed 16 warm-up searches sent two at a time straight after the change and restart, of which 11 answered; they are not counted. Every refusal was returned as the 503 the [failure table](#authentication-quotas-and-failures) lists rather than as an empty `organic` list. In the runs at three in flight each was `serp_blocked` (8.99 to 17.97 s); in the 200-search run it was the first search sent. In the run at twelve, 222 were `serp_blocked` (6.87 to 21.75 s) and 5 `serp_timeout` (18.78 to 22.37 s), and a refusal took a median of 14.89 s against 4.16 s for an answer. They were not spread evenly: 184 of the first 250 searches sent were answered, and 2,089 of the 2,250 after them. The run did not record why. An answer can carry fewer than five results. In the 200-search run, 10 answers lost one or two to [doorway pages](#doorway-pages) (11 pages left out), and three carried 3, 1 and 0 results for a reason the run did not record. In the 100-search run, all 8 short answers were doorway removals. In the 2,500-search run, 16 answers lost one each to doorway pages and one carried 4 for a reason the run did not record. The runs recorded no attempt counts, so how many answers needed a second try is not known; 94 of 100, 193 of 200 and 2,209 of 2,500 finished under 7 s. Five is the default because Google unlocks a page's `/goto` result links in turn and a page waits for its slowest link: on 2026-11-03, links 1 and 2 answered at once, links 3 to 5 within about 2.3 s of the page settling, link 6 after about 7.6 s and links 7 to 10 after about 24 s (see [Run on owned infrastructure](#run-on-owned-infrastructure)). Earlier 100-search runs that week at `maxResults: 10` took 26.1 s at the median. These are three runs on one host within a day, on one kind of query. They describe those runs, not coverage of the wider web or a guarantee of availability, and Bing was not measured. The runs called the retrieval in the host's own process, so per-key quotas did not apply: twelve in flight is what the host sustained, not what one key may send. ## Retrieval routes and retries The operator chooses `auto`, `direct`, or `browser` transport. The following describes ordinary routes; the experimental owned adapter below replaces Google's routes when configured. With `auto` and no configured proxies, Vitna prefers a JavaScript browser for Google and direct HTTPS for Bing, with bounded fallback to the other available transport. With configured proxies, `auto` uses only browser routes pinned to those proxies. Explicit `browser` uses a browser route per proxy, or a local browser route when no proxies are configured. Configured proxy routes never fall back to local egress. Explicit `direct` uses direct HTTPS and refuses proxy configuration. Proxy pools rotate their starting route across requests while respecting route health and capacity. SERP browser captures allow ordinary GET images and fonts through the same public-address guard and request, byte, and deadline limits. This internal option does not allow arbitrary POSTs, media, or WebSockets. The requested Google `hl` or Bing `setlang` language is validated and forwarded to the browser locale; without a language hint the browser keeps its English default. These settings do not establish that Google will accept an egress route. Pinning a proxy URL keeps its endpoint and effective credentials consistent throughout one browser capture. Structured proxy entries can generate a new provider session credential for each admitted attempt. For browser captures, the provider must keep the same exit for that whole context, including redirects and subresource connections. A gateway that rotates on each connection can change exits within one capture even though its configured URL stays the same. Vitna does not verify the provider's exit IP or session stickiness. Route health and pacing track declared egress groups, not measured public exit IPs. The browser transport expects an HTTP(S) CONNECT tunnel to the checked destination address and retains target TLS certificate verification. Managed scraping proxies that intercept target TLS or return pages fetched by their own browser need a separate integration. For example, Crawlbase's [Smart AI Proxy documentation](https://crawlbase.com/docs/smart-proxy) states that it intercepts TLS and asks clients to disable verification. Vitna does not automatically disable verification for these endpoints. The default request tries at most three distinct routes, bounded by the available routes and the shared whole-request deadline. Each attempt has a separate four-second default cap within the remaining deadline. Only blocks, unreachable engines, attempt timeouts, and unrecognized pages permit another route attempt. Requests keep the selected engine and query throughout. Vitna returns the first validated result page; it does not combine rankings from several routes or change Google requests to Bing requests. Route concurrency and health are tracked in the current service process. The experimental owned Google adapter below adds a persistent admission gate. A route allows one active request per engine by default. Egress groups also allow one active request per engine by default; direct HTTPS and the local browser share that group. The first actual upstream block cools matching routes for that engine/egress group for five minutes by default. A redirect to the engine's own consent or sign-in page counts as a block. A browser challenge flag counts only with HTTP 403 or 429; otherwise the attempt fails as `serp_invalid_response` without a cooldown or another attempt. Repeated blocks double that cooldown, up to a one-hour default cap. A valid upstream capture on the same group resets its block streak; denied requests and cache hits neither reset nor escalate it. Other transient failures use a thirty-second default cooldown. A Google cooldown does not block Bing. If an uncached request cannot begin an attempt because all eligible routes are occupied or cooling, the service returns HTTP 503 with `Retry-After`. When attempts have already failed, their final retrieval failure is returned. Restarting the process resets ordinary route health. Ordinary service replicas do not share these pacing, concurrency, or cooldown limits. If replicas use the same physical exit, coordinate retrieval starts across them or run one retrieval process for that exit; adding replicas otherwise multiplies its outbound traffic. Independent retrieval starts are paced per engine and egress group. The default interval is selected uniformly between five and fifteen seconds, averaging roughly six starts per minute for each engine/egress group. Direct HTTPS and the local browser share both pacing and the egress concurrency slot. Fresh requests follow this pacing; cache hits can return without starting a new retrieval. Internal fallback within one logical request can reuse an egress only after its previous upstream attempt and cleanup have settled, within the same request deadline. The API does not queue requests until their next start time. If every otherwise eligible route is paced, it returns HTTP 503 `serp_routes_throttled` with `Retry-After` in seconds and `meta.retrieval.attempts: []`. Clients can retry after that interval. Busy routes take precedence. Otherwise, when waiting routes differ, the code and `Retry-After` describe the route that can start soonest, whether it is paced, cooling, or awaiting its runtime retry; equal start times report pacing. A paced route that is due sooner is never reported as another egress's longer cooldown. The shared thirty-requests-per-minute client limit caps admission; it does not promise that many upstream starts. Ordinary route pacing and egress slots are process-local; horizontally scaled instances do not coordinate them. Browser dependencies load lazily when a browser attempt is needed. Starting the service or reading its status does not launch a browser or probe a search engine. Missing browser dependencies can therefore produce an explicit runtime-unavailable response on a browser attempt. That response pauses the affected route, for every engine it serves, for sixty seconds. Requests in that window return HTTP 503 `serp_runtime_unavailable` with `Retry-After` and no new attempts. The first request after it makes one fresh attempt, so a slow browser start does not disable the route until restart. A request never retries the same route. A missing dependency keeps failing; repair the runtime setup and restart before verifying that route again. ## JSON cache and freshness Successful parsed JSON responses can be reused by the same client for thirty seconds by default, and for up to one hour when the operator sets `VITNA_SEARCH_SERP_CACHE_TTL_MS`. The cache is process-local and bounded to 128 entries and 8 MiB of serialized responses. Entries are kept per API client, including pages its multi-page jobs capture, so a cache hit, its age, and its capture timestamp never reveal another client's search. HTML and failed retrievals are not cached. Responses still carry HTTP `Cache-Control: no-store`; the short-lived cache belongs to the service, not an intermediary or the caller's browser. When a larger `maxResults` capture for the same client and query parameters is already cached, smaller `maxResults` requests slice from that superset entry (`cacheEvent: "superset"`). Operators can also configure stale-while-revalidate (`VITNA_SEARCH_SERP_CACHE_SWR_MS`), stale-if-error (`VITNA_SEARCH_SERP_CACHE_STALE_IF_ERROR_MS`), and an optional WAL-mode SQLite L2 store (`VITNA_SEARCH_SERP_CACHE_SQLITE_PATH`) so warm JSON entries survive process restarts. Live delivery and latency gates can be evaluated against a running host with `search:serp:benchmark`: ```bash npm run search:serp:benchmark -- --url https://search.vitna.ai --fresh ``` Set `fresh: true` to bypass a cached response. A successful new capture refreshes that cache entry. Cached results retain their original engine capture timestamp; a cache hit does not imply the search engine was contacted for that request. Cached requests still use the shared API authentication, quota, and concurrency controls. Restarting the process clears the cache. Its transient in-memory query/result data is separate from receipts, which omit queries and result content. When the cache is full, its least recently used entry leaves first, even before its TTL ends. Captures made with `fresh: true` are stored too, so a load test that searches fresh can push other clients' entries out. The status `cache.evictions` count shows how often that happens. Operator metrics count each answered request by where its page came from (`capture`, `cache`, or `shared` for a joined capture, below), split by test and real traffic in `vitna_api_serp_answer_sources_total`. Together they show how often searches repeat within the TTL before a longer one or a durable store is worth having. Matching in-flight JSON requests from the same client can share one capture in the current process. The canonical engine, trimmed query, effective offset, country, language, result limit, JSON format, and `fresh` flag must match. Fresh requests share only with matching fresh requests. Another client's identical request never joins; it needs its own route capacity like any other request. Each caller still uses its own API authentication, quota slot, request deadline, cancellation, request identifier, and latency. Cancelling one caller leaves the capture available to the others; when the last caller leaves, the producer is aborted. A shared capture runs on the route's whole timeout so later callers can join it. A route that rotates its own exits spends every second it is handed on more tries, so it gets no more than the latest waiting caller still wants. A lone Google search beside a Bing fallback hands the owned route 49 of the 55 s, so its tries come back before it leaves. Until 2026-10-12 the route got all 55, and a search that ran to its deadline lost its tries when its abandoned capture was aborted. HTML captures are separate, and failures never become cached responses. The sharing pool is bounded to 128 active captures and 32 pending callers per shared capture, including its first caller. An abandoned producer keeps its capacity reservation until it settles. Capacity refusals use HTTP 503 `serp_routes_busy` with `Retry-After: 1` and no new attempts. `meta.retrieval.route` identifies the route that captured the returned page. `meta.retrieval.attempts` lists this request's attempts with `route`, `transport`, `outcome: "success"` or `"failure"`, `latencyMs`, and an error `code` for failures. A cache hit has no new attempts. The sibling `cache` object reports `hit` and the cached entry's `ageMs`; a new capture reports `hit: false, ageMs: 0`. Route identifiers such as `direct`, `browser-direct`, and `proxy-1` are safe labels, not proxy addresses. Joined callers receive `meta.retrieval.coalesced: true` with `attempts: []`; the first caller receives `coalesced: false`. Authenticated `GET /v1/status` exposes `serp.retrieval` with configured attempt limits, cooldowns, cache counts, bytes and evictions, `minIntervalMs`, `maxIntervalMs`, `maxConcurrentPerEgress`, `maxBlockedCooldownMs`, `runtimeRetryMs`, and the safe engine transport preference map `preferredTransports`. Its `coalescing` object reports `active`, `waiters`, `maxEntries`, `maxWaiters`, and `hits`, without queries or key hashes. Its `routes` list includes safe route identifiers, transports, and per-engine state. Each engine state reports `ready`, `busy`, `cooldown`, `throttled`, or `unavailable`, active route count `inFlight`, shared egress count `egressInFlight`, attempt/success/failure counters, `consecutiveBlocks`, `cooldownUntil`, `unavailableUntil`, `nextRequestAt`, and `lastSuccessAt`. The latter timestamps use Unix milliseconds, with no success recorded as null. These are process-local operational observations; a `ready` route has capacity and is neither cooling nor paced, without an upstream availability probe. ## Multi-page Google jobs Use an asynchronous job to request up to 100 distinct organic Google links across at most ten observed result pages. The existing `POST /v1/serp` and `POST /request` remain single-page requests with `maxResults` at most 10. Multi-page jobs return parsed JSON and support Google only. ```bash curl http://127.0.0.1:8789/v1/serp/jobs \ -H "Authorization: Bearer $VITNA_SEARCH_KEY" \ -H 'Content-Type: application/json' \ -d '{"engine":"google","query":"lunar science","maxResults":100,"maxPages":10,"timeoutMs":300000}' ``` | Field | Default | Allowed values | |---|---|---| | `engine` | `google` | `google` only | | `query` | Required | Same validated query string as single-page SERP retrieval | | `country`, `language` | Unset | Same localization hints as single-page requests | | `maxResults` | 100 | Integer 1 to 100; target distinct result count | | `maxPages` | 10 | Integer 1 to 10; maximum captured result pages | | `timeoutMs` | 300,000 | Integer 1,000 to 300,000; whole-job deadline, including queue time | | `fresh` | `false` | Boolean; `true` bypasses completed-job reuse and requires a new collection unless an identical fresh job is already active | New collection jobs begin at offset zero and make fresh, sequential, isolated captures. They follow the recognized `pagination.next_page_link` from each fetched page, rather than asking Google for `num=100` or inventing offsets. A page with fewer than ten results can still have an observed continuation. Each attempt has a deadline of at most sixty seconds within the remaining job time and the instance's configured shared and SERP deadlines. Blocks and cooling stop the job instead of switching engine or bypassing the connection's penalty. Busy or paced routes use the exact private millisecond wait when available, subtracting cleanup time already spent; otherwise they use the public `Retry-After` seconds. The durable admission gate is rechecked before capture. Waits stay within the job deadline, with at most twenty tries for a logical page. The private timing hint is not an API field. A page refused admission because every service slot is in use (`service_busy`) or the per-minute limit is reached (`rate_limited`) also waits its `Retry-After` within the job deadline, with at most twenty admission tries. A daily limit or unavailable request accounting stops the job. Creation returns HTTP 202 with a `job` envelope, a `Location` status path, and `Retry-After: 1`. Its `id` identifies the job; a new job begins `queued`, while a reused job retains its existing status. `progress` reports `pagesCompleted`, `returnedResults`, `requestedResults`, and `maxPages`. `result` appears when captured results are available, and an optional `error` contains a safe code. Jobs expose creation, start, finish, and expiry timestamps when applicable. An admitted job becomes `running` when its worker starts. With `fresh: false`, an exact same-client request can reuse a successful `completed` job for thirty seconds after its finish time. Only jobs that reached their requested target without an error qualify; partial, failed, cancelled, and interrupted results are not reused. Engine, trimmed query, localization hints, result target, page limit, and job deadline must match. The original job ID, results, and capture timestamps remain intact. Identical same-client `queued` or `running` requests with the same `fresh` flag join that one job. This is one shared cancellable job: cancelling its ID stops it for every same-client caller using it. `fresh: true` bypasses a completed result and joins only an identical active fresh job. Reuse does not start more page captures or consume more page quota, but each POST still uses the job-control allowance. The creation response adds `job.reuse` with `kind: "none"`, `"active"`, or `"completed"`, `requestedFresh`, `ageMs`, and `ttlMs`. Read responses omit that request-specific reuse marker. Stable `job.freshness` records the original job's `requestedFresh` policy and latest page `fetchedAt`, when available; reused results retain those capture times. Use the same client bearer authentication to read or cancel that job: ```text GET /v1/serp/jobs/{id} DELETE /v1/serp/jobs/{id} ``` Creating and reading jobs share a separate limit of sixty requests per minute per client. Cancellation has its own ten-per-minute allowance so exhausted polling does not prevent cancellation. These controls do not consume query quota. Reads return HTTP 200 and cancellation returns HTTP 202, both with `{job}`. Unknown, expired, evicted, or another client's jobs return HTTP 404 `job_not_found`. Each admitted logical page consumes one shared SERP quota slot, including bounded route waits, with its concurrency slot held through cleanup. Job reads and cancellation remain available while page slots are occupied. Requesting 100 links does not purchase or guarantee 100 results. Cancellation preserves captured partial results. A running job can report `cancelling` while its worker remains occupied until the actual browser and network cleanup settles. The aggregate `result` contains `general`, `organic`, `pages`, and `meta`. Duplicate detection strips URL fragments. For exact `/watch` URLs on `youtube.com`, `www.youtube.com`, or `m.youtube.com` with one valid 11-character `v` value and no nondefault port, it also ignores `xstg`. Other query parameters, their encoding and ordering, and host aliases remain distinct. The first observed entry wins with its original link and discovery order; recovery preserves saved legacy results. Single-page SERP output preserves its observed listing rows. Each organic entry retains its page-local `rank` and adds `position` for the aggregate discovery order. Its `source` records the sequential capture `page`, actual observed `offset`, original `pageRank`, `fetchedAt`, and `serpUrl`. That aggregate position is not an invented Google global rank. `pages` retains each capture's URL, time, result counts, and safe metadata. Aggregate `meta.timingMs` separates observed `admission`, `retrieval`, `pacingWait`, and `checkpoint` time. Admission includes waits after a busy or per-minute refusal. Retrieval includes backend capture and physical cleanup; these are measured phases, not search-quality scores. Compare a fresh job separately from completed-result reuse. Google listings and snippets are a different output from native index matches or the research API's verbatim cited passages. An owned-route provider fee of zero excludes infrastructure, browser, network, storage, crawling, and operating costs. Only reaching the requested distinct-result target produces `completed`, `meta.complete: true`, and `meta.stopReason: "target_reached"`. Fewer results produce `partial` when usable links are retained, including `no_next_page`, `page_limit`, or a retrieval error; no usable results finish `failed`. Other stop reasons include `blocked`, `cooling`, `deadline` when the whole-job deadline ends collection, `timeout` when a page capture times out with job time remaining, both with error code `serp_timeout`, and `cancelled`. Failed, cancelled, or interrupted jobs retain usable partial results when available. The limits and observed continuation do not guarantee that Google will deliver 100 links. One authenticated live job on 2026-10-05 at 12:37:07 UTC was accepted with HTTP 202 and requested 100 links across at most ten pages. It returned `partial` with 94 unique publisher URLs in 107,047 ms, ending with `meta.stopReason: "page_limit"` and `meta.complete: false`. The ten captured offsets were 0 through 90 in steps of ten. Their counts were 7, 10, 10, 10, 10, 10, 10, 9, 10, and 10: 96 raw results with two duplicate links removed. All captures used `owned-google`; result source references matched their captured pages, with no reported error or challenge. The local stock browser used the owned OVHcloud exit under the same durable admission gate. This verifies one multi-page API run. The new multi-page console controls were not part of that live check; an authored real-Edge fixture separately verified creation, sequential polling, result cards, and page provenance. It does not establish sustained production reliability, geographic targeting, or guaranteed delivery of 100 links. The manager allows one active worker, eight queued jobs, at most 64 retained records, and two nonterminal jobs per client. Records expire one hour after creation. Under load, finished results can be evicted sooner: when all 64 records are in use, a new job removes a finished one, the submitting client's oldest first, otherwise the oldest of any client. Queued and running jobs are never evicted. Service startup defaults to memory. Opt into private JSON snapshots with `VITNA_SEARCH_SERP_JOBS_DIR=.search-data/serp-jobs` locally, or `/var/lib/vitna-search/serp-jobs` on a Linux service host. These snapshots contain queries and results; protect their directory as private operator data. An exclusive writer lease permits only one service per jobs directory. Its holder refreshes the lease every thirty seconds. A restart reclaims a same-host lease whose process is gone, whose PID now names a later process on Linux, or whose PID is the restarting process itself, as PID 1 finds after a container restart. A lease left unrefreshed for two minutes is stale from any host, so a replacement container can start once its predecessor's lease is that old. A service that finds its lease replaced or removed stops admitting jobs and writing snapshots. A restart marks recovered nonterminal jobs `interrupted`, preserving saved partial results without automatically resuming retrieval. Recovery deletes an expired snapshot once its ID and expiry are readable, without validating the rest, and accepts timestamps that a wall-clock step or host sleep left out of order. Authenticated status exposes these availability, persistence, queue, record, and limit counts under `serp.collection`. ## Run without an index SERP retrieval does not require a SQLite search index or a crawler worker. Mint client credentials as described in the [search overview](overview.md#starting-an-alpha-instance), then enable the SERP capability in the service's private environment: ```dotenv VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_CLIENT_KEY_SHA256"} VITNA_SEARCH_SERP_ENABLED=true VITNA_SERP_RESALE_APPROVED=true VITNA_SEARCH_SERP_TRANSPORT=auto VITNA_SEARCH_SERP_TIMEOUT_MS=10000 ``` Enabling retrieval does not authorize redistribution. `VITNA_SERP_RESALE_APPROVED=true` records the operator's decision that collecting Google and Bing results and redistributing them through Vitna's API, including fetched HTML, is permitted. Keep both settings unset until that decision is made. Startup and the live checker refuse `VITNA_SEARCH_SERP_ENABLED=true` without this confirmation, rather than running with SERP retrieval silently off. Start the service: ```bash node --env-file=.env.search platform/search/start.mjs ``` The direct transport uses the built-in Node runtime and makes bounded HTTPS requests to supported search engines. It requires no Bright Data API key, zone, search adapter, or provider account. Starting the process makes no search-engine probe. | Environment variable | Default | Meaning | |---|---|---| | `VITNA_SEARCH_SERP_ENABLED` | `false` | Enable independent Google/Bing SERP retrieval; also requires `VITNA_SERP_RESALE_APPROVED=true` | | `VITNA_SERP_RESALE_APPROVED` | `false` | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted; `true` or `false` | | `VITNA_SEARCH_SERP_TRANSPORT` | `auto` | Bounded direct/browser route selection; explicit `direct` or `browser` also supported | | `VITNA_SEARCH_SERP_TIMEOUT_MS` | 10,000 | Integer 100 to 60,000; retrieval deadline also bounded by the shared request deadline | | `VITNA_SEARCH_SERP_PROXIES` | Unset | Operator JSON array of at most 20 HTTP(S) proxy URL strings or structured route entries; requires `auto` or `browser` transport | | `VITNA_SEARCH_SERP_ATTEMPTS` | 3 | At most 1 to 5 distinct attempts per request, within one deadline | | `VITNA_SEARCH_SERP_ATTEMPT_TIMEOUT_MS` | 4,000 | Integer 100 to 60,000; per-attempt cap within the remaining deadline | | `VITNA_SEARCH_SERP_ROUTE_CONCURRENCY` | 1 | Integer 1 to 4; concurrent requests per route/engine pair | | `VITNA_SEARCH_SERP_EGRESS_CONCURRENCY` | 1 | Integer 1 to 4; concurrent requests per engine/egress group; shared by direct HTTPS and local browser | | `VITNA_SEARCH_SERP_MIN_INTERVAL_MS` | 5,000 | Integer 0 to 60,000; minimum jittered interval between independent starts | | `VITNA_SEARCH_SERP_MAX_INTERVAL_MS` | 15,000 | Integer 0 to 60,000; maximum interval; must be at least the minimum | | `VITNA_SEARCH_SERP_COOLDOWN_MS` | 30,000 | Integer 1,000 to 300,000; transient-failure route cooldown | | `VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS` | 300,000 | Integer 1,000 to 3,600,000; engine/egress cooldown after a block | | `VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS` | 3,600,000 | Integer 1,000 to 3,600,000; repeated-block cooldown cap; must be at least the base blocked cooldown | | `VITNA_SEARCH_SERP_CACHE_TTL_MS` | 30,000 | Integer 0 to 3,600,000; successful JSON reuse window, at most one hour; 0 disables reuse | | `VITNA_SEARCH_SERP_CACHE_ENTRIES` | 128 | Integer 0 to 1,024; maximum cached responses, also bounded by 8 MiB; 0 disables caching | | `VITNA_SEARCH_SERP_JOBS_DIR` | Unset; in-memory jobs | Optional private JSON snapshot directory, such as `.search-data/serp-jobs` locally or `/var/lib/vitna-search/serp-jobs` on Linux; one service writer per directory | | `VITNA_SEARCH_SERP_BROWSER_PROFILE` | Inherited Unblocker profile, otherwise `native` | `native` or `cloakbrowser`; the latter requires a browser executable path | | `VITNA_SEARCH_SERP_BROWSER_CHANNEL` | Inherited Unblocker channel, otherwise bundled Chromium | Optional installed browser channel, such as `msedge` | | `VITNA_SEARCH_SERP_BROWSER_PATH` | Inherited Unblocker path, otherwise Playwright default | Optional operator-selected browser executable path | | `VITNA_SEARCH_SERP_OWNED_SSH_HOST` | Unset | Experimental Google route: operator-configured SSH host alias, such as `owned-host`; requires `auto` or `browser` | | `VITNA_SEARCH_SERP_OWNED_EGRESS_ID` | Unset | Required declared physical exit label, such as `owned-exit`; follows the proxy egress-label rules | | `VITNA_SEARCH_SERP_OWNED_BROWSER` | `stock` | Experimental owned browser: `stock` uses the local installed browser; `camoufox` uses a separately provisioned remote Linux worker | | `VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT` | Unset | Required for `camoufox` only: private remote installation root `/tmp/vitna-firefox.` followed by 8–32 letters, digits, underscores or hyphens | | `VITNA_SEARCH_SERP_OWNED_STATE_PATH` | `.search-data/serp-browser-check-state.json` | Private persistent gate shared with matching diagnostics; relative paths resolve from the service working directory | | `VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT` | `.search-data/browser-controls` | Private root for fresh local stock-browser profiles; relative paths resolve from the service working directory; unused by `camoufox` | | `VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS` | 5,000 | Integer 1,000 to 60,000; minimum interval for the owned route's persistent admission gate | | `VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS` | 15,000 | Integer 1,000 to 60,000; maximum interval; must be at least the owned minimum | | `VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS` | `false` | `camoufox` only: the remote daemon rotates, spaces and cools its own residential exits, so the route neither paces nor rests (its interval settings are unused) and a failed search records a strike only | | `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` | 1 | Integer 1 to 32; captures the route runs at once; above 1 requires rotating exits. Slot N holds its lease in `<state path>.slot-N` | | `VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON` | `false` | `camoufox` only: send each search to the daemon's own socket (`nc -U` over the SSH host alias, or a local Unix socket when `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET` is also set), instead of starting the private worker, which saves about 150 ms a search. Without `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET`, needs a daemon running in the root and OpenBSD `nc` on the remote host. Route N sets its own `_N_` value | | `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET` | Unset | `camoufox` with `VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true` only: safe absolute `.sock` path (at most 100 characters, such as `/run/vitna-camoufox/daemon.sock`) when Search and the Camoufox daemon run on the same host, connecting over a local Unix domain socket with no `ssh` or `nc` process. When set, `VITNA_SEARCH_SERP_OWNED_SSH_HOST` and `VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT` are optional. Route N sets its own `_N_` value | | `VITNA_SEARCH_SERP_FALLBACK_ENGINE` | Unset | `bing` only: a Google search the owned route does not answer, for any reason but a cancelled request (blocked, cooling, throttled, busy, unreachable, an invalid answer, or out of its time), goes to Bing with the rest of the deadline. Until 2026-10-12 only the first four and running out of time did, and over 4,500 load-test searches that night 43 failed outright without trying Bing: 25 on busy routes, 15 unreachable, 3 invalid answers. Owned Google then stops 6 s before the deadline (at most a quarter of it) so Bing can still answer, and a Bing route still in its pacing interval is waited for whenever its next turn leaves 1.5 s to answer, however many turns other searches take first; until 2026-10-18 a search that lost one turn to another failed, about 5% of the AA benchmark's searches at a 30 s Google share. A Bing route busy with another search's hand-off is polled every 100 ms until it frees, while 1.5 s would still be left; until 2026-10-13 the second of two searches handed over together failed at once as `serp_routes_busy`. On 2026-10-10/11, 41 of 500 long-tail searches spent the whole 55 s on Google and failed without trying Bing, while 10 Google deliveries came in the last 6 s; a free Bing route answered in about 0.4 s | | `VITNA_SEARCH_SERP_CONTENTS` | `false` | `true` lets a request ask for [page contents](#page-contents), read live from each result page; `false` answers such a request 503 `serp_contents_unavailable` | | `VITNA_SEARCH_SERP_EXCERPT_RERANK` | `false` | `true` ranks [excerpts](#excerpts) by meaning as well as keyword. It needs the embedding settings and `VITNA_SEARCH_EMBED_BATCH_URL`, since the engine answers one request at a time and a search's query embedding would otherwise wait behind excerpt windows | | `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` | Unset | 1000 to 60000: the most of a search's time owned Google gets before timing out or handing off to Bing, never more than the deadline less any Bing reserve. Unset, Google gets all of that. The answered p90 is the searches Google answers late or never. By day on 2026-10-12, 102 of 457 Google answers came after 10 s, and the 8.6% Bing answered came at 37 to 49 s. Replayed with a 20 s share, the p90 fell from 32 to 21 s and Google's share from 91 to 84%; with 15 s, to 16 s and 81%. Overnight, when far more searches miss Google, a 20 s share cut Google from 66 to 49%. Bing's route takes a start every 5 to 15 s, so check it keeps up with what the share sends it | | `VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS` | Unset | 1000 to 60000: overrides `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` when `maxResults` is 5 or fewer, never more than the deadline less any Bing reserve. Under `--goto-respect-num 1 --goto-pace-ms 1000`, `maxResults <= 5` stays inside Google's 5-link zero-hold `/goto` regime (~3.8 to 4.1 s delivered p50) while `maxResults = 6` hits a ~7.6 s 6th-link hold (~9.6 s delivered p50) and `maxResults` 7 to 10 waits out the 22 to 24 s `/goto` hold ceiling (~26 to 27 s delivered p50), so operators can set a ~20 s share for short requests and a ~42 s share for 6-to-10-result requests | The SERP-specific browser profile, channel, and path override the corresponding `VITNA_UNBLOCKER_BROWSER_PROFILE`, `VITNA_UNBLOCKER_BROWSER_CHANNEL`, and `VITNA_UNBLOCKER_BROWSER_PATH` settings. Omitted SERP-specific settings inherit those operator values. The stock owned Google route instead requires an explicit absolute `VITNA_SEARCH_SERP_BROWSER_PATH`; its stock browser does not inherit a channel or Unblocker executable path. Camoufox requires its remote installation root and does not require a local browser executable. An owned route requires the SSH host, egress label, and the selected browser's configuration. `OWNED_REMOTE_ROOT` is refused with `OWNED_BROWSER=stock`. The opt-in browser integration test uses authored local response fixtures and the installed runtime; it does not query search engines: ```bash VITNA_SEARCH_SERP_TEST_BROWSER=1 node --test tests/search-serp-browser.test.mjs ``` Set the Unblocker browser path, channel, and profile when using an installed browser instead of bundled Chromium. The ordinary local browser transport reuses Vitna Unblocker's browser runtime and requires its Playwright dependency and Chromium installation. Install those in `platform/unblocker` as documented in the [Unblocker overview](../unblocker/overview.md). Proxy URLs contain no path beyond `/`, query, or fragment. Effective routes must be unique, as described below. Proxy configuration belongs in the private operator environment; URLs, credentials, and templates are not exposed through responses, authenticated status, receipts, or diagnostic reports. Browser retrieval does not run a CAPTCHA solver or wait for a challenge to be solved. Browser requests keep the installed runtime's coherent browser headers and user agent. The service does not rotate random user-agent strings to disguise one browser as another. Direct retrieval sends a bounded `Accept-Language` matching the initial engine URL's `hl` or `setlang`, with `en-US,en;q=0.9` when no hint is supplied. It requests `Accept-Encoding: identity` so its HTML size bound applies to the received body. Each browser attempt uses an isolated context and its configured pinned proxy, when present. On Linux, the host or container runtime must support Chromium's sandbox and user namespaces. Run the browser service as an unprivileged user with the sandbox enabled. A sandbox launch failure requires fixing that runtime's sandbox support. ## Run on owned infrastructure Vitna can run the API, browser captures, organic parser, route scheduler, and CONNECT gateways on operator-controlled machines. This path requires no managed scraping service or proxy provider API. Run the [owned egress gateway](../unblocker/overview.md#run-an-owned-egress-gateway) on each chosen machine, then register its authenticated endpoint in the search service's private environment: ```dotenv VITNA_SEARCH_SERP_ENABLED=true VITNA_SERP_RESALE_APPROVED=true VITNA_SEARCH_SERP_TRANSPORT=browser VITNA_SEARCH_SERP_PROXIES=[{"url":"https://serp:URL_ENCODED_PASSWORD@owned-egress.example.com:8792","egressId":"owned-cloud"}] ``` The endpoint is a placeholder for an operator-owned listener with TLS. Use a separate gateway port when Search already occupies that machine's port 8791. Keep the gateway's credential hash on the gateway and its plaintext connection credential only in the private browser configuration. The search release includes the gateway modules and the three `deploy/vitna-egress` templates. Adapt the service template's working directory, Node executable, and environment path to the host installation. Packaging these files does not install a service or publish a listener. Declare the same `egressId` for listeners or machines sharing a physical exit. Adding containers or gateways on that exit adds capacity without establishing a new network route. Owning the stack does not establish that Google accepts its requests; validate each available route with the diagnostics below and retain its block cooldowns. ### Experimental stock Edge over owned SSH An opt-in Google adapter runs the installed desktop Edge browser through an operator-owned SSH exit with public IPv6 connectivity. Configure a noninteractive SSH alias privately, with host verification and TCP forwarding available. Browser-to-Google TLS retains target certificate verification. This SSH transport selects checked public IPv6 targets, with no IPv4 fallback. The desktop must remain online. A cloud API process cannot use this Windows browser without a connected worker on that desktop. ```dotenv VITNA_SEARCH_SERP_ENABLED=true VITNA_SERP_RESALE_APPROVED=true VITNA_SEARCH_SERP_TRANSPORT=auto VITNA_SEARCH_TIMEOUT_MS=30000 VITNA_SEARCH_SERP_TIMEOUT_MS=30000 VITNA_SEARCH_SERP_OWNED_BROWSER=stock VITNA_SEARCH_SERP_BROWSER_PATH=C:/Program Files (x86)/Microsoft/Edge/Application/msedge.exe VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.json VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT=.search-data/browser-controls ``` These settings select `owned-google` exclusively for Google, with one attempt and no fallback to ordinary routes. Bing keeps its configured transport and proxies. Allow twenty to thirty seconds for both the shared request deadline and SERP deadline; the smaller deadline still bounds the capture. Existing route defaults remain unchanged until explicitly configured. Each capture uses a fresh profile and headed stock browser, without headless or browser automation flags. It submits the actual Google GET search form. Before accepting a capture, it checks that result rows and the observed next-page URL agree across two observations 150 milliseconds apart. Owned-route starts retain a random five-to-fifteen-second interval by default. Operators can set `VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS` and `VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS` within the bounds above. These settings change the same durable admission gate; they neither reset a cooldown nor change the declared egress identity. On Windows, the stock launcher assigns the suspended browser to its owned Job Object before resuming it. Cleanup terminates that job and requires confirmation that its assigned process tree is empty before releasing the lease or deleting the profile. This containment applies to the Windows helper. Transient profile locks permit bounded removal retries, with the allocated directory, root identity, and real paths rechecked before every retry. A replaced directory is preserved; missing drain confirmation keeps the lease held and prevents runtime reuse. The renderer accepts delayed drain confirmation only within the original request deadline. Exact cleanup proof releases a temporary launcher fence; unverified physical cleanup keeps the fence in place. The renderer resolves recognized opaque Google `/goto` links with bounded native-browser GETs through fresh subpages in the same context, using `maxRedirects: 0`. Native interception reads status and `Location` headers, then aborts every response, including HTTP 200, before following a redirect, downloading its body, or executing scripts. Validated publisher destinations are recorded for the parser without fetching them or rewriting the captured HTML. An unrecognized wrapper response is an explicit failure. At most two link-resolution workers share this capture's context, request budget, and deadline. Mappings retain the observed listing order, and failure or cancellation retains both workers' cleanup before releasing the lease. SERP pages themselves remain sequential. Non-navigation POSTs are restricted to the main frame at exactly `/gen_204`, `/log`, `/httpservice/retry/enablejs`, and `/httpservice/retry/jserror` on `https://www.google.com`: at most 32 per capture, with at most 64 KiB of reported body data per request. A response-stage gate aborts POST redirects before another hop. Public-address, request, byte, and deadline guards remain in force. Native proxy authentication supplies the capture's private credential only for a Basic challenge from its exact loopback gateway with the `Vitna` realm. Server authentication and mismatched or repeated challenges are refused. Credentials stay out of general HTTP authentication and target headers. A gateway 407 is a runtime/setup failure, without a Google block penalty. The dedicated Google capture admits at most 128 browser requests, while SSH remains capped at 80 tunnels. HTTP request interception does not prevent every speculative TLS preconnection. Those connections still pass the gateway's DNS, public-address, transfer-byte, and tunnel limits. The shared state file reserves starts five to fifteen seconds apart and persists block cooldowns from five minutes up to one hour. Matching declared egress identities and state paths share this gate across API and diagnostic invocations; the exclusive lease remains held through actual cleanup, even after caller timeout. Keep the physical exit label stable. It is an operator declaration, not verification of the exit's public IP. Ordinary SERP pacing and cooldown environment settings do not reconfigure this durable gate. The owned route's process-local retrieval still follows the operator's cooldown and cache settings, as Bing's routes do: `VITNA_SEARCH_SERP_COOLDOWN_MS`, `VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS`, `VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS`, `VITNA_SEARCH_SERP_CACHE_TTL_MS`, and `VITNA_SEARCH_SERP_CACHE_ENTRIES`. A capture still needs the gate's admission. Authenticated status joins the gate deadlines this process last observed to `owned-google`'s process-local state. A reserved start reads `throttled` until its `nextRequestAt`, and a durable block cooldown reads `cooldown` until its `cooldownUntil`. Each capture opens the gate afresh, so a start reserved by another process sharing the state file appears only after this process observes it, usually through a refusal. The implementation and live retrieval evidence remain distinct: | Check | Verified evidence and limit | |---|---| | Windows process lifecycle | Native Edge authored capture and early-abort fixtures passed 2/2, confirming process drain and profile cleanup | | Native response interception | Authored POST redirect and `/goto` HTTP 307, 308, and 200 fixtures passed; this verifies guards, not live Google access | | Gateway authentication | Authored proxy-auth fixture suite passed 5/5, including native Edge CONNECT and server-challenge checks | | Earlier Google prototype | One capture produced nine listings, eight with opaque `/goto` links; one separate GET returned a publisher destination with HTTP 302 | | Full authenticated owned API | Three distinct fresh Google requests returned HTTP 200 through `owned-google`/`browser` on 2026-10-05: “example domain” returned nine publisher URLs in 5,477 ms, “NASA solar system” five in 4,989 ms, and “Python documentation” seven in 4,588 ms; all 21 URLs resolved without a challenge | | Multi-page owned API | One authenticated job captured ten observed Google pages on 2026-10-05, returning 94 unique publisher URLs from 96 listings in 107,047 ms; source references matched, with no error or challenge; the 100-link target ended partial at the page limit | | Local console browser flow | An isolated browser connected with the existing local key, selected Google SERP, submitted “example domain,” and rendered five publisher links after HTTP 200 in 3,959 ms | These samples used the online PC's stock Edge browser and an owned cloud exit. The Python sample also verified full service startup at the local Search console. Health, readiness, and authenticated status returned HTTP 200, with the existing 65-page pilot index loaded. Browser and SSH cleanup completed. This small sample does not establish production reliability, geographic targeting, broad Google coverage, or a full replacement for Bright Data's retrieval infrastructure. The route remains experimental and can still face challenges. ### Experimental Camoufox on an owned Linux host Set `VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox` to use an operator-provisioned remote Firefox worker through the same private SSH alias, declared exit, state file, and owned pacing. This mode needs no local stock browser or desktop worker. Provision `platform/search/serp-camoufox-worker.py` in the private remote installation alongside Camoufox 0.5.7, Playwright 1.62.0, the pinned Firefox 156 beta 34 build, private setup metadata, and permissions for its authenticated cgroup v2 parent. The repository has no supported public provisioner. The worker validates private paths and declared pin metadata; operators must verify installed binaries and packages during provisioning. ```dotenv VITNA_SEARCH_SERP_ENABLED=true VITNA_SERP_RESALE_APPROVED=true VITNA_SEARCH_SERP_TRANSPORT=auto VITNA_SEARCH_TIMEOUT_MS=60000 VITNA_SEARCH_SERP_TIMEOUT_MS=60000 VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT=/tmp/vitna-firefox.PRIVATE_INSTALLATION VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.json ``` Replace the installation placeholder privately. Stock remains the default; the remote option changes no ordinary routes or deadlines unless configured. Each search on such a route runs the private worker over SSH, and the worker starts a second Python process for the capture. On the box on 2026-10-17 each start cost about 85 ms (17 ms for the interpreter, 19 ms to compile the worker, 39 ms for its imports), so about 170 ms of every search went to starting processes before the daemon was asked. With `VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true` the service instead writes the search to the daemon's socket, `<root>/daemon.sock`, with `nc -U` over the same SSH host alias, and reads the daemon's one-line answer: about 20 ms outside the daemon, measured the same night. When Search and the daemon run on the same Linux host, `--socket-path` and `--socket-group` on `serp-camoufox-daemon.py` expose an additional Unix socket in a non-world-writable runtime directory (such as `/run/vitna-camoufox/daemon.sock`, mode `0660` when `--socket-group` is set and `0600` otherwise), and `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET` connects to it directly from Node with no `ssh` or `nc` process at all (under 1 ms per capture vs ~150 to 300 ms to spawn `ssh ... nc -U`). The answer is checked exactly as the worker's is (the page's identity, its redirect mappings and a parse that is neither a block nor a failure), and the daemon's budget keeps the worker's margin of 2 s or a tenth of the request. The daemon owns its browsers and their cleanup, so no receipt is awaited: a cancelled search closes its SSH client, and the daemon finishes or abandons that search on its own, as it does when a worker is stopped. Keep the SSH alias's `ControlMaster` connection open (`ControlPersist`): a search that has to open a new one pays its handshake, about 240 ms on the box, on either path. A daemon with a residential pool tries up to 20 exits per search and spaces and cools each exit itself. The route's own rests were built for one IP and work against that pool: a search the daemon could not land locked the whole route for minutes, and one search at a time turned a second caller away. Set `VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS=true` on such a route and it neither paces nor rests: failures count strikes only, and its interval settings are unused. `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` (up to 32) lets that many searches run at once, each holding its own durable lease slot. When every slot is taken, a new search waits for one to free instead of being refused, as long as its deadline would still leave 8 s to capture (for a shared capture, the deadline of the latest caller still waiting on it). A search with less time left, or one whose try already failed, is still answered `serp_routes_busy` at once. A waiting search holds one of the instance's `VITNA_SEARCH_CONCURRENCY` request slots, so set that well above the route's concurrency: at the default of 4, a route serving 3 leaves room for one waiter and the next request is refused as 429 `service_busy` instead. Give the daemon more warm browsers than that (`--max-slots`, one upstream session line each): a search fails over browser by browser and a blocked one relaunches, so three searches on three browsers kept waiting on each other. Such a route also keeps all but 15 s of the request deadline per fallback route, rather than an even share. A route searching from a server's own address must keep both unset. Numbered routes never inherit them from route 1. The worker creates a fresh headless context and checks stable result and continuation evidence while preserving the canonical search identity. Its authenticated CONNECT guard validates every DNS answer before pinning a public target address. Browser traffic is GET-only, bounded to 128 routed requests and 128 CONNECTs, 20 MiB of aggregate bidirectional proxy TCP payload, and 2 MiB of captured HTML. Unknown background destinations remain denied without invalidating an otherwise valid capture; forbidden main navigation still fails. A reCAPTCHA frame or widget marks a challenge on any page. Challenge wording counts only on a page without recognized organic results, and never in the title, search form, result cards or scripts, which echo the query. Up to ten observed opaque `/goto` links use Python TLS header-only GETs through that same guard, with at most two workers. This is a hybrid browser/Python retrieval path, not browser-native redirect mapping. It does not share Firefox cookies, follow publisher destinations, or rewrite captured HTML. Each attempt has isolated private request files. Stdin EOF, exact-attempt cancellation, and deadlines stop its owned cgroup; a matching durable receipt must prove child exit and an empty group before the API releases its gate, within the bound below. Losing SSH does not itself prove browser cleanup. A capture whose SSH run ends without that receipt fails as `serp_unreachable`, never as the caller's cancellation, unless a receipt recovered afterwards proves its page or a different failure. Recovery resends exact-attempt cancellation until the worker answers it, then polls for the receipt, backing off from 5 to 60 seconds. A cancellation that finds no directory for its attempt, because SSH failed before the run created it or the host was unreachable, records a terminal `request_cancelled` receipt in that directory's place, so recovery ends with proof. A run that arrives later for the same attempt then refuses before starting a browser. If the installation itself is gone, for example after a reboot cleared `/tmp`, nothing can answer and the bound applies: recovery holds the gate for at most 180 seconds, longer than any worker supervisor for the attempt can run, then releases it without proof. The capture's private drain then reports `serp_camoufox_cleanup_unconfirmed`, and service shutdown, which waits no longer than that bound, logs that a resource could not finish closing. Neither names the host or the query. Once its receipt proves cleanup, an attempt keeps only that receipt: the worker deletes its request, progress, capture and DOM diagnostic files and its private home and temporary directories, so captured pages are not kept for later inspection. Each new run first removes at most eight attempt directories: any whose cleanup receipt is more than 15 minutes old, and any other attempt more than two hours old for which no cgroup exists. Removal happens as runs start, so an idle host keeps its latest receipts until its next run. On 2026-10-05, this optional route was verified through authenticated local API requests with `fresh: true` and the actual Search console, using the owned OVH Camoufox worker without a paid retrieval service, proxy provider, or solver: | Integrated live check | Observed result | |---|---| | Fresh API, “Python asyncio TaskGroup documentation,” page 1 | HTTP 200; nine verified organic results in 36,202 ms | | Fresh API, “What are the latest US AI chip export restrictions?” | HTTP 200; nine organic results in 23,252 ms, including Congress and Reuters listings | | Fresh API, the same Python query at page 2 / offset 10 | HTTP 200; ten organic results in 7,843 ms; the two Python pages contained 19 distinct publisher URLs | | Local console, “site:bis.gov H200 January 2026 license review policy” | HTTP 200; four BIS links rendered in 6,875 ms | Further controlled fresh samples returned nine Rust results in 13,935 ms, nine headphone results in 11,136 ms, nine for an accented French query in 9,506 ms, and three results on matching hosts for a quoted MDN query with `maxResults: 3` in 9,697 ms. Python page 3 at offset 20 returned ten results in 9,398 ms, bringing the three pages to 29 distinct publisher URLs. A fresh Rust collection targeting 18 links across two pages finished `completed` with 18 distinct links: 26,069 ms wall time and 23,162 ms reported job time. The fresh PostgreSQL 18 MERGE baseline returned nine results with nine snippets in 8,463 ms, including 8,354 ms of owned capture, without cache reuse or coalescing. These additional checks included six successful single-page captures and two Rust collection pages before the empty-page change. After that change, a verified empty response brought the accepted checks to nine captures: seven standalone pages, including the empty result, and two Rust collection pages. No CAPTCHA, HTTP 403, or HTTP 429 was recorded in this additional sample. Limits appeared alongside those successes. A NASA `site:` query retained its address-bar identity but yielded mostly unrelated or off-site listings; the scope guard refused it with `serp_query_mismatch`. The first French call returned HTTP 503 with `Retry-After: 2` before capture. An improbable quoted query intended to test empty results twice reached HTTP 504 near sixty seconds; the stored DOM later showed the genuine Google title, query and canonical URL, an explicit “About 0 results” count, an AI-only overview, and no organic listings. The initial shell completed with three main-document HTTP 200 responses. Narrow parser and worker support passed an offline regression with that stored AI-only DOM. A later fresh retry of the original quoted query returned HTTP 200 with nine organic listings in 8,122 ms, without cache reuse or coalescing. Google had changed its response; this failed the expected empty-result check and is excluded from the accepted captures. A separate nonexistent scoped query returned HTTP 200 with `organic: []` in 6,884 ms through one fresh owned attempt, followed by clean idle state. This verifies genuine empty API output, but does not establish that the original AI-only layout was reproduced live. Forced-cancellation receipts can retain stale initial counters, so zero counters alone do not prove zero traffic. A separate no-Google cgroup self-test confirmed child and grandchild cleanup. The earlier four-result diagnostic below retains its separate browser and header-mapping timings; those timings are not API latency. The integrated route remains experimental. This small controlled sample does not establish sustained reliability, soak or load performance, unchallenged future access, or public production deployment. #### Warm standby browser daemon and latency optimizations On 2026-10-07, the Camoufox worker was augmented with an optional standby daemon (`serp-camoufox-daemon.py`) to reduce single-query latency from the 3.6s to 4.7s cold baseline down to sub-second and low-second times. Cold browser process startup accounts for approximately 1.1s to 1.3s of each independent search attempt. When active, `vitna-camoufox-daemon.service` maintains a pre-warmed headless Camoufox browser instance listening on a private Unix domain socket at `/tmp/vitna-firefox.PRIVATE_ROOT/daemon.sock` (permissions `0600`, owned by the unprivileged worker user). Key architectural properties of the standby daemon include: - **Isolated ephemeral contexts**: Each incoming search query creates a fresh `BrowserContext` via `browser.new_context()`. The context navigates the query, settles the DOM, unwraps redirects, and is closed immediately upon completion. No cookies, cache, storage, or history leak across queries. - **Parallel redirect unwrapping**: Google `/goto` opaque redirect links are unwrapped concurrently through the local guarded CONNECT proxy with `GOTO_WORKERS = 4`, reducing redirect resolution overhead from ~800 ms to ~250 ms. - **Resource filtering**: Image requests (`resource_type == "image"`) are aborted at the route level, eliminating unnecessary asset downloads (~170 KB payload reduction per query) without affecting search result layout or text. - **Gecko subsystem tuning**: Firefox preferences disable hardware video decoding, media autoplay, background animations, and session history to minimize memory consumption and thread churn. - **Resource bounds and idle auto-shutdown**: The daemon automatically closes the browser process after 10 minutes of inactivity (`IDLE_TIMEOUT = 600`), guaranteeing zero idle browser RAM footprint. It recycles the browser process after every 50 requests (`MAX_REQUESTS = 50`) to prevent memory accumulation. The production unit's cgroup allowed `MemoryMax=6G` and `TasksMax=2048` on 2026-10-13, when its ten browsers used about 2.6 GB and 982 tasks. - **Profiles a stop leaves**: Playwright removes each browser's temporary profile when the browser closes, but stopping the service kills its driver first, so every restart left each browser's profile in the unit's `TMPDIR` (`ROOT/work`): 1,185 of about 29 MB, 34 GB, by 2026-10-17. At start, before it launches a browser, the daemon lists the Playwright and stock Chrome directories there (`playwright_firefoxdev_profile-*`, `playwright-artifacts-*`, `vitna_chrome_slot-*`), and once it serves it removes them, sparing any a running process names on its command line. It does this only when `TMPDIR` is `ROOT/work`, never in a shared one. - **Seamless cold fallback**: The supervisor worker checks for the daemon socket first. If the daemon is stopped, restarting, or unresponsive, the worker falls back immediately to cold in-process browser execution. Live benchmarks on the primary `vitna-api` host on 2026-10-07 confirmed: | Benchmark Query | Cold Execution Latency | Warm Daemon Server Latency | Total Round-Trip (Remote Client) | |---|---|---|---| | Single-result query (“apple investor relations”) | 3,658 ms | 745 ms | 1.57 s | | 9-result query with 9 unwrapped redirects (“nvidia quarterly earnings report”) | 3,820 ms | 1,512 ms | 2.17 s | All supervisor receipt checks, attempt isolation, and cgroup cleanup guarantees remain fully enforced. `--fast-relaunch 1` (off by default) keeps each browser slot's Playwright driver running across relaunches, replacing it only after a launch on it fails, and lets an old browser close while its replacement starts. A relaunch after a refusal took about 2.5 s on the API box on 2026-10-13 (medians of 5): closing the old browser 306 ms, the exit check 721 ms, starting the new browser 828 ms and its first context and page 619 ms. A slot launched, relaunched five times and closed there with the real browser (no exit check, no search traffic) took 1,751 ms a relaunch without the setting and 1,145 ms with it: the close it waited for went from 336 ms to none, and the browser start from 835 to 567 ms. A slot's driver stops when the slot closes, after the old browsers on it. The first try on each browser records `browserStartMs` and `browserCloseMs`, the close its relaunch waited for (null when no relaunch came before it). #### Failover across verified exits With upstream residential proxies, each search fails over across up to 20 distinct exits (`DEFAULT_MAX_ATTEMPTS`) within the request's SERP budget. Every attempt has 8 s and runs on one browser slot whose exit was checked as a US address that is not cooling; a challenged exit cools for ten minutes and its slot rotates to a new session. Failed tries got faster (1.7 to 1.9 s each by 2026-10-12), so the cap, not the deadline, became what ends Google: 384 of 427 searches Google missed stopped at exactly 20 tries, with 38.9 s used at the median of a budget near 48 s, while tries after the 13th still delivered 6.4%. `--max-attempts` (1 to 40, default 20 in `legacy` profile, 2 in `--profile standard`) sets the total try cap, and `--max-block-attempts` (0 to 40, default 0 off in `legacy` profile, 2 in `--profile standard`) stops a search immediately once Google returns that many `serp_blocked` responses so a 120-second query hold never cascades across additional warm residential exits; a 40-try history is about 100 KB. The worker reads the history back through a progress file capped at 1 MiB, which holds the longest one a search can make: 42 tries (the cap plus a race's overrun, below) with every field at its bound, about 750 KB, built by the daemon tests. Every answer says what ended it in `endedBy`: `delivered`, `attempts` (the cap), `exits` (no browser could give it an exit, below), `deadline`, or `stopped` (a race another chain won). A search the deadline ends has to report before the launcher stops waiting, and the launcher counts its budget from before the worker started. So the worker hands the daemon its own remaining time less 2 s, or a tenth of it on short budgets, as room to write its files and exit. Until 2026-10-12 it handed over all of it, and such searches were cut off with their answer unread: 48 of the 49 that Google missed at `--max-attempts 32`, and 3 of 3 at 40, left receipts without the daemon's tries. A search with no exit to be had ends at once as `serp_unreachable`, which Search hands to Bing, instead of waiting out its deadline. That is so in two cases. Every browser has already failed to find this search an exit, or the provider has stopped giving exits: 20 exit probes in a row have failed across the pool, the last within 30 s, and no browser is still live. On 2026-10-12, from 16:00:38 to about 16:13, every DataImpulse line answered `407 TRAFFIC_EXHAUSTED`, no browser could launch, and each search waited its whole 49 s before Bing answered it. A probe that answers ends the run of failures, and after 30 s without a failed probe the next search probes again, so the pool resumes by itself once the provider is back. An exit that just delivered (proven) delivered 79.8% of the first tries it served and 25.9% of the later ones, against 29.5% and 7.3% for other exits (22,708 tries, 2026-10-10 to 12), yet 72% of its tries went to searches that had already missed. `--proven-first-wait-ms` (0 to 5000, unset by default) keeps proven exits for first tries: a later try takes one only when no other warm browser is free, and a first try waits up to that long for one that is busy or coming off its spacing. Each try records `firstTry`, `provenExit` and, on a first try under the setting, `provenWaitMs`. A 100-search run on 2026-10-08 delivered 56. In its last 43 searches, 246 exit attempts went as follows: 177 met Google's reCAPTCHA, 31 stalled, 21 delivered. A stall is a results page whose header Google flushes (title and search box, about 84 KB of DOM) and whose body never arrives; each was waited out to the attempt deadline. Two changes spend the budget on more exits instead: - **Stall cut-off**: an unrecognized page that stops changing for 2.5 s (`STALLED_PAGE_GRACE`) ends the attempt as `serp_timeout` with `unrecognizedReason: "google_page_stalled"`. Healthy captures settle within about a second; no stall recovered in production. - **Exit freshness**: an exit proved by a launch probe or a post-capture check within 15 s (`EGRESS_FRESH_SECONDS`) skips the before probe, about a second per attempt. A cooling exit is still refused, the post-capture check still runs on every delivered page, and the attempt history marks the skip with `egressCheckSkipped` and `egressVerifiedAgeMs`. - **A page that changes while its links resolve**: Google can keep rendering after a results page has held still. When the final check finds the page changed but still a results page for the same search, the daemon resolves only the links that are new and checks once more, instead of discarding a delivered page (about 280 KB through the exit) and starting a fresh try. It fails as before if the page changes again. The attempt history marks this with `pageChangedDuringRedirects`. About 2% of searches hit it on 2026-10-09. `--fingerprint-os` (`linux`, `windows` or `macos`, default `linux`) sets the operating system the browser fingerprint presents. Each line of the upstream proxy file (`--upstream-proxies-file`) is one warm browser's session: `host:port:user:pass`, `user:pass@host:port` (DataImpulse's dashboard export) or a full `http://user:pass@host:port` URL. After a block the browser rotates to a new exit before relaunching, in the provider's own way. Decodo keeps the session in the username (`...-session-<id>-...`), so the daemon swaps the id. DataImpulse keeps it in the port: each port from 10000 to 20000 holds one exit for the dashboard's rotation interval (measured 2026-10-10), so the daemon moves the browser to a random new port on the same login. Set DataImpulse to Sticky; with Rotating, the check page, the results and the exit checks would each leave from a different IP. Either way the exit is verified through `ip.decodo.com/json`, which answers through both. `--direct-exits` (`4`, `6` or `4,6`, unset by default) adds one browser per family that leaves from the host's own IPv4 or IPv6 address, through the same guard with no upstream line. Direct browsers come on top of `--max-slots` and first: a direct address is tried whenever its spacing allows, and the residential browsers take the rest. `--direct-interval-ms` (4000 to 60000, default 6000) spaces searches on each direct address, proven or not. On the API box on 2026-10-13, one browser a family at 10 searches a minute delivered 593 of 601 first tries over IPv4 and 297 of 301 over IPv6 (98.7% each, median 0.84 and 0.80 s), against 61 to 75% through the residential lines beside them. An address's standing moves with use, though: by 2026-10-14 the IPv6 delivered 78%, steadily, with `udm=14` or the full page alike, and the IPv4 was refused 4 times in 5. Google refused single E2B addresses within 8 to 10 minutes at 27 or more searches a minute, and within about 40 at 15. Nothing can rotate a direct address, so a refusal relaunches its browser on it, and the address cools only when 7 of Google's last 10 answers to it were refusals, for 10 minutes; one more refusal after that cools it again until deliveries push the refusals out. With `--keep-proven-exits 1`, a refusal after a delivery keeps the browser instead, unless it is the one that cools the address. Simulated at 10 searches a minute, a steady 22% refusal rate keeps an address in use 91% of the time and 80% keeps it in use 1%. A residential exit's cooldown after every refusal idled both addresses for most of a 30-minute run beside production, and since a first version counted direct browsers within `--max-slots`, their cooling left that arm two browsers. A browser closed while its address cools is relaunched in the background once it has cooled, at most every 30 s, so a family whose route is down costs one probe a half-minute. A direct browser stays warm whatever `--idle-timeout` says, is launched without geoip as it was measured, and has its address probed at launch only: nothing can move it under a page, and probing each search would load the echo from one address. Its probes leave the provider-outage watch alone, and its record stays out of the network judgments, which its rate would skew. Each try records `directFamily` (4, 6 or null). `--direct-sources` (public addresses of this host, comma-separated, unset by default) adds one direct browser per address, bound to it: its connections and its launch probe leave from that address, and the address echo must report that same address back or the browser is not launched. It is for addresses added to the host for searching, such as OVH Additional IPs on `ens3`, each spaced `--direct-interval-ms` apart (60000 for one search a minute) and cooled by the same rule. By 2026-10-14 the box's own IPv6, after a day of tests at ten a minute, was refused on all 30 searches at one a minute; whether a fresh address used gently stays healthy is what a day at one a minute on new addresses is to show. `--static-proxies-file` (a file, unset by default) adds one browser per line whose exit never changes, such as a dedicated ISP proxy's own port, written as the upstream lines are; blank lines and `#` comments are skipped. Static browsers come after any direct ones and ahead of the residential lines, on top of `--max-slots`, and are kept as a direct address is: never rotated, spaced `--direct-interval-ms` apart, cooled only when 7 of Google's last 10 answers to the address were refusals, never closed for idleness, and relaunched in the background once cooled. Unlike a direct browser, a static one launches with geoip from its exit, as a residential one does, and its pages are checked after they deliver, since its provider could move it. Its launch probe, through its own line, proves the exit until such a check fails, so its tries skip the probe before Google. Its probes leave the provider-outage watch, which is for the rotating lines, alone. With `--keep-proven-exits 1`, a refusal after a delivery keeps its browser, but never one that cools the address. A fixed exit, static or direct, takes a search's first try only: a later try runs while Google holds the search, when any exit is refused, and each refusal costs an address that cannot be swapped (after one, a proven exit's next first try failed 24.6% of 658 times against 15.3% of 2,850 after a delivery, 2026-10-14 and 15), so later tries go to the residential browsers, and a daemon with none ends a refused search at once. `--fixed-retry-exits 1` (off by default) relaxes that exclusion for at most two failed static proxy tries per search (`static_failures < 2`) while still keeping direct host addresses (`--direct-exits`, `--direct-sources`) first-try-only and preferring free proven exits ahead of unproven static lines in retry windows, so a first try that lands on a flagged static ISP port can fail over onto one more healthy static ISP port before falling through to residential exits. Once a warm fixed exit has delivered at least once, or during its first two cold tries (`consecutiveBlocks < 2`), it takes every first try it can, whether or not its last try delivered: in the ISP trial of 2026-10-15 a line whose last try was a refusal it had not kept delivered 32 of its next 36 first tries, against 247 of 327 for the residential exits that took first tries while it waited; a never-proven fixed exit (`successes == 0`) with 2 or more consecutive refusals yields priority to warm rotating and proven exits so externally hot static lines on a fresh health ledger cannot starve rotating exits across 70 cold refusals before the 7-of-10 window cools them. A fixed exit also rests 3 hours when its first tries over the last hour delivered under `--fixed-decay-floor-pct` (0 to 99, default 85, 0 never), once it has 20 and at least 8 of them were refused: the box's own IPv4 delivered 98.7% of first tries on 2026-10-13 and was refused 4 times in 5 a day later, a slide the 7-of-10 window never sees. The 8 refusals keep a short streak from resting a good line, since 20 first tries is two minutes at 10 searches a minute: on 2026-10-16 a line rested for 4 refusals in its first 20 and delivered 168 of its next 172. Replayed over every ISP line's first tries from 2026-10-15 to 17, the 8 turn 10 rests into 3, and a line refused 1 time in 5 still rests, by about its 40th first try. A line with peers, the other fixed exits with 20 first tries in the last hour, rests only when it delivers 8 points or more under their median, or under 70% whatever they do. When every line dips at once, as all five did from 05:30 to 07:40 UTC on 2026-10-16 (84% to 91%), a rested line's first tries go to the residential exits, which delivered 73% of theirs that night, while the two left running delivered 92% and 94% of their first tries over the next two hours. Over the same replay this keeps the one rest that was a line's own (84% beside peers at 98%) and drops the two in that dip (84% and 85% beside 91% and 88%). A fixed exit refused on 8 first tries in a row, with no delivery between them, rests 1 hour, however long they took and whatever its peers do, and back from that rest 2 more refusals in a row rest it again: the 7-of-10 window cools a refusing line 10 minutes at a time, one first try a cooldown, so a line Google has flagged never reaches 20 first tries in an hour. On 2026-10-17 four ISP lines refused every first try (0 of 9 each) for an hour after the AA benchmark's agent queries and were never rested, each try holding its search until Google's window. A line delivering 85% refuses any 8 first tries in a row about once in 4 million, and 2 in a row 2.25% of the time. The rest is an hour, not three, because a burst of re-sent queries, which Google refuses from any exit, gave every working line its 8 within minutes the same night, and 3-hour rests left no line at all while plain queries delivered 20 of 20. A flagged line costs 2 refused first tries an hour. `--fixed-long-words` (3 to 32, default 0, off) makes a search of that many words or more a long one, as an agent's are, and a fixed exit takes a long search's first try at most once every `--fixed-long-interval-ms` (1000 to 3600000, default 60000). A long first try that finds every fixed exit paced goes to the residential browsers, and never waits for a proven exit it may not take; short searches keep the lines' own spacing. Ten ISP lines took short queries at 8 to 10 a minute each for 6 to 10 hours at 97% delivered (2026-10-16), while the AA benchmark's agent queries, long and often quoted, at about 3 to 5 a minute an address flagged five of them within minutes (2026-10-17). Each try records `longQuery`. Each try records `staticLine`, the line's number in the file from 1. The daemon refuses a file with a line it cannot read, a repeated line, or a line on a port that rotates its exit: a DataImpulse port, which is a session, or port 8000 of Oxylabs's self-service dedicated ISP gateway, `disp.oxylabs.io`, which gives a random address of the plan on each request where each port from 8001 holds one (Oxylabs's documentation, retrieved 2026-10-14), or the same port of `isp.oxylabs.io`, the gateway an account's dashboard gives for the plan. Its errors name the line's number, never its text, which carries a password. `--proven-interval-ms` (1000 to 10000, default 5000) spaces a proven exit, one whose last outcome was a delivery within the last five minutes, this far apart; every other exit keeps 10 s. At 10 s for all, a browser on a good exit sat idle while searches queued for one. An A/B on 2026-10-10 (four alternating runs of 100 searches): proven exits reused under 7 s apart delivered 86% of tries (84% at 10 s), the median wait for a browser before a try fell from 513 ms to 0 (p90 3.0 s to 1.1 s) and the median search from 6.3 s to 5.4 s, with blocked tries and bandwidth unchanged. Each attempt records the spacing its admission set as `egressIntervalMs`. `--proven-seconds` (60 to 3600, unset by default, which keeps 300) sets how long an exit stays proven after it delivers, for the slot preference, the spacing above and each try's `provenExit` alike. Replayed over the load tests' 31,636 tries to 2026-10-12, a first try delivered 76.6% of the time when its exit had delivered under a minute before, 74.2% at 5 to 15 minutes (31 tries), 43.8% at 15 to 60 (16) and 33.3% past an hour (33), against 29.4% for an exit never seen. Under load no first try met a gap of 1 to 5 minutes, so five minutes ends the preference just when sparse traffic could use it. The samples past five minutes are small: the setting is for a trickle test of 900 to 1800 s, not yet a default. `--race-width` (1 to 3, default 1) runs up to that many chains of tries for one search at once. A search fails only when every try that fits before its deadline is refused: a failed try costs about 2.3 s, so about 22 fit, and with 7.6% of long-tail tries delivering that predicted 17.6% failures against 16% measured (2026-10-11); across eight runs of the same 100 queries, misses per query followed chance, not the query. The extra chains take only a warm browser while no other search waits for one, so a busy daemon serves exactly as before. They share the exits already tried, the first delivery answers, and the others finish the try they are in and stop. They share the cap too: no chain starts a try once the chains have made `--max-attempts` between them, so a raced search ends at most two tries over it, the ones still running. Capped chain by chain, three chains at 40 could make 120 tries, more than the progress limit holds, and a history past that limit fails the search with its delivered page. Every chain's tries are in the answer's `attemptHistory`, tagged `raceChain`, with `raceWidth` and `raceChainsTried` beside them. `--retry-window-ms` (1000 to 30000, unset by default) changes what a search does once Google has refused it. Google lets a refused search through again about every 10 s: over 5,710 tries at production's setting on 2026-10-11 and 12, a later try admitted 10 to 11.5 s, 20 to 21.5 s or 30 to 31.5 s after the search's first delivered 21.9% of the time, and one admitted between those windows 4.6%. The gaps held 68% of tries and 64% of bytes but 28% of Google's answers. Live receipts the same night agreed, 9.2% against 2.8%, on browsers of the same age inside and outside the windows. With the flag, a refused search's later tries wait, holding no browser, for windows at its first admission plus the period, twice the period and so on, each 1.5 s long, and are admitted on the mark. `--retry-window-tries` (1 or 2, default 2) lets a window take a second try, no earlier than 0.8 s past the mark, when the first ends inside it; a window's tries never run together, which is how racing lost. `--retry-window-after-refusals` (1 or 2, default 1) lets a refusal on the first try or an unproven exit try one more exit immediately before windows begin, while a later refusal on a proven exit still starts windows at once. `--retry-window-from-refusal` (0 or 1, default 0) anchors each retry window a full `--retry-window-ms` period after the latest Google refusal finishes rather than from the first try's pre-navigation admission, so `probeBefore`, proxy TLS and immediate strikes do not shorten the 10 s quiet gap before `Window 1` or fire a second turn 0.8 s after a refusal inside a window; window turns are counted only after `probeBefore` succeeds, so an idle sticky session whose exit rotated (`ip_changed`) before Google is contacted yields immediately to the next warm slot without consuming `Window 1`. A failure that is not a refusal does not push the search to the next window, and a window that cannot fit before the deadline ends the search, leaving Bing the time. Each try records `sinceFirstAdmissionMs` and, when it aimed at a window, `windowTargetMs`. The flag needs `--race-width 1`. What an A/B of the two has to show is whether the answers found between windows today come back in the next window. `--window-proven-exits 1` (off by default; needs `--proven-first-wait-ms`) lets a later try that starts in one of those windows, every 10 s after the search's first admission and 1.5 s long, take a proven exit that is free, as a first try would, without waiting for one. In the windows a proven exit doubles a later try's chance: over the production-setting runs of 2026-10-12/13, later tries admitted in a window delivered 36.5% on a proven exit (52 tries) against 17.0% on another (802). Only 6% of window tries had one, since the proven first-try wait keeps proven exits for first tries, where they are worth more still (75.0% against 40.7%), so a window try never waits for one. The window is judged when the pool hands the try its browser, not when the try starts waiting for one, so a try that waits into a window takes a proven exit there and one whose wait outlasts a window does not. A try aimed at a window by `--retry-window-ms` takes its browser 0.5 s before the mark, before Google's window opens, and counts as that window's try; without that, the two flags together never gave a window try a proven exit. Each try records `windowProven`. `--keep-proven-exits 1` (off by default) keeps an exit that has delivered through its first failure in a row. After a refusal the exit and its browser stay, and the refusal's cooldown is cleared; after a timeout or an unreachable check the exit stays and the browser is relaunched on it. A second failure in a row moves the browser to a new exit, as without the flag. Without it, a refusal sends an exit that has been delivering to the back of the queue for at least 10 minutes and its browser hunts through fresh exits, each one a browser relaunch. In A/Bs beside production on 2026-10-13 (one browser per arm, 10 searches a minute, one try a search), a proven exit kept through one refusal delivered the next search 16 times in 25, where a fresh exit delivered 8 to 13%, and keeping proven exits took first tries from 60.7% to 82.6% (about 448 searches an arm). Each try records `keptExit`: `browser`, `exit`, or null when the slot moved on. `--spare-gap-refusals 1` (off by default) treats a later try's refusal between Google's windows (above), on an exit that has delivered, as the query's and not the exit's: it neither cools the exit nor counts toward `--keep-proven-exits`' second failure, the browser stays, and the exit's record changes only in when it was seen. Google holds a refused query whatever the exit: with both levers on beside production on 2026-10-14 (1,200 searches), later tries between windows delivered 7.6% on exits that had just delivered and 8.4% on the rest, and an exit kept through such a refusal delivered the next first try on it 31 times in 42. Yet two of them in a row were what swapped out 133 of the 179 exits that run dropped after delivering. An exit that never delivered is swapped and cooled as before, which keeps fresh exits coming for refused searches, and a search may take an exit it spared once more, in one of Google's windows, where later tries on proven exits delivered 17 of 26 in the same run. A first version spared every exit, and beside production its refused searches ran out of exits: 18 of 211 ended at their deadline after 5 to 7 tries, against 4 of 214 for the control. Nor does a refusal between windows give any exit its first verdict, which `--network-yield-floor-pct` judges networks by: in the same run 915 of the 1,145 fresh exits Google first answered met it between windows, where 79 delivered (8.6%), against 34 of 76 on a first try, so the floor was judging how often a network's exits were handed held queries. A refusal on a first try or in a window, and any failure that is not a refusal, counts as before. Such a try records `keptExit` as `spared`, and a try on an exit the search spared records `sparedRetry`. `--after-check` (`wait` or `background`, default `wait`) decides whether a delivered page waits for its post-capture exit check, the fresh probe through the same line that starts when the page settles. On 2026-10-11 that wait was about 0.5 s at the median, and the check passed 132 times out of 132. With `background` the validated page goes back at once (`probeAfterWaitMs: 0`, `exitCheckAfterReply: true`) and the check finishes on its own, keeping its browser until it settles, so the next try there still starts from a proven exit. A check that fails withdraws the exit's proof, so the next try probes it before Google, and the page already returned stands: what `background` gives up is the guarantee that that page came from its stated exit. Each try reports the daemon's running `afterReplyChecksPassed` and `afterReplyChecksFailed`. Measured the same night over 400 long-tail searches in alternating arms, `background` made the delivering try 0.50 s shorter at the median and 0.61 s at p90. The wait for a browser, tries, blocks and bytes did not change, and 1 of 162 background checks failed. Production runs `background` from 2026-10-11 23:46 UTC. Every try that gives up as `google_page_stalled` records `mainBodyFinished`: false when Google sent a page's header and never finished its document, true when a complete page simply showed no results. On long-tail queries on 2026-10-10, a stalled try held a page of 74 to 75 KB every time (121 tries) and cost 3.6 s before the 2.5 s grace ended it. `--stall-samples` (0 to 32, default 0) keeps the last N such pages for an hour in `stall-observations/` under the daemon root, beside a JSON report of the status, page size, hold time and `mainBodyFinished`, so the page can be recognized at once. The directory is private (mode 0700, files 0600, generated names) and is never mixed with `challenge-observations/`. A page can carry the query, so it stays off unless an operator is diagnosing stalls; each try that kept one names it as `stallSampleId`. Every try also records `requestKinds`: each request the page made and what the guard did with it (`allowed`, `cached`, `skipped`, or `refused:` for its host, method or type), as method, request type, a Google host name (any other host reads `other`) and at most two path segments, never a query, twelve kinds at most. `failedRequestKinds` counts allowed requests the browser then failed, by browser error code, and a stalled try records `pendingRequestKinds`, the allowed requests still unanswered when the grace ran out. Serving a stalled check page the one Google file it asked for left it stalled (2026-10-11), so these name what else it waits on. They showed what most stalls were: Google's quiet refusal of the results request. The check page passes, and the try's second document is a complete page titled for the query, with the search form and its empty containers but none of the results column (`#search`, `#rso`, `#rcnt`, `#center_col`, `#botstuff`); it asks only for a logging beacon and a font, as every delivered page does, and never moves on. Once that second document has finished arriving and the latest response is a success, the daemon ends the try after 1 s unchanged as `serp_blocked`, with `unrecognizedReason` `google_page_held` and `heldMs`, instead of waiting out the 2.5 s stall as a timeout. The exit then cools down like any other refusal (a timeout records nothing against it), and the operator monitor counts the try as a challenge. A first document, one still arriving, a page moving on, a page with a results column, or one titled only "Google Search" still stalls as before. `--held-grace-ms` (100 to 5000, default 1000) sets how long that page must stay unchanged. About one long-tail try in four ends this way, so the grace is paid often. A page that looked held and then turned into results is what a shorter grace would lose, so a try that recovers records `heldRecoveredMs`, how long its page looked held, and every try reports the daemon's running `heldPagesEnded` and `heldPagesRecovered`. Before a browser launches on a new exit, the daemon may pass over the exit's network and probe another (at most twice a pick, about 0.8 KB a probe). By default a network is passed over when, over 30 or more of its tries, it delivered under half as often as the pool, except for one exit in five. `--network-yield-floor-pct` (0 to 50, default 0, off) judges a network by Google's first verdict on each of its fresh exits instead, a delivery or a block, so one proven exit delivering again and again cannot lift it: a network whose fresh exits delivered under the floor, over 5 or more of the exits the daemon remembers, is passed over except for one exit in ten. Replayed over 17,773 fresh long-tail exits from 2026-10-10 to 12, a 5% floor would have passed over 24% of them, which delivered 4.6% of the time against 12.0% for the rest. A try Google redirects away from its own search ends as `serp_redirect_blocked` (7.6% of long-tail tries in one run on 2026-10-11). It records `redirectDrift`, which says what moved and never a value: its `source` (the navigation request, its response, the page URL, or the proxy's CONNECT guard), the target's host class and first path segment, whether a fragment appeared, how many `q` parameters there are and whether the query matches (exactly, ignoring case, or containing ours, with the change in word count), whether `start` matches, and the names of query parameters added or removed against ours. When the redirect leaves `www.google.com` for another host, the slot moves to a new exit at once and the exit cools for an hour, as after a block, though it counts as no challenge. Every redirect read on 2026-10-11 (103 tries in two runs) came from one exit that Google sent to `www.google.com.hk/url`, its country-domain redirect, which follows the exit rather than the query; the slot that kept that exit failed every search that reached it. A redirect that stays on `www.google.com` keeps its exit, since another query or offset may be the query's own. `--goto-resolver` (`http2` or `browser`, default `browser`) chooses who asks Google where each result link points. `http2` is the Python HTTP/2 client described above, from the same exit, and sends no cookies or Referer. `browser` uses the page's own `fetch` with a manual redirect, `credentials: "same-origin"` and no Referer, so the lookups carry Firefox's TLS and reuse the non-anonymous HTTP/2 connection that loaded `/search` (rather than partitioning into `LOAD_ANONYMOUS` and opening extra upstream proxy `CONNECT` tunnels: 11 vs 13 `connects` per capture and `326 ms` vs `586 ms` 2-link redirect median in a 16-query Latin-square on one warm residential exit on 2026-11-03, `16/16` delivered with 0 holds); the daemon reads `Location` from the response the browser reports, and the publisher is never requested. Each attempt records `gotoResolver`. An A/B on 2026-10-09 found no difference in how often Google blocked an exit's next search after either, so TLS on the lookups is not a block signal; `browser` is the default because it skips the lookup client's own tunnel and handshake, about 10 KB of residential traffic per delivered page (296 KB against 306 KB for a delivering attempt). When `--goto-respect-num 1` targets 1 or 2 initial links under `--goto-refresh-max-ms` or `--goto-pace-ms`, `render_search` also starts that zero-pace burst once during DOM stability settling (never cancelling or restarting in-flight tokens, cutting 2-link `redirects` phase median to `251 ms`). `--goto-refresh-max-ms` (0 to 60000, default 0, off) lets the `browser` and `http2` resolvers wait out Google's hold on a result link. On 2026-11-02 the first two or three links on a page answered a 302 and each further one a 200 "Verifying your request" page whose meta refresh names a new `/goto` link and a wait that grew link by link: 3, 4, 5, 7, 12, 22 and 24 s. Once its wait was up, a plain fetch of the new link answered the 302, with no cookies and no script (34 of 34 across six pages). Off, such a link fails the search as `serp_invalid_response`, which made every served page fail. On, navigation and page settling keep their 8 s attempt ceiling, link lookups wait until the page has settled so streaming cards do not cancel in-flight holds, and a settled page extends its redirect budget by `--goto-refresh-max-ms` within the search's deadline. Each lookup reads the 200's body, waits the named seconds plus half a second, and fetches the link it names, up to two holds per link; a hold whose waits add up past the cap, or that cannot end before the search's deadline, is refused at once and fails the search as before. A page's links wait side by side, so its slowest hold sets the cost: 22 to 24 s on a page of ten links, longer than the 18 s `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` production ran on 2026-11-02, so turning this on means setting at least `30000` and raising that share and the caller's budget past 35 s. Each try records `gotoHolds`, `gotoHoldsRefused` and `gotoHoldWaitMs`, the longest wait one link was given. `--goto-max-links` (1 to 10, default 10), `--goto-respect-num` (0 or 1, default 0) and `--goto-pace-ms` (0 to 2000, default 0) shape how many discovered `/goto` links a capture unwraps and how 3-to-5-link lookups are spaced. On 2026-11-03 a 49-page, 7-arm Latin-square sweep (`n = 7` per arm across 4 warm residential Camoufox slots, `294` `/goto` lookups, `49/49` delivered) mapped Google's per-impression `/goto` unlock curve (shared across exits, browser contexts and document navigations for a single SERP impression; `94 to 120`-byte AEAD protobuf tokens, zero publisher URLs in HTML, `<script>` blocks or XHRs): links 1 and 2 answer instant 302s (`~0.64 to 0.80 s` total SERP latency on warm slots), links 3, 4 and 5 unlock non-uniformly at `~0.50 s`, `~1.05 s` and `~2.30 s` after page settle (`14/14` zero-hold 302s on links 3 and 4 at `550 ms` step, `12/14` zero-hold on link 5 at `750 to 850 ms` step where the `14%` held tail pushed 5-link `p90` to `6.25 to 6.54 s`, and `7/7` zero-hold on link 5 past `2.65 s` at `3.15 s` unwrap p50 / `3.23 s` p90), link 6 hits a `~7.6 s` server-side hold (`49/49 = 100.0%` held across every arm from `0` to `1000 ms`, `8.03 to 8.79 s` unwrap / `~9.63 s` capture p50), and links 7 through 10 hit a `~24 s` wall-clock unlock ceiling regardless of spacing. `--goto-max-links` caps unwrapped `/goto` links per capture; `--goto-respect-num 1` bounds that cap by the search URL's `num` parameter (unwrapping only `num` links upfront when `num < 5` and pulling spare links up to the 5-link ceiling on demand only when an earlier result resolves to a doorway page, while `num=5` caps strictly at 5 links rather than spilling into link 6's `7.6 s` hold, clamped to `--goto-max-links`); and `--goto-pace-ms` scales the non-uniform per-index unlock schedule `(0, 0, 550, 1100, 2650) ms` by `pace_ms / 1000` when 3 to 5 links are unwrapped, while leaving 1-to-2-link and 6-to-10-link captures unpaced so neither pays extra delay. Each try records `gotoDiscovered`, `gotoSelected`, `gotoSpareUsed` and `gotoPaced`. When `--goto-refresh-max-ms` or `--goto-pace-ms` is active, non-first-try attempts and `--retry-window-ms` window sleeps reserve the expected post-settle `/goto` unwrap time (`minimum_unwrap_seconds`: 24.75 s for 7-to-10-link pages under a 25 s+ hold cap, 8.75 s for 6-link pages, 2.90 s for 5-link pages, 1.35 s for 4-link pages, and 0.80 s for 3-link pages at `--goto-pace-ms 1000`, 5.75 s for unpaced 5-link pages, and 0 s for 1-to-2-link pages) on top of the 2.5 s try minimum (27.25 s total for `num=10`, 5.40 s for paced `num=5`), breaking immediately instead of sleeping 10 s into a retry window that cannot finish unwrapping `/goto` links before the deadline. That reserve lets a 20 s `maxResults=5` budget and a 42 s `maxResults=10` budget each fit one post-hold 10 s retry window and end failed searches at ~12.3 to 12.8 s rather than burning 32 to 57 s across doomed late windows. `--ab-arm-b` and `--ab-salt` run an A/B search by search rather than in blocks of runs. On 2026-10-12, runs of 100 searches on one unchanged setting swung from 68 to 93 Google answers, as much as any setting tested, so an A/B that alternates runs measures the hour as much as the setting. With both flags, each search falls in arm a, the daemon's own settings, or arm b, those given to `--ab-arm-b` on top, by a hash of the salt and its query with each run of whitespace as one space. Both arms then share every minute, the pool and its exits; a report can tell any search's arm from its query alone; and a new salt each run moves about half the queries to the other arm, so over several runs each query is seen in both. Arm b takes only settings a search reads for itself: `--max-attempts`, `--max-block-attempts`, `--proven-first-wait-ms`, `--held-grace-ms`, `--retry-window-ms`, `--retry-window-tries`, `--retry-window-after-refusals`, `--retry-window-from-refusal`, `--window-proven-exits`, `--keep-botguard-cookies`, `--warm-google-session`, `--goto-refresh-max-ms`, `--goto-max-links`, `--goto-respect-num`, `--goto-pace-ms` and `--fixed-retry-exits`, with the same bounds as the daemon's own; windows in arm b need `--race-width 1`. The arms still share one pool, so the proven first-try wait, which changes which browser a search takes, and retry windows, which free a browser between windows, are measured beside the other arm's searches rather than in a pool of their own. Each answer records its arm as `abArm`. `--browser-engine` (`camoufox` or `chrome`, default `camoufox`) and optional `--chrome-path` let the daemon run stock Chrome or Chromium instead of Camoufox behind the same socket protocol, guarded local proxy, exit-health record and supervisor checks. When `DISPLAY` is unset on Linux, the pool starts one private `Xvfb` display (`1366x768x24`, `-nolisten tcp`, discovered through `-displayfd`) shared across its slots and restarts it if it dies. Each slot spawns Chrome in its own process group with a private `vitna_chrome_slot-*` profile and `HOME` directory under `TMPDIR`, `--remote-debugging-port=0` on `127.0.0.1`, `--disable-blink-features=AutomationControlled`, QUIC and non-proxied WebRTC UDP disabled, and `--host-resolver-rules` mapping every non-loopback hostname off-host so DNS and TCP stay inside the slot's `StandingGuardedProxy` (`127.0.0.1:0` with `require_auth=False`, since Chrome does not take proxy credentials in `--proxy-server`). Playwright attaches over loopback CDP (`connect_over_cdp`) and reuses the slot's driver across relaunches when `--fast-relaunch 1` is set. On 2026-11-02 Google's BotGuard refused CDP `browser.new_context()` (incognito context with `1280x720` device-metrics overrides) and `context.route_web_socket()`'s injected `WebSocket` wrapper script, so `ChromePageContext` opens a fresh tab on `contexts[0]` with page-level request routing and clears cookies and Google origin storage (`Storage.clearDataForOrigin`) when the tab closes, preserving Google's challenge-pass cookies (`__Secure-STRP`, `AEC`, `NID`, `SEARCH_SAMESITE` and legacy `SG_SS`; on 2026-11-03 Google retired `SG_SS` in favor of `__Secure-STRP` + `NID` + `SEARCH_SAMESITE`, and retaining the full set per verified exit IP collapsed cold 2-hop BotGuard redirects from `1.03 s` to a single `200 /search` hop in `0.64 to 0.72 s` across `49/49` warm Camoufox captures) when `--keep-botguard-cookies 1` is enabled. With `--warm-google-session 1` (off by default), each stock Chrome slot primes `https://www.google.com/?hl=en` through its guarded exit on launch and retains Google cookies (`NID`, `AEC`, `__Secure-STRP`, `DV`, `SEARCH_SAMESITE`) and origin storage across delivered captures on that same exit IP, wiping cookies and `Storage.clearDataForOrigin` immediately on any block, capture failure, cold A/B arm or session rotation; each try records `warmSessionApplied`, `warmSessionRetained`, `googleSessionWarmup` and `googleSessionWarmupMs`. Because Chromium over CDP suppresses Playwright `page.on("response")` events on manual opaque redirects, `BrowserRedirectClient` reads `3xx` `Location` headers from a page CDP session (`Network.responseReceivedExtraInfo` and `Network.requestWillBeSent` `redirectResponse`) while waiting out 200 refresh holds under `--goto-refresh-max-ms`. On the OVH host on 2026-11-02, a 26-slot warm Chrome pool (10 static ISP lines plus 16 sticky residential lines) plus `Xvfb` warmed with `--warm-google-session 1` in 13.0 to 15.6 s using 7.98 to 8.05 GB of physical memory (`Pss`, about 296 to 298 MB per slot), `--fast-relaunch 1` started replacement Chrome browsers in 274 to 519 ms, and each try records `browserEngine` (`camoufox` or `chrome`). In sequential 50-query `maxResults=10` runs at concurrency 3 (`--goto-refresh-max-ms 30000`, `timeoutMs=58000`), cold Chrome delivered 30/50 (60.0%, 13/50 first-try, 17/37 retry recovery, 290 holds waited and 12 refused; delivered-only latency 36.4 s p50 and 56.6 s p90, 25.9 s first-try p50), whereas two runs adding `--warm-google-session 1` and `--fixed-retry-exits 1` delivered 33/50 (66.0%, 28/50 first-try) and 42/50 (84.0%, 35/50 first-try, 7/15 retry recovery, 345 holds waited and 2 refused; delivered-only latency 25.9 s p50 and 35.4 s p90, 25.6 s first-try p50), or 75/100 (75.0%, 63/100 first-try) pooled across two 50-query runs of the same query list. In standalone 100-distinct-query runs on separate fresh 26-slot pools at concurrency 3 (`--warm-google-session 1 --fixed-retry-exits 1 --goto-respect-num 1 --goto-pace-ms 1000 --goto-refresh-max-ms 30000`), `maxResults=10` (`timeoutMs=58000`) delivered 79/100 (79.0%, 95% Wilson CI 70.0 to 85.8%; 66/100 first-try, 13/34 retry recovery; quartiles Q1 21/25, Q2 20/25, Q3 15/25, Q4 23/25; unconditional all-100-query latency 26.1 s p50, 47.7 s p75, 52.9 s p90; delivered-only latency 25.7 s p50, 37.3 s p90; 651 holds waited and 6 refused), and `maxResults=5` (`timeoutMs=35000`, started immediately after on an empty health ledger while the 10 static lines were hot: 0/9 static ISP vs 72/91 = 79.1% residential first-try) delivered 76/100 (76.0%, 95% CI 66.8 to 83.3%; 72/100 first-try; quartiles Q1 6/25 while cooling the 10 hot static lines, then Q2 23/25, Q3 23/25, Q4 24/25 on warm residential slots; 0 `/goto` holds waited or refused across all 76 delivered pages; unconditional all-100-query latency 5.74 s p50, 28.2 s p75, 32.3 s p90; delivered-only latency 4.06 s p50, 6.18 s p90, 4.03 s first-try p50). Failed queries run to deadline, so unconditional all-query p90 remains deadline-bound whenever 10% or more of queries fail. To remove hour-to-hour and pool-draw confounding, a simultaneous two-period AB/BA crossover across two 13-slot daemons (40 paired `maxResults=10` queries dispatched in lockstep and swapped across both pools) delivered 17/40 (42.5%, 10/40 first-try, 25.8 s delivered p50) with `--warm-google-session 1 --fixed-retry-exits 1` against 4/40 (10.0%, 0/40 first-try, 37.4 s delivered p50) on cold Chrome (13 vs 0 discordant delivered pairs, two-sided McNemar `p = 0.00024`; 10 vs 0 discordant first-try pairs, `p = 0.0020`). On a fresh 26-slot pool running 60 `maxResults=5` queries (`timeoutMs=45000`) split query-by-query via `--ab-arm-b="--goto-pace-ms 1000"`, overall delivery was 50/60 (83.3%, 36/60 first-try): most of the latency drop relative to `maxResults=10` comes from Google rendering 5 `/goto` links instead of 10 (avoiding the ~24 s hold ceiling on links 7 through 10), while `--goto-pace-ms 1000` cut first-try p50 from 6.80 s (29/32 delivered, 88 holds waited, median 3 holds per page) to 4.09 s (21/28 delivered, `p = 0.17` vs unpaced; 0 holds waited across all 21 delivered pages; 4.30 s vs 7.12 s delivered p50, and 5.02 s vs 7.14 s unconditional all-query p50). Follow-on batches on an already-pruned pool delivered 19/20 (`maxResults=5`, 4.39 s delivered p50) and 14/15 (`maxResults=2`, 2.76 s delivered p50, 0.88 s min, where Google already renders only 2 `/goto` links so pacing and `num` capping are no-ops). Follow-up 100-query fresh-pool runs with `minimum_unwrap_seconds` and `_fixed_ready` measured tighter deadlines across all intermediate trials (each wiping `egress-health.json` while the 10 static lines were already cooling): cutting to zero retry windows (`12 s` `maxResults=5` and `35 s` `maxResults=10` on re-sent query sets) ended failures in 2.8 to 3.0 s p50 (5.2 s and 26.9 s unconditional p90) but dropped delivery to 12/100 and 28/100 (rising to 17/20 in the final 20 `maxResults=10` queries once proven residential exits stabilized), while retrying between windows (`15 s` / `35 s`) recovered only 5/86 (5.8%) and 1/62 (1.6%) inside Google's 10 s query hold (19/100 and 39/100 overall, Q4 25/25 on `maxResults=10`; reverted), and an asymmetric `20 s` (1-window) / `35 s` (0-window) re-run on reused sticky ports delivered 35/100 (`maxResults=5`, 12.1 s unconditional p50, 13.8 s p90, 4.39 s delivered p50, 12.8 s failed p50) and 3/100 (`maxResults=10` at 44.3 queries/min closed-loop retry flood). Sizing both budgets for one post-hold 10 s retry window plus unwrap reserve (`20 s` `maxResults=5` via `VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS` and `42 s` `maxResults=10` via `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS`, disjoint sticky ports) delivered 69/100 on `maxResults=10` (69.0%, 95% CI 59.4 to 77.2%; 67/100 first-try, 0/10 hot static ISP cleared in 10 tries, 67/90 = 74.4% residential first-try; Q1 1/25, Q2 18/25, Q3 25/25, Q4 25/25, or 60/60 first-try across deciles D5 to D10; 0 holds refused; unconditional all-100 latency 26.1 s p50, 27.6 s p75 vs 47.7 s at 58 s, 27.8 s p90 vs 52.9 s at 58 s; delivered-only 27.4 s p50, 28.0 s p90; failed-only 12.8 s p50, 14.4 s p90 vs ~53 s at 58 s) while the repeatedly re-sent `maxResults=5` set delivered 7/100 (`2.5 s` proven spacing) and 2/100 (`10 s` proven spacing; failed-only 12.3 to 12.5 s p50). On previously unseen 100-query sets (`bench_query(1000..1099)` and `bench_query(1100..1199)`), `maxResults=5` at `20 s` delivered 48/100 overall (48.0%, 95% CI 38.5 to 57.7%; Q1 0/25 and Q2 5/25 during cold residential warmup, then Q3 21/25 = 84% and Q4 22/25 = 88%, or 43/50 = 86.0% first-try across the warmed second half; 0 `/goto` holds waited or refused; unconditional all-100 latency 11.7 s p50, 12.4 s p75 vs 28.2 s at 35 s, 13.6 s p90 vs 32.3 s at 35 s; delivered-only 4.49 s p50, 5.11 s p90; failed-only 12.3 s p50, 14.9 s p90) and `maxResults=10` at `42 s` (started 30 s later on a fresh cold pool) delivered 6/100 (6.0%, 95% CI 2.8 to 12.5%; Q1 0/25, Q2 0/25, Q3 1/25, Q4 5/25; 25.9 s delivered p50; 12.5 s failed p50, 15.5 s unconditional p90). Two follow-up unseen fresh-pool pairs with `--retry-window-from-refusal 1` and `--retry-window-after-refusals 1` first exposed `probeBefore` (`ip_changed` on idle sticky sessions) consuming `--retry-window-tries 1` window turns before Google was contacted (`bench_query(1200..1299)` at `22 s`: 28/100, Q1 0/25, Q2 0/25, Q3 12/25, Q4 16/25, 4.01 s delivered p50, 13.3 s failed p50, 22 `serp_unreachable`; `bench_query(1300..1399)` at `44 s`: 0/100, 13.1 s failed p50, 42 `serp_unreachable`), and after moving window-turn accounting past `probeBefore` (`bench_query(1400..1499)` at `25 s` and `bench_query(1500..1599)` at `46 s`) reduced `serp_unreachable` to 0/100 in both runs while delivering 2/100 (`D10` 2/10, 4.34 s first-try, 12.9 s failed p50, 16.0 s unconditional p90) and 0/100 (13.6 s failed p50, 16.3 s unconditional p90), confirming that one-window budgets bound failed-query tail latency to ~12.3 to 13.6 s p50 (~13.6 to 16.3 s p90) with zero `/goto` holds refused, while cold-pool runs with `egress-health.json` wiped and all 10 static ISP lines externally hot (~1 to 2% fresh residential discovery rate per cold IP vs 64 to 100% once proven) bifurcate sharply by how early 2 to 3 clean residential exits are discovered. A simultaneous paired head-to-head benchmark across two symmetric 13-slot cold pools (`5` randomly split static ISP lines plus `8` disjoint sticky residential ports per daemon, `egress-health.json` empty on both, concurrency 3 per daemon, `200` previously unseen distinct queries `bench_query(1600..1799)` paired by topic and coin-flipped per pair so zero query strings overlapped across engines; even pairs `maxResults=5` at `25 s`, odd pairs `maxResults=10` at `46 s`; shared `--goto-refresh-max-ms 30000 --goto-respect-num 1 --goto-pace-ms 1000 --fixed-retry-exits 1 --retry-window-ms 10000 --retry-window-from-refusal 1 --retry-window-after-refusals 2 --retry-window-tries 2 --window-proven-exits 1 --keep-proven-exits 1 --spare-gap-refusals 1 --proven-first-wait-ms 1000 --proven-interval-ms 2500 --fast-relaunch 1`, plus `--warm-google-session 1` on stock Chrome) isolated cold-pool exit discovery from hour-to-hour variance: Camoufox delivered 68/100 (68.0%, 95% Wilson CI 58.3 to 76.3%; 54/100 first-try, 4/15 static ISP and 50/85 = 58.8% residential first-try; 14/46 = 30.4% retry recovery across 190 tries and 40 window sleeps; quartiles Q1 14/25, Q2 14/25, Q3 21/25, Q4 19/25, or 40/50 = 80.0% with 95% CI 67.0 to 88.8% and 37/50 = 74.0% residential first-try across the warmed second half; 29 `serp_blocked` and 3 `serp_timeout`; unconditional all-100 latency 16.6 s p50, 26.1 s p75, 27.5 s p90; delivered-only 25.7 s p50, 28.0 s p90; failed-only 15.4 s p50, 17.9 s p90) against 0/100 (0.0%, 95% CI 0.0 to 3.7%; 0/6 static ISP and 0/94 residential first-try across 305 tries and 90 window sleeps; 99 `serp_blocked` and 1 `serp_timeout`; failed-only 14.3 s p50, 17.2 s p90) on stock Chrome (68 vs 0 discordant topic pairs, two-sided exact McNemar `p = 6.8e-21`). Within Camoufox's 100 queries, `maxResults=5` (`n = 50`, `25 s`) delivered 33/50 (66.0%, 95% CI 52.2 to 77.6%; 25/50 first-try at 4.85 s p50 and 5.60 s p90, 8/25 = 32.0% retry recovery; 0 `/goto` holds waited and 0 refused; unconditional latency 6.63 s p50, 16.5 s p75, 18.7 s p90; delivered-only 5.17 s p50, 18.6 s p90; failed-only 15.6 s p50, 19.7 s p90) and `maxResults=10` (`n = 50`, `46 s`) delivered 35/50 (70.0%, 95% CI 56.2 to 80.9%; 29/50 first-try at 26.3 s p50 and 27.8 s p90, 6/21 = 28.6% retry recovery; 280 `/goto` holds waited and 0 refused; unconditional latency 26.1 s p50, 26.8 s p75, 39.2 s p90; delivered-only 26.4 s p50, 39.8 s p90; failed-only 14.2 s p50, 16.7 s p90). Both engines exited with zero surviving processes or leftover directories (Camoufox 4.65 GB PSS across 13 slots, 625 ms fast-relaunch p50; stock Chrome 4.24 GB peak PSS across 13 slots, 306 ms fast-relaunch p50); `camoufox` remains the default engine because its cold residential discovery rate held up where stock Chrome's collapsed, while `--browser-engine chrome` stays opt-in. #### Proxy traffic per capture Residential exits are billed per GB. On 2026-10-08 a delivering attempt moved about 1.1 MB through its exit, 952 KB of it Google's script bundles (`www.google.com/xjs/`), OneGoogle bar (`www.gstatic.com/og/`) and fonts, all cacheable for a year and downloaded again by every fresh context. Organic results are in the HTML, so by default the daemon does not load those assets; BotGuard (`/js/bg/`) and everything else still load. A delivering attempt then moves 165 to 187 KB, and 23 such captures each delivered ten organic results. `--load-static-assets` restores them. Whatever static assets still load are kept in a cache shared by the slots (`--static-cache-mb`, default 96, 0 to disable): public, long-lived, cookie-free responses only, keyed by exact URL. A challenged attempt moves about 62 KB. ## Alternative retrieval experiments On 2026-10-05, an isolated Python environment on the owned OVH host ran Google-only SearXNG at [revision `d48c4b555421e824342c51d68482dd0898e54d0f`](https://github.com/searxng/searxng/tree/d48c4b555421e824342c51d68482dd0898e54d0f), bound to loopback. Its `/wml/search` request used `curl_cffi` with a Nokia user agent. All three measured paths received Google HTTP 403; SearXNG returned API HTTP 200 with zero results and an access-denied report: | Tested outbound path | Observed latency | |---|---| | Host default connection | About 421 ms | | Fixed owned IPv6 | About 371 ms | | Private SSH CONNECT relay through the PC's own network | About 645 ms; one relay connection transferred 8,641 bytes | The host worked, but these samples did not establish unchallenged Google retrieval. They used no third-party search service, Custom Search fallback, or production provider integration. 4get's pinned [Google implementation](https://git.lolcat.ca/lolcat/4get/src/commit/03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae/scraper/google.php) and [configuration guide](https://git.lolcat.ca/lolcat/4get/src/commit/03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae/docs/configure.md) at revision `03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae` require a separate 4play Firefox service for Google; the 4get Docker container alone is insufficient. This alternative was reviewed, not installed or live-tested. Whoogle's [official end-of-life announcement](https://github.com/benbusby/whoogle-search#whoogle-has-reached-the-end-of-the-road--24-jul-2026) dated July 24, 2026 says it no longer returns results: Google blocked its last working non-JavaScript user agent, and the Custom Search fallback was unworkable. The repository was archived on GitHub on August 14, 2026. Whoogle was reviewed only, not installed or live-tested. The reviewed GitHub topic entries largely required paid-provider accounts. An independent candidate, [omkarcloud/google-scraper at revision `11a5562251a8c330475a1fd312e5f572db85d7e8`](https://github.com/omkarcloud/google-scraper/tree/11a5562251a8c330475a1fd312e5f572db85d7e8), uses a Camoufox Firefox session to obtain cookie identity, then replays requests with `curl_cffi`. Its optional paid solver and proxy services were excluded from this investigation. An earlier separate private experiment installed Camoufox 0.5.7, Playwright 1.62, and browser build 156 beta 34 on the owned OVH host, with verified cgroup process cleanup. A fresh direct headless capture for `site:bis.gov H200 January 2026 license review policy` returned HTTP 200 without a detected challenge in 36.237 seconds, including browser startup. Four BIS organic results were verified with the existing `parseSerpHtml` parser, canonical search identity, and single-site guard. After an admission gap, a separate direct Python SSL diagnostic made header-only GETs to the four observed opaque Google `/goto` links. All returned HTTP 302 destinations on `www.bis.gov` or `media.bis.gov`, with 423 ms of actual network time. It followed no publisher destination and closed every connection before recording proof. This step used no Firefox cookies, user-agent spoofing, or third-party provider. These were separate admitted diagnostics, not an end-to-end API latency measurement or production integration, and do not establish sustained reliability. The browser diagnostic checks identity only at the final search URL and accounts for bytes after response bodies complete; these limitations apply only to the private experiment. ## Configure proxy routes Existing URL strings retain their fixed credentials. A structured entry accepts `url`, optional `egressId`, and optional `session` with `username` and/or `password` credential templates. For example, this placeholder configuration requires the operator's provider to support the illustrated username syntax: ```json [ { "url": "http://proxy.operator.example:8000", "egressId": "shared-exit", "session": { "username": "ACCOUNT-session-{session}", "password": "OPERATOR_PASSWORD" } } ] ``` Store the array in private `VITNA_SEARCH_SERP_PROXIES`. Credential templates are literal, unencoded strings; Vitna encodes the resulting URL credentials. At least one supplied template must contain `{session}`; other placeholders are refused. Each admitted attempt generates one cryptographically random 32-character lowercase hexadecimal token and substitutes that same token everywhere `{session}` occurs. That attempt uses one generated proxy URL throughout its isolated browser context. A later attempt generates a new token. A supplied template replaces its entire credential field; fields without a template keep the URL's static credential. Template syntax and the provider's session stickiness remain operator responsibilities. `egressId` is an operator declaration that routes share a physical exit. It accepts 1 to 32 lowercase letters, digits, and hyphens, starting with a letter. `direct`, `browser-direct`, and `proxy-N` names are reserved. Different proxy URLs with the same label share per-engine pacing, concurrency, and block cooldown in the service process. Declare the shared label when different endpoints use the same exit; separate route labels do not establish separate physical exits. Replicas still have no shared coordinator. Routes receive safe `proxy-1` through `proxy-20` identifiers in array order. Duplicate detection compares the endpoint and effective fixed credentials or templates. Changing a credential that its template overrides, or assigning another egress label, does not make the same effective route distinct. Every attempt uses an isolated context without ordinary browser state, and a configured proxy pool has no implicit local-egress fallback. ## Validate an operator route Run the explicit live checker after enabling SERP and configuring the intended runtime and egress routes: ```bash node --env-file=.env.search platform/search/serp-check.mjs --engines google,bing --rounds 1 --timeout-ms 10000 ``` This command performs actual engine HTTPS/browser requests. It is separate from startup, readiness, and the local fixture integration test. By default it checks both engines once, using three fixed public queries per engine: six fresh JSON retrieval requests. Each request still has its configured route attempt budget. `--engines` can select `google`, `bing`, or both; `--rounds` accepts 1 to 3, bounding the run to at most eighteen samples. `--timeout-ms` accepts 100 to 60,000 and is clamped to the configured SERP timeout. Omitting it uses that configured timeout. Before another fresh sample, the checker respects the observed `nextRequestAt` and can wait up to sixty seconds for pacing. A sample refused with `serp_routes_busy` or `serp_routes_throttled` started no capture; the checker waits that refusal's interval and retries the same sample, within sixty seconds of such waits and sixty retries per sample. A refusal needing a longer wait remains a failure. These client-side waits are separate from the sample's latency measurement, and each retry has its own per-probe deadline. The checker does not sleep through cooldowns or retry failed samples; those outcomes remain failures in its report. The command writes a JSON report to stdout with `ok`, `samples`, `delivered`, `successRate`, latency percentiles `p50Ms` and `p95Ms`, actual route-attempt count `attempts`, safe per-route counts in `routeAttempts`, the configured `transport`, and numeric route count `configuredRoutes`. Its `engines` entries report those sample/delivery/latency counts and failure-code totals per engine. `successRate` is a fraction from 0 to 1. `samples` counts logical retrieval requests; `attempts` includes the actual backend attempts and retries. Queries, search URLs, result content, HTML, credentials, and proxy addresses are omitted. Save stdout to `.search-data/serp-check.json` if an operational record is useful. Configuration or invocation failures return a safe `ok: false, error` report. Exit code 0 means every requested response was delivered; exit code 1 means a request or the setup failed. A successful bounded run establishes outcomes for its sampled requests and configured routes, without establishing whole-engine coverage or future availability. ## Diagnose a browser route The optional standalone browser diagnostic submits one fixed public Google query through the current connection, using an isolated browser context. Choose one configuration for each invocation: ```bash node platform/search/serp-browser-check.mjs --channel=msedge node platform/search/serp-browser-check.mjs --channel=msedge --headed --navigation=form ``` The default is the `native` profile, headless mode, direct search navigation, and a 12,000 ms timeout. `--channel=msedge` or `--channel=chrome` selects an installed browser; otherwise the configured SERP or Unblocker channel is used, then bundled Chromium. `--headed` opens an automated browser window. `--navigation=direct` opens the search URL; `--navigation=form` opens the Google homepage and submits its visible search form. Both use one deadline. `--timeout-ms` accepts 1,000 to 20,000, using the equals form, such as `--timeout-ms=20000`. `--profile=cloakbrowser` uses an already installed runtime selected by `VITNA_SEARCH_SERP_BROWSER_PATH`, or its Unblocker path fallback. The command does not install a browser. It refuses a nonempty `VITNA_SEARCH_SERP_PROXIES` pool; use the proxy route diagnostic below for that comparison, or the retrieval checker above to validate production route selection. This diagnostic makes no automatic retry and runs no challenge solver. Every configuration shares persisted Google/current-connection health in `.search-data/serp-browser-check-state.json`, protected by an exclusive `.lock` file. `--state=PATH` can select another state file; use the same path when comparing profiles, headed mode, or navigation. A cooling state refuses the probe before browser launch. The first observed block starts a five-minute cooldown; repeated blocks double it up to one hour. A successful capture resets the block streak. Changing browser settings does not bypass this shared cooldown, and a concurrent diagnostic is refused while its lease is held. Healthy diagnostic starts durably reserve an interval selected uniformly between five and fifteen seconds before capture. The state records `nextRequestAt` across CLI invocations, browser profiles, headed or form modes, and generated session tokens. An early repeated probe returns `serp_routes_throttled` with zero attempts; a cooling route returns `serp_routes_cooling`. Neither diagnostic waits for pacing or automatically retries. State version 2 adds this pacing record; version 1 files migrate automatically while preserving their block history and cooldowns. The file guard coordinates these diagnostics on its local filesystem. Ordinary production routes use separate process-local coordination; the experimental owned adapter can share this durable gate when its state path and declared egress identity match. A crash or a failed state save can leave the lock in place. Repair the state storage and confirm that the diagnostic has ended before clearing an abandoned lock. A failed save returns `serp_probe_state_invalid` and prevents another invocation from trusting an older cooldown record. The command writes a JSON report to stdout with `ok`, `engine`, `profile`, `headed`, `navigation`, `attempts`, and safe observed `stages`. A failure includes a fixed `error` code. `latencyMs` is included after an attempted capture; `count` is included only on success. Available state is reported as numeric `consecutiveBlocks`, `cooldownUntil`, `nextRequestAt`, and `retryAfterMs`. Queries, URLs, result text, HTML, browser paths, and credentials are omitted. Exit code 0 indicates a parsed capture; exit code 1 indicates refusal or failure. `navigationResponses` retains at most eight main-frame HTTP observations, with only a fixed destination category, status code, and challenge flag. It records challenge responses before an immediate abort closes the page. When a page was opened, `network` reports request lifecycle counts and fixed method, resource-type, and failure categories. Each count is capped at 1,024; `truncated` marks overflow. These observations help identify failed POSTs or redirects without retaining request URLs, bodies, cookies, or headers. A comparison also requires the operator to make the same search manually in an ordinary unsigned-in browser on the same connection. An automated headed run is not that manual baseline. Automated failures alone do not establish that the connection's IP address is blocked. ## Compare configured proxy routes The proxy route diagnostic uses the same bounded browser probe on the configured proxy list: ```bash node --env-file=.env.search platform/search/serp-route-check.mjs --channel=msedge --navigation=form ``` By default it submits one fixed public Google query per configured proxy, up to twenty routes. It performs the probes sequentially and waits for each browser to close before the next route. It does not retry a failure or wait through pacing or a cooldown. With no configured proxies, it returns `serp_proxy_not_configured` before launching a browser. It does not add an implicit current-connection probe. An explicit selection can include that baseline: ```bash node --env-file=.env.search platform/search/serp-route-check.mjs --routes=direct,proxy-1 --channel=msedge --navigation=form ``` `--routes` accepts up to twenty-one unique labels: `direct` and the configured `proxy-1` through `proxy-20` routes. The whole selection is validated before traffic; an unconfigured label is refused. `direct` here means the isolated browser on the current connection. The standalone `serp-browser-check.mjs` CLI remains current-connection-only. The route checker accepts the same `--profile`, `--headed`, `--navigation`, `--channel`, `--timeout-ms`, and `--state` browser options described above. Its defaults are headless, direct navigation, and a 12,000 ms deadline per probe; profile and channel inherit configured SERP or Unblocker settings, with the native profile as the fallback. Only already installed runtimes and operator-supplied proxy credentials are used. Both browser diagnostics use the same state file and exclusive lock. Persisted proxy health uses opaque identities derived from the effective configuration. An explicit shared egress label gives aliases the same key, including across browser profiles, navigation modes, and credential rotation. That key shares both persisted pacing and block cooldown, so a new session token does not bypass either guard. The current-connection `google:direct` ledger remains shared with the local browser diagnostic. Keep the same `--state` path for comparisons; paced or cooling routes are refused before launch with zero attempts. Shared aliases can therefore be throttled within one sequential checker run. The same five-minute first block penalty and one-hour cap apply. These local diagnostic ledgers do not coordinate ordinary production routes or distributed replicas. The experimental owned adapter can share their durable gate on the same filesystem and declared egress identity. Neither identity scheme measures the provider's actual exit IP. Stdout contains a sanitized JSON aggregate with `ok`, `samples`, `delivered`, and `routes`. Each route entry has its safe route label and browser probe outcome, stages, latency, result count on success, and available numeric pacing and cooldown counters, including optional `nextRequestAt`. Queries, URLs, credentials, credential templates, session tokens, and persisted identity hashes are omitted. Configuration failures return a fixed safe `error` code before traffic. Exit code 0 means every selected route delivered a parsed capture; exit code 1 means a refusal or failure. A successful sample does not establish Google coverage, physical exit identity, or provider session stickiness. ## Browser container Build the dedicated SERP image from the repository root: ```bash docker build -f deploy/vitna-search-serp.Dockerfile -t vitna-search-serp . ``` This image includes the isolated browser runtime and selects `auto` SERP transport with browser fallback. SERP retrieval stays off until the operator sets `VITNA_SEARCH_SERP_ENABLED=true` and `VITNA_SERP_RESALE_APPROVED=true`, as described in [Run without an index](#run-without-an-index). A native index is optional. Supply those settings, the private search client registry and any operator proxy configuration at runtime, and use a Linux host or container runtime that permits the Chromium sandbox and user namespaces. The image's readiness probe checks service configuration; it does not establish successful access to a search engine. `GET /readyz` follows the default `/v1/search` route: when a native index is the default provider, it returns HTTP 503 while that index is empty or unavailable, even with SERP retrieval enabled. A SERP-only instance is ready while its retrieval adapter reports itself available. Authenticated `GET /v1/status` still reports SERP availability if the native index status read fails, with `index: {ready: false, unavailable: true}`. The existing `deploy/vitna-search.Dockerfile` also supports direct SERP retrieval without browser dependencies. Its defaults assume a native index at `/data/index.sqlite` and `VITNA_SEARCH_PROVIDERS=vitna`. For SERP-only operation, override the index to an empty value, replace that provider order with `brave,exa,tavily` without supplying gateway credentials, and set `VITNA_SEARCH_SERP_ENABLED=true`, `VITNA_SERP_RESALE_APPROVED=true` and `VITNA_SEARCH_SERP_TRANSPORT=direct`. Unconfigured legacy providers are skipped; SERP retrieval remains a separate capability. Direct egress may encounter consent, rate limits, or challenges. Browser transport and optional operator proxies add another retrieval route but do not guarantee success, verified geographic targeting, or broad production coverage. This implementation does not yet provide a full substitute for a vendor's distributed retrieval infrastructure. Local live Google browser testing has returned `serp_blocked`; verify access on the intended operator route before relying on it. ## Authentication, quotas, and failures Single-page SERP requests use the same search client registry, per-client minute/day quotas, service concurrency limit, and whole-request deadline as indexed search, extraction, and research. An admitted SERP failure consumes one quota slot, including route-busy, cooling, or throttled responses. Invalid requests and the shared API's `service_busy` refusal consume no quota. Receipts record operation, request identifier, outcome, and latency, without the query, search URL, fetched HTML, snippets, authorization header, or proxy secrets. Responses use `Cache-Control: no-store`. Failures use the shared `{"error":{"code":"...","message":"..."}}` envelope. Invalid input returns HTTP 400; authentication uses HTTP 401, and shared rate limits use HTTP 429. Retrieval and parsing failures return a safe explicit error, without forwarding a challenge page or an upstream error body as a result. Every retrieval failure, a missed deadline included, returns HTTP 503 and is told apart by its code. A proxy in front of a hosted instance can replace the body of a 502 or 504 with its own page, which would hide the code; a 503 keeps it. Before 2026-10-22 these failures returned 502, and `serp_timeout` 504. A failed request's receipt records the same code as its `outcome`, so an org's console activity and the service's journal tell failures apart too. | Status and code | Meaning | |---|---| | 503 `serp_unavailable` | SERP retrieval is not enabled on this instance | | 503 `serp_jobs_unavailable` | Multi-page jobs or their private storage are unavailable | | 429 `serp_jobs_busy` | Job capacity is full; includes `Retry-After` | | 503 `serp_runtime_unavailable` | A required browser runtime could not be loaded or started; refusals while its route is paused include `Retry-After` | | 503 `serp_routes_busy` | Eligible retrieval routes are at their concurrency limit (a route that rotates its own exits first waits for a free slot while 8 s of the deadline remain); includes `Retry-After` | | 503 `serp_routes_cooling` | Eligible retrieval routes are cooling after failures; includes `Retry-After` | | 503 `serp_routes_throttled` | Independent retrieval starts are paced; includes `Retry-After` and no new attempts | | 503 `serp_blocked` | Challenge, consent or sign-in page, or access block | | 503 `serp_parse_failed` | Page not recognized by the organic-result parser | | 503 `serp_unreachable` | Search engine could not be reached | | 503 `serp_response_too_large` | Fetched HTML exceeds the 2 MiB limit | | 503 `serp_invalid_response` | Upstream response is not supported nonempty HTML, or a browser challenge flag without HTTP 403 or 429 | | 503 `serp_redirect_blocked` | Redirect outside the supported search route or past the redirect limit | | 503 `serp_query_mismatch` | Final search URL changed the requested query or offset, or results violated an unambiguous single positive `site:` hostname constraint | | 503 `serp_timeout` | Search engine did not finish within the request deadline | | 499 `request_cancelled` | Caller cancelled retrieval | The `site:` result guard permits the exact hostname and its subdomains. It skips complex, quoted, multiple-site, or invalid-DNS syntax; it does not interpret arbitrary search operators. ## Indexed search and cited research `POST /v1/search` with `mode: "index"` searches Vitna's stored corpus. `POST /v1/extract` returns stored indexed page text. `POST /v1/research` assembles cited passages from indexed evidence. These endpoints remain available when the operator configures their independent index and crawler. Research source selection uses bounded candidate metadata before reading, then rechecks the extracted page's URL and title before passage selection. A small registry recognizes PostgreSQL, Python, and Node.js ownership and canonical documentation version paths. When the query identifies one requested version, explicit incompatible versions in those paths or subject-specific titles are excluded; unknown or mixed version evidence stays eligible. Snippet numbers, URL query parameters, and arbitrary numeric paths do not establish a version. Cross-product or multiple-version queries do not infer one requested version. Documentation queries prefer recognized documentation metadata, known primary domains, then compatible version evidence, with stable input-order ties. `coverage.selectionFilteredSources` counts selection exclusions. This policy does not verify content facts or establish comprehensive coverage. Private controlled replays on 2026-10-05 checked these changes without new Google captures. Replaying stored HTML and exact redirect mappings produced 61 result identities across seven pages from 64 original rows and 62 legacy URL identities. It retained the first tracked YouTube listing and its original source, excluding the later bare-link variant; distinct identities do not establish distinct content. For the PostgreSQL 18 selection replay, known primary-domain sources among the top five increased from four to five, and the canonical 9.0 manual was excluded. Three separately guarded direct publisher reads returned HTTP 200 and passed the URL/title version checks. A native research replay over those real pages returned four exact Unicode-offset citations covering 685 quoted characters from the 18.0 release notes. It remained `partial` and non-exhaustive, using one source after three attempts and two skips. Saved raw Google results were unchanged, and the replay did not mutate the indexed corpus or API results. These are bounded regression observations, not an independent quality benchmark or proof of broader factual accuracy or coverage. SERP retrieval does not enqueue native discovery, add organic links to the index, fetch result pages, or produce cited research. A SERP snippet alone is not the indexed source text needed by the research API. See the [search overview](overview.md) for those separate contracts and the [migration guide](live-engine.md) for former Live/Hybrid callers. --- <!-- https://vitna.ai/docs/web/overview.md --> # Vitna Web APIs Hosted pilot at `https://web.vitna.ai` (OpenAPI at `/openapi.json`, MCP manifest at `/.well-known/mcp.json`, Agent Card at `/.well-known/agent-card.json`), with keys by invitation while public signup is closed. Turn public web pages into Markdown, structured JSON, crawl datasets, change alerts (`diffMode`), structural redlines (`/v1/diff`), parsed PDF/Office documents (`/v1/parse`), company dossiers (`/v1/entity`, `/v1/ir`), cited enrichment matrices (`/v1/grid`), claim verifications (`/v1/verify`), structured registry firehose events (`/v1/firehose/events`), screenshots, and printable PDFs (`pageRanges`, `scale`, `printBackground`), alongside 16 official data sources (`courtlistener`, SEC EDGAR, Federal Register, CISA KEV, FRED, FINRA, Treasury, NY Fed, BLS, and more) and 25 MCP tools (`/v1/mcp`). Native retrieval, lexical reranking, observed field extraction, and bounded research analysis run in the Vitna process by default. The product page and request playground are at `/apis`. The native customer account console is at `/apis/account`. To get a key, request an invitation at [vitna.ai/start](https://vitna.ai/start#web); a person reads the request. Activate the invitation on the [account page](https://vitna.ai/apis/account), which shows an API key and an owner token once each. Nothing is billed during the pilot: usage is measured on a monthly statement, no rates are set, and payment collection is off. An org's platform key (`vitna_key_`) works too; see [One key for the hosted APIs](../one-key.md). Material for the people who operate the service is collected at the end, under [Operators: running the service](#operators-running-the-service). ## Authentication and the website relay The hosted API answers at `https://web.vitna.ai`. Since 2026-10-07 a browser that opens it, or its `/docs`, is sent to `/apis` and this reference on vitna.ai, where the product's pages live: the operator sets `VITNA_WEB_PAGE_URL` and `VITNA_WEB_DOCS_URL`, and a self-hosted instance with neither set serves its own reference page. Web operations use a `vitna_web_` bearer key: ```bash curl http://127.0.0.1:8790/v1/read \ -H "Authorization: Bearer $VITNA_WEB_KEY" \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/","format":"markdown","render":true}' ``` The website can forward requests to the standalone service. Set `VITNA_WEB_SERVICE_URL` in the website environment, for example to `http://127.0.0.1:8790`. The relay forwards the caller key and maps `/api/web/read` to `/v1/read`. It also supports the job and monitor routes listed below. Every route carries a credential, so the relay sends nothing over plain HTTP except to a literal loopback address (`127.0.0.1` or `[::1]`); any other service URL must be HTTPS, and the relay answers `503` otherwise. The website relay is mounted as a catch-all route: ```text GET /api/web/[...path] POST /api/web/[...path] DELETE /api/web/[...path] ``` Replace `[...path]` with the supported operation or job path when making a request, such as `read` or `jobs/JOB_ID`. With no service configured, the relay returns an explicit configuration error. The `/apis` playground starts with illustrative output; **Execute live request** sends a real request through the relay. Its password field keeps the key in component memory, with no browser storage. ## API accounts and keys An operator issues each invitation (see [issuing invitations](#issuing-invitations)). It carries a one-time activation token, which lasts 24 hours by default. Redeem it once, on the [account page](https://vitna.ai/apis/account) or directly: ```bash curl http://127.0.0.1:8790/v1/accounts/activate \ -H 'Content-Type: application/json' \ -d '{"activationToken":"REPLACE_WITH_PRIVATE_ACTIVATION_TOKEN","label":"Application"}' ``` The HTTP 201 response contains `apiKey`, `ownerToken`, account metadata, and initial key metadata. Save the returned tokens privately; subsequent reads do not return them. API keys use `vitna_web_`; owner tokens use `vitna_web_owner_`. The owner token manages the account and keys, while an API key executes the web operations. | Endpoint | Owner-token operation | |---|---| | `GET /v1/account` | Account, key metadata, and accounting status | | `POST /v1/account/rotate` | Replace the owner token; send an empty JSON object | | `GET /v1/keys` | List active, expired, and revoked key metadata | | `POST /v1/keys` | Issue a key with optional `label` and `expiresAt` | | `POST /v1/keys/:id/rotate` | Replace that key, invalidating its previous token | | `DELETE /v1/keys/:id` | Revoke that key | | `GET /v1/invoices/YYYY-MM` | Metered monthly usage statement | Use `Authorization: Bearer $VITNA_WEB_OWNER_TOKEN` for those routes. Creation and rotation bodies accept optional `label` and an ISO UTC `expiresAt`, such as `2026-12-01T00:00:00.000Z`. Labels accept up to 80 characters. API keys expire within 366 days; up to 20 keys per account are retained. New requests reject revoked and expired tokens immediately. Owner rotation invalidates the old owner token. Activation requests are limited to 10 per minute per process; owner operations to 60 per minute per account. These management routes use JSON bodies up to 4,096 bytes and do not accept query parameters. The SQLite store retains account metadata and credential hashes. Replaying the startup invitation configuration cannot redeem a used invitation again or restore a rotated credential. Customer payments are not collected. Public signup is closed on the hosted pilot, and keys are issued by invitation. `GET /v1/accounts/signup` reports whether it is open; the operator's switch is described under [optional public signup](#optional-public-signup). A lost owner token cannot be shown again. Ask through [the contact form](https://vitna.ai/start#contact) and an operator issues a new one, as described under [offline operator recovery](#offline-operator-recovery). ## Indexed search ```text POST /v1/search ``` Search the local index with the same Web bearer key: ```json { "query": "product pricing", "maxResults": 5, "domains": ["example.com"], "maxAgeDays": 30 } ``` `query` accepts 1 to 1,000 characters; `maxResults` defaults to 5 and accepts 1 to 20. Optional `index` targets `web` or a configured vertical index, and optional `effort` (`fast`, `balanced`, `deep`) tunes native retrieval vs. inline excerpt depth. Optional `domains` and `excludeDomains` filter public hostnames; `maxAgeDays` accepts 1 to 3,650 and filters observed crawl age. Optional `asOf` (or `contents.snapshotAsOf`, accepting `YYYY-MM-DD` or ISO 8601 with a UTC offset) queries historical snapshots captured on or before that instant with zero lookahead; it requires lexical ranking and cannot accompany live `scrape`. `timeoutMs` accepts 1,000 to 60,000 milliseconds. Optional `contents` attaches stored indexed text, `highlights`, or `excerpts` (with optional `maxTotalCharacters` / `maxTotalTokens` and `injection` guard) in one native call without a live fetch. The response contains `query`, ranked `results`, engine information, and request `meta`. Titles, snippets, and crawl timestamps come from the indexed pages. By default, search makes no live page fetch, and no matching indexed pages returns an empty result list. See [Vitna Library](/library) on vitna.ai for the full catalog of vertical indexes and native data sources. To search and fetch each matching page in one call, set `scrape: true` or pass a `scrape` object with `format`, `maxCharacters` (default 20,000, up to 100,000), `onlyMainContent`, `includeTags`, `excludeTags`, `render`, `waitFor`, `timeoutMs`, `retries`, browser controls, `highlights`, `question`, `extract`, `injection`, or `redact`. Each result then carries a `scrape` object with `status: "ok"` and the page document, or `status: "failed"` and the error code when that single URL could not be read. When `scrape.highlights` is enabled without its own `query`, it defaults to the search `query`. Each scraped hit counts against the client's page-attempt quota. ### Experimental local semantic ranking Lexical retrieval remains the default. Operators can enable `VITNA_WEB_SEARCH_SEMANTIC=true` on the native backend, then request `ranking:"semantic"` on `POST /v1/search`. This fits a local TF-IDF/LSA model over at most 128 currently enabled indexed documents, bounded to 300 title characters and 3,500 plaintext characters per document. It uses eight retained dimensions by default and a cosine cutoff of 0.2. Training and ranking use no external provider or model download. The model learns associations within that bounded corpus. It abstains on unobserved query terms and detected negation/exclusion. It cannot establish every logical constraint or factual implication. Strict lexical admission requires `ranking:"lexical"`; combining semantic ranking with strict mode is rejected. Scores are latent cosine similarities, not probabilities. Operator host scope restricts the training corpus. Domain, site and age filters restrict returned IDs; the model still trains over the broader allowed-host corpus. Source policy and current plaintext are rechecked after asynchronous fitting, and a changed corpus makes the query abstain instead of returning a stale passage. Content changes refresh the cached fingerprint; timestamp-only refreshes need not retrain unchanged text. Run the frozen authored evaluation locally: ```bash npm run search:semantic ``` On its 32-document, 26-query synthetic fixture, default eight-dimensional ranking raises held-out nDCG@5 from 0.566860 to 0.837790 and recall from 0.625000 to 1.000000. MRR falls from 0.916667 to 0.861111; negative accuracy falls from 5/8 to 4/8. Exact-detail queries include regressions, and observed cross-topic combinations still produce false positives. The frozen report at `.search-data/reports/local-semantic.json` records all predeclared dimensions and regressions. These measurements describe this authored fixture, not broad web relevance or a general semantic model. ## Reader and scrape ```text POST /v1/read POST /v1/scrape ``` These routes are aliases. Read a page as Markdown, readable text, or cleaned HTML. The reader prefers main/article content, removes page furniture and scripts, and includes page metadata and links. ```json { "url": "https://example.com/docs", "format": "markdown", "render": true, "waitFor": "main article", "timeoutMs": 30000, "maxCharacters": 50000 } ``` | Field | Default | Meaning | |---|---|---| | `url` | Required | Public HTTPS page, up to 4,096 characters | | `format` | `markdown` | `markdown`, `text`, or `html` | | `render` | `true` | Execute page JavaScript | | `waitFor` | Omitted | CSS selector for content readiness; requires rendering | | `timeoutMs` | 30,000 | Request deadline, from 1,000 to 60,000 milliseconds | | `maxCharacters` | 100,000 | Output limit, from 1 to 1,000,000 characters | | `onlyMainContent` | `true` | Prefer `<main>`/`<article>` and omit `<nav>`/`<aside>` chrome; `false` reads the full `<body>` (still omitting scripts and styles) | | `includeTags` | Omitted | 1 to 20 CSS selectors (up to 100 chars each); render only matching subtrees in document order | | `excludeTags` | Omitted | 1 to 20 CSS selectors (up to 100 chars each); omit matching subtrees everywhere, including inside `includeTags` | | `extract` | Omitted | `schema`, `selectors` and `prompt`, as for Structured extraction; adds `data` from the same render | | `highlights` | Omitted | `true` or `{ query?, maxCharacters? }` (up to 4,000, default 1,000); returns verbatim `highlights` with `offsetUnit: "unicode_code_points"` | | `question` | Omitted | A question (up to 1,000 characters) answered from the rendered page; returns `answer` and `answerEngine`. Runs locally on the native extractive model or on a configured language model | | `injection` | Omitted | `scan` reports what could steer a model reading the page; `strip` also removes invisible characters from `content` (see below) | | `redact` | Omitted | `true`, or a list of `email`, `phone`, `card`, `iban`, `ssn` and `ip`; takes personal data out of the response | The HTML output is cleaned content, not the complete serialized browser DOM. Use [Unblocker](../unblocker/overview.md) when the complete DOM is needed. Truncated reader output is marked in the response. The configured service deadline can be shorter than the requested deadline. **Personal data.** `redact: true` takes email addresses, phone numbers, payment card numbers, IBANs, US Social Security numbers and IPv4 addresses out of the response, each replaced by a marker such as `[redacted:email]`. A list such as `["email", "phone"]` takes only those kinds. The response's `redacted` counts what was replaced, by kind, for example `{ "email": 2, "phone": 1 }`. Redaction covers `content`, `title`, `metadata.description`, `links` and extracted `data`, and `/v1/extract` and `/v1/crawl` accept the same field. The content is redacted before it is cut to `maxCharacters`, so a cut cannot leave half an address behind. A language model reading the page for extraction is given the redacted text, and its answer is redacted too, since a model can repeat what it has seen elsewhere. A page's own address (`url`, and the `url` of a failed page in a crawl) is how a result is matched to its page, so it stays as read. A crawl follows a page's links as the page wrote them and returns them redacted. An address a page splits with invisible characters, such as a zero-width space, is still found, and the replacement takes the invisible characters with it, so `injection: "strip"` cannot rejoin one. With `injection`, the findings' excerpts are redacted too: they quote the page's hidden parts, which the content leaves out. Each kind is found by its shape and a check, never by guessing: | Kind | Found when | |---|---| | `email` | An address with a domain. Image names such as `logo@2x.png` and remotes such as `git@github.com:org/repo` are not addresses | | `phone` | A number with `+` and a country code, 8 to 15 digits; a `tel:` link; or a North American number with its punctuation, such as `(212) 555-0147`, `212-555-0147` or `212.555.0147` | | `card` | 13 to 19 digits, run together or in the usual groups, with the prefix and length of Visa, Mastercard, American Express, Diners Club, JCB, Discover or UnionPay, passing the Luhn check | | `iban` | A country code, two check digits and up to 30 letters and digits, passing the mod-97 check | | `ssn` | Three, two and four digits with dashes, never starting 000, 666 or 900 to 999, with no all-zero group | | `ip` | Four numbers of at most 255. Loopback, multicast, reserved and documentation addresses are left, as are network blocks such as `10.0.0.0/8` and numbers after "Section", "§" or "version" | **What redaction does not find.** Names, street addresses, dates of birth and other personal data with no shape of its own; IPv6 addresses; Social Security numbers without dashes; and phone numbers outside North America written without their `+` country code. A page about one of these formats, such as an article on IP addressing, has its examples redacted. A reference number that passes a card's checks reads as a card, and its marker shows where. An asynchronous extraction keeps the pages it has read in its restart checkpoint until the job ends, with or without redaction. ### Documents A URL that serves a PDF, Word (`.docx`), Excel (`.xlsx`) or PowerPoint (`.pptx`) file is read as text, with no extra field. A document is recognised by its content type, or, when the server sends a generic type such as `application/octet-stream`, by its file name, and its first bytes must agree. The same reading applies wherever a page is read: Scrape, Extract, Crawl, Monitors and Research. A page that sends the browser on to a document, as a cleared challenge page or a script redirect can, is read as that document. | Kind | Output | |---|---| | PDF | A section per page, headed `Page N`, with each page's text in the order the page draws it | | Word | Headings, paragraphs, bulleted lists, web links and tables | | Excel | A table per sheet, headed `Sheet: name`; dates and times as their cell formats show them, in ISO 8601, and other numbers as stored | | PowerPoint | A section per slide in the deck's order, headed `Slide N` and the slide's title | The response's `metadata.document` describes the file: `kind`, `contentType`, `filename`, `bytes`, and `pages`, `sheets` or `slides`, with `truncated: true` when the document passed the reading limit of 1,000,000 characters. Pictures, charts, formulas, speaker notes and other formatting are not read. Page controls (`waitFor`, `actions`) do not apply to a document, so a crawl that sets them still reads the documents it links to. `/v1/screenshot` and `/v1/pdf` capture what the browser shows and do not read documents. [Linked documents](#linked-documents) downloads the files themselves. | Code | HTTP | Meaning | |---|---|---| | `document_too_large` | 422 | The file exceeds 16 MiB | | `document_invalid` | 422 | The bytes are not a readable file of the kind they claim | | `document_encrypted` | 422 | The PDF is encrypted | | `document_no_text` | 422 | The file has no text; a scanned PDF needs OCR, which is not available | | `document_unreadable` | 422 | The file's text could not be read, for example a PDF whose fonts carry no map to letters | | `document_timeout` | 504 | Reading did not finish within 20 seconds or the request deadline | A document is requested twice: once by the browser, which is stopped as soon as the response's headers show a document, and once to read it. A download link that works only once fails on the second request. **Prompt-injection scan.** Set `injection` to `scan` on `/v1/read`, `/v1/scrape` or `/v1/crawl` to have each page checked for what could steer a language model that reads it. The scan uses rules, not a model, so it costs nothing beyond the read. It reports mechanisms and phrases, and does not judge intent: a page that writes about prompt injection is reported, because it holds the phrases. | Finding | What it is | |---|---| | `unicode_tags` | Invisible tag characters spelling text a model can read; the finding shows what they spell. Flag emoji are not reported | | `invisible_run`, `invisible_characters` | Four or more zero-width characters in a row, the shape of a hidden encoding, or sixteen or more in all. The joiners emoji, Indic and Persian text need are not reported | | `bidi_override` | A right-to-left override that makes text display other than it reads | | `ignore_instructions`, `new_instructions`, `role_token`, `system_prompt_probe`, `persona_shift`, `conceal_from_user`, `addressed_to_ai` | Instruction-like text, such as "ignore previous instructions", or the role tokens of a chat template, such as `im_start` and `[INST]` | | `exfiltration_link` | A Markdown image whose address carries a placeholder for data to send out | Each finding says where it was found (`text`, `characters`, `hidden_text`, `comment`, `attribute` or `meta`), whether that text reaches the returned `content` (`inContent`), its severity, and an excerpt. Hidden places include elements the reader leaves out, comments, `alt`, `title`, `aria-label` and meta descriptions, and text hidden by inline styles the reader keeps, such as zero opacity or font size and off-screen positioning. Text hidden by a stylesheet class is not seen. ```json "injection": { "risk": "high", "findings": [ { "kind": "unicode_tags", "severity": "high", "where": "characters", "inContent": true, "excerpt": "28 invisible tag characters spelling: ignore previous instructions" } ] } ``` `risk` is the most serious finding that reaches the content, or `low` when every finding sits where the content does not reach. `strip` removes tag characters, zero-width characters and bidi controls from `content`, keeping flag emoji and joiners, and reports how many it removed as `stripped`. It never rewrites visible text. Whatever a request asks, those characters are removed from everything Vitna's own language model reads. ### Batch scrape ```text POST /v1/batch ``` Scrape a known list of public HTTPS URLs with shared reader, browser, action, extraction, highlight, question, injection, and redaction options. ```json { "urls": [ "https://example.com/pricing", "https://example.com/security" ], "format": "markdown", "maxCharacters": 20000, "highlights": { "query": "SOC 2", "maxCharacters": 800 }, "extract": { "selectors": { "heading": "h1" } } } ``` Synchronous requests accept 1 to 20 URLs and answer `200` with `urls`, `pages`, `errors`, `counts` (`total`, `completed`, `failed`), `truncated`, and `meta`. Pass `async: true` to run up to 100 URLs as a background task (`202`), polled with `GET /v1/jobs/:id`, cancelled with `DELETE /v1/jobs/:id`, and optionally delivered to `webhookUrl`. `maxCharacters` defaults to 20,000 per page (up to 100,000), and `retries` defaults to `0`. Each URL is isolated: a page that fails to render or extract is recorded in `errors` (or `extractError` / `answerError` when reading succeeded) without aborting the rest of the batch. Background batch jobs checkpoint after every URL and recover across restarts, recording an in-flight ambiguous URL as `outcome_unknown` rather than repeating work twice. Result payloads respect the job byte budget and set `truncated: true` if remaining pages would exceed it. ## Browser controls Page-reading requests also accept these options: ```json { "url": "https://example.com/", "country": "US", "session": "product-run-01", "language": "en-US", "viewport": { "width": 1440, "height": 960 }, "headers": { "dnt": "1" } } ``` | Field | Meaning | |---|---| | `country` | Two uppercase letters selecting a configured proxy route that serves that country | | `network` | `auto` (the default) starts on the cheapest configured route and steps up a tier after a block; `datacenter`, `isp`, or `residential` uses only routes of that kind | | `session` | 1 to 64 letters, numbers, underscores, or hyphens; sticky proxy selection | | `persistSession` | Explicit boolean; `true` requires `session` and retains tenant-isolated cookies/localStorage | | `actions` | Up to 50 scroll, wait, click, write or press actions; requires rendering. Bounds, the GET-only guard and failed-step reporting (`error.action`) are under Browser actions in the [Unblocker request reference](../unblocker/overview.md#request). `/v1/read` and `/v1/scrape` also take screenshot and scrape steps, below | | `language` | BCP 47 locale, such as `en-US`, up to 35 characters | | `viewport` | Width from 320 to 4,096; height from 240 to 4,096 | | `headers` | Only `accept-language` up to 200 ASCII characters and `dnt` equal to `0` or `1` | A country request needs a matching route configured by the operator. An unavailable country fails explicitly, as does a `network` with no route of that kind (`network_unavailable`). When the operator configures residential or ISP provider routes, a blocked attempt's retry moves up a tier, and the bytes an attempt moves through those routes are metered for pricing; see [proxy routes](../unblocker/overview.md#proxy-routes). When the operator runs the [auto browser profile](../unblocker/overview.md#the-auto-profile), a challenged attempt's retry also moves from Chromium to Camoufox, so a page behind a browser check can still be read with the default `retries`. So can a [document](#documents): Camoufox takes the file from the response to the navigation that cleared the check, so `/v1/read` and each file [`/v1/documents`](#linked-documents) downloads return it with the same 16 MiB limit and first-bytes check, and without asking for it a second time. A document Camoufox still cannot hand over, such as one whose bytes are not the file they claim to be, fails as the `target_challenge` that sent the read to Camoufox. A session keeps route selection deterministic while the route list stays unchanged. Every attempt still uses a fresh browser/context. With explicit `persistSession: true`, the renderer restores and saves that tenant's cookies and localStorage after successful delivery and cleanup. State stays in RAM with a 15-minute sliding TTL, 100 total sessions, 20 per tenant, and 256 KiB per session. It is lost on renderer restart. Same-session concurrent persistence returns `session_busy`. `DELETE /v1/sessions/SESSION_ID` clears the caller's session and cancels its active attempt. The caller cannot submit proxy credentials, cookies, authorization headers, or browser scripts. **Screenshot and scrape steps.** On `/v1/read` and `/v1/scrape`, an action sequence can capture the page partway through: `{ "type": "screenshot" }` images what the page shows at that step, and `{ "type": "scrape" }` reads it. A list paged by a button reads as one request: ```json { "url": "https://example.com/filings", "actions": [ { "type": "scrape" }, { "type": "click", "selector": "button.next" }, { "type": "scrape" }, { "type": "screenshot", "fullPage": true } ] } ``` The response's `content` is the page after the last step, as always, and `actions` holds the captures, each with the `index` of the step that took it: ```json { "actions": { "screenshots": [{ "index": 3, "contentType": "image/png", "base64": "iVBORw0KGgo...", "bytes": 48213 }], "scrapes": [ { "index": 0, "url": "https://example.com/filings", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": ["https://example.com/filings?page=2"], "truncated": false }, { "index": 2, "url": "https://example.com/filings?page=2", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": [], "truncated": false } ] } } ``` A step waits for the page's text to settle before it captures, as the final read does, so a scrape after a click takes the next page rather than the one before it. Scraped pages use the request's `format` and `maxCharacters`. A screenshot shows the viewport unless `fullPage` is `true`, and is a PNG unless `format` is `jpeg`. A sequence may hold 10 screenshot and scrape steps, which together may capture 8 MiB (images, and the HTML of scraped pages). A step past that budget, or a full-page image larger than 16,384 pixels on a side or 32 million pixels in all, fails with `action_output_too_large` (HTTP 422) and names itself in `error.action`. Other routes, crawls included, refuse these steps, and `/v1/status` lists them under `capabilities.actionOutputs`. A scraped page is treated as the final one is: with `injection` it carries its own `injection` report, and with `redact` its personal data comes out and is counted in the response's one `redacted`. A screenshot is an image, which neither the scan nor redaction reads. Without `waitFor`, the browser allows an initial 500-millisecond grace period, then waits for 200 milliseconds of stable meaningful visible content, within five seconds or the remaining request deadline. A persistent empty or loading state fails with `target_incomplete`. Set a populated content selector for delayed applications; page stability cannot establish that every requested product field has finished loading. A `waitFor` selector that never becomes visible fails with `selector_timeout` (HTTP 504) shortly before the deadline rather than `request_timeout`, and a retry waits within what is left of `timeoutMs`. ### Encrypted sessions across restarts Session state remains in RAM by default. For local browsers, pair `VITNA_WEB_SESSION_DATA` with `VITNA_WEB_SESSION_KEY_HEX`, an exact 32-byte hexadecimal operator key. Use a separate private directory from `VITNA_WEB_DATA`; with the supplied systemd unit it can be `/var/lib/vitna-web-sessions`. Requests still need a named session and `persistSession:true`. Browser-worker deployments configure persistence on each worker instead of the API coordinator. Cookies and local storage are encrypted using AES-256-GCM with fresh nonces and authenticated tenant/session identity, expiry and byte bounds. The same stable key restores unexpired sessions after restart. A wrong key fails closed and preserves the prior database. A key that opens some rows but not others is the right key with damaged rows: each damaged row is set aside unchanged for inspection and counted, and the other sessions open normally. After startup a damaged row affects only its own session, which its owner can delete. TTL, per-client/global capacity, successful-state commit and explicit deletion continue to apply to durable rows. Deletion cannot be undone by a stale active lease. Renderer shutdown retains ciphertext and closes storage only after physical browser cleanup settles. The helper rejects directory/data-file links and database hardlinks that could bypass the writer lock. Existing POSIX storage must belong to the service user and grant no group/other access. Windows deployments must configure equivalent private ACLs. Session keys are operator secrets; back up the key separately from a consistent stopped-service database backup. Key rotation and replicated session storage are not implemented. ## Structured extraction ```text POST /v1/extract ``` Provide CSS selectors for known fields. A string selector reads text; an object with `attribute` reads an attribute from matching elements. This path runs locally without a model. Supply up to 32 selector fields. Supported syntax includes tags, IDs, classes, attribute matches, descendant and child combinators, and comma-separated groups. A supplied selector scopes its field; a missing match is not replaced with an unrelated structured-data value. ```json { "url": "https://example.com/products/desk", "selectors": { "name": "h1", "price": ".price", "image": { "selector": "img.product", "attribute": "src" } } } ``` To shape published structured data, provide a supported JSON Schema. Extraction reads JSON-LD, microdata/RDFa, supported embedded application data, and any supplied selectors. Canonical URLs, explicit primary-entity references, product identifiers, and visible headings help select the page's company or product. An unresolved choice between multiple entities returns `extraction_ambiguous` (HTTP 422); explicit selectors can select the intended fields. Schema fields can use object, array, string, number, integer, and boolean types, including nullable unions with `null`. Supported keywords include `properties`, `required`, `items`, `enum`, `additionalProperties`, `description`, and `title`. ```json { "url": "https://example.com/products/desk", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": ["number", "null"] }, "currency": { "type": ["string", "null"] } }, "required": ["name", "price", "currency"], "additionalProperties": false } } ``` Missing fields use the same bounded observed-fact parser as native Research. It recognizes explicit JSON objects, labelled literals, dotted/indexed fields, two-column tables, definition lists, and a finite set of unambiguous prose facts, including named company founding dates, exact employee counts, headquarters, and product prices/weights. It preserves the source wording, units, currencies, billing periods, and ranges internally. A measured value such as `2 kg` needs a string field; it is not reduced to the number `2`. The symbol `$` alone does not establish the currency code `USD`. The parser rejects conflicting requested observations with `extraction_ambiguous` (HTTP 422). Missing required fields return `extraction_incomplete`; invalid extracted values return `extraction_invalid`, also HTTP 422. If the configured native engine is called for remaining fields, its unsupported or conflicting observations return `native_extraction_unsupported` or `native_ambiguous_data`. An optional self-hosted compatible model can handle language-based fields beyond these rules. The final result must satisfy the schema, including nested objects, arrays, required fields, and enums. Schema responses include configured engine metadata in `extractionEngine`; extracted `data` follows the requested schema. **Extract with a prompt.** Add `prompt`, up to 10,000 characters, to tell the model what to extract in plain language. With a schema, the prompt guides how the model fills it; without one, the model's JSON object is the data: ```json { "url": "https://example.com/investors/governance", "prompt": "List every board member with their role and the committees they chair." } ``` A prompt needs a language model (`VITNA_WEB_MODEL_BACKEND=self-hosted`); the native engine returns `model_not_configured` (HTTP 503) rather than ignoring it, and `/v1/status` reports `capabilities.promptExtraction`. With a prompt, the model reads every field that no selector names, because published data cannot follow an instruction such as "prices in euros". Without one, published data, selectors and observed facts come first and the model fills only what they miss. A prompt cannot accompany a recipe, and selectors with a prompt need a schema that declares their fields. **Long pages and repairs.** A language model reads the page in parts of 60,000 characters, cut at paragraph breaks, at most eight of them; each part is a metered model request. The answers combine in page order: a list gathers values from every part without repeats, and a single value is the first a part established. An answer that does not match the schema is sent back once with the problem named before the request fails with `extraction_invalid`. A response that used a language model reports how it read the page: ```json "extraction": { "parts": 3, "truncated": false, "repaired": 0 } ``` `truncated` means the text ran past eight parts. Data from a prompt without a schema is limited to 1 MiB (`extraction_too_large`) and 32 levels of nesting. The native engine reads a single pass, as before, since it refuses observations that conflict and must see them together. **Personal data.** With `redact`, as described under [Reader and scrape](#reader-and-scrape), a language model reads the pages with the named kinds already taken out, every string in `data` (and in `rows`) is redacted as well, and the response's `redacted` counts what came out of them. **Scrape and extract in one request.** `/v1/read` and `/v1/scrape` accept an `extract` object with `schema`, `selectors` and `prompt`, under the same rules. The response carries the page's `content` and the extracted `data` from one browser render: ```json { "url": "https://example.com/products/desk", "format": "markdown", "extract": { "prompt": "The price in euros and whether it is in stock" } } ``` ### Extract across pages Supply `urls` instead of `url` to combine up to 20 pages into one result: ```json { "urls": ["https://example.com/about", "https://example.com/contact"], "schema": { "type": "object", "properties": { "name": { "type": "string" }, "foundingDate": { "type": "string" }, "employees": { "type": "integer" } }, "required": ["name"], "additionalProperties": false }, "async": true, "jobTimeoutMs": 120000 } ``` For site discovery, provide `url` and `maxPages`, up to 20, with the map discovery and path-filter options below. The default extraction page budget is 1. Arrays are combined without duplicate values; scalar fields use the first valid local value, and compatible nested objects can fill complementary fields. Flat product extraction combines pages only when their selected products have matching identity; one product's name cannot borrow another's price. Use an object array such as `products` for multiple product pages. Conflicting requested prose observations remain an error. Observed-fact parsing and the optional model fallback use at most 100,000 combined content characters. The parser also bounds source count, nesting, arrays, and fact count; it does not read every page of a domain automatically. The result includes `url`, successfully read `urls`, combined `data`, and per-page `errors`. With `async: true`, the response is a job receipt instead. Poll `/v1/jobs/:id` for progress and the final result. `jobTimeoutMs` accepts 1,000 to 600,000 milliseconds for background work; `timeoutMs` remains the per-page browser deadline. An optional `webhookUrl` can receive a job completion notification. Fetch the result from the job endpoint. ## Extraction recipes ```text POST /v1/recipes/draft POST /v1/recipes GET /v1/recipes GET /v1/recipes/:id DELETE /v1/recipes/:id ``` A recipe names the fields to read from one kind of page, and how to read each, once, so every page built from the same template can be extracted or watched by its id. A draft builds one from examples: name a sample page and, for each field, its value as the page shows it. ```json { "url": "https://shop.example.com/products/desk-lamp", "samples": ["https://shop.example.com/products/office-chair"], "fields": { "name": "Desk Lamp", "price": "$24.99", "image": { "example": "https://shop.example.com/img/lamp.jpg", "attribute": "src" }, "reviews": { "examples": ["Great lamp", "Too bright"] }, "sku": { "describe": "the product's SKU" } } } ``` | Field form | Meaning | |---|---| | `"value"` or `{ "example": "value" }` | Text the page shows, matched regardless of case and spacing; an element whose text closely contains it also matches | | `{ "examples": ["a", "b"] }` | Two to 20 values of one repeated field, which becomes a list | | `{ "example": "value", "list": true }` | One value of a repeated field | | `{ "example": "value", "attribute": "src" }` | An attribute's value rather than text. Without `attribute`, an example not found as text is looked for in `content`, `src`, `href`, `datetime`, `value`, `alt`, `title` and `data-src`, and a relative URL matches its absolute form | | `{ "describe": "words" }` | No example: extraction finds the value from published data, observed facts and the operator's model when one is configured, and that value becomes the example | The draft renders the pages with the request's browser controls, finds the element holding each example in text a reader sees, never under a hidden ancestor, and keeps the selector that reaches it most steadily. An id comes first, then `itemprop`, `data-testid` and similar attributes, then classes, then a class or tag scoped by a named ancestor. Generated class names such as `css-1x2y3z` or `Button_label__a8Kq3` are never used. A single field's selector must reach the example before anything else, because extraction keeps the first match; a list field's must reach every example. Each candidate is run on the page before it is chosen. The selector engine has no positions, so an element told from its siblings only by position gets no selector. The response gives each field its `selector`, `attribute` and `list`, with `basis` (`id`, `attribute`, `class`, `scoped` or `tag`), the number of elements it `matches`, and whether the example matched as `text`, by `contains` or as an `attribute`. A field without a selector gives its `reason`: `example_not_found`, `examples_disagree`, `no_stable_selector` or, for a description, `value_not_found`. A described field's value is returned as `described`. `recipe.fields` is ready to save, and `preview` reads every page with the recipe: ```json { "preview": [ { "url": "https://shop.example.com/products/desk-lamp", "data": { "name": "Desk Lamp", "price": "$24.99", "reviews": ["Great lamp", "Too bright", "Works fine"] }, "missing": [] }, { "url": "https://shop.example.com/products/office-chair", "data": { "name": "Office Chair", "price": "$89.00" }, "missing": ["reviews"] } ] } ``` This preview is illustrative. A draft is one request, and it reads the sample page and up to three `samples` through the same browser slots, page quota and accounting as any read. A sample that cannot be read appears in the preview with its error code; a first page that cannot be read fails the draft. Save a recipe with a `name`, its `fields`, and optionally `render` and `waitFor`. Each field is a selector string, or an object with `selector`, `attribute`, `list` and `describe`, of which a selector or a description is required. A draft's `recipe.fields` can be sent as returned. A selector the engine cannot parse is refused when the recipe is saved. A key holds up to 50 recipes of up to 32 fields each, and only that key can read, run or delete them. Run one by sending `recipe` in place of `schema` and `selectors`: ```json { "urls": ["https://shop.example.com/products/desk-lamp", "https://shop.example.com/products/office-chair"], "recipe": "RECIPE_ID" } ``` A recipe run adds `rows`, one `{ "url", "data" }` per page read, beside the combined `data`; a page whose own extraction fails keeps its row with an `error` code. A described field without a selector is found by extraction on each page. The recipe's `render` and `waitFor` are defaults that the request's own values override. `POST /v1/monitors` with `recipe` watches the recipe's selected fields; a recipe with none is refused with `recipe_unmonitorable`. An unknown or another key's recipe is `recipe_not_found` (404), and a key at its limit is `recipe_limit` (429). ## Site map ```text POST /v1/map ``` Discover links, sitemap entries, and indexed search results. Include and exclude filters match case-sensitive pathname substrings, and excludes take priority. Discovery defaults to the starting hostname. Additional domains must be explicitly allowed. ```json { "url": "https://example.com/", "maxPages": 100, "maxBreadth": 100, "sitemap": "include", "maxSitemaps": 3, "includePaths": ["/products/"], "excludePaths": ["/account/"], "timeoutMs": 30000 } ``` The default `maxPages` is 50, with a maximum of 1,000 for maps. Filters can contain up to 20 paths per list. The seed page can be read to discover links even when it does not match an include filter. The response reports discovery counts and errors in addition to the URLs. | Field | Default | Meaning | |---|---|---| | `maxDepth` | 20 for maps; 2 for crawls | Link depth from the seed, from 0 to 20 | | `maxBreadth` | 100 | Maximum discovery breadth, from 1 to 1,000 | | `sitemap` | `skip` | `skip`, `include`, or `only` | | `sitemapUrls` | Omitted | Up to 5 explicit public HTTPS sitemap URLs | | `maxSitemaps` | 3 | Sitemap document budget, from 1 to 10 | | `searchQuery` | Omitted | Optional discovery query, up to 500 characters; uses the selected indexed Search backend | | `searchLimit` | 20 | Search-result budget, from 1 to 100 | | `includeSubdomains` | `false` | Allow public subdomains of the starting host | | `allowedDomains` | `[]` | Up to 20 explicitly allowed public hostnames | For example, add `searchQuery: "site:example.com products"` to discover pages through the selected indexed Search backend. Search results and sitemap entries pass the same destination checks, host scope, and path filters as links found on a page. `sitemap: "only"` uses sitemap discovery without a full link walk. A map request remains bounded by its total `timeoutMs`, and by the service's request deadline when that is shorter. A map that runs out of time, by either deadline, fails with `crawl_timeout` (HTTP 504); only a caller that disconnects ends one with `request_cancelled`. ## Crawl jobs ```text POST /v1/crawl GET /v1/jobs/:id DELETE /v1/jobs/:id ``` Collect cleaned pages in a background job: ```json { "url": "https://example.com/docs/", "maxPages": 100, "maxDepth": 4, "maxBreadth": 100, "format": "markdown", "includePaths": ["/docs/"], "excludePaths": [], "timeoutMs": 30000, "jobTimeoutMs": 120000 } ``` The create response includes a job `id`, `status`, `pages`, and `errors`. Poll `GET /v1/jobs/:id` using the same client key to retrieve progress and collected pages. `DELETE` cancels queued or running work. The default page budget is 50 and the default depth is 2 links from the seed. The supported depth is 0 to 20; the page budget is 1 to 500. A page budget bounds attempted reads, while `maxCharacters` bounds each collected document. `timeoutMs` bounds each browser read. `jobTimeoutMs` bounds the whole background job, from 1,000 to 600,000 milliseconds, and defaults to `timeoutMs` when omitted. Partial page failures appear in `errors`. `redact` takes personal data out of every collected page before the job stores it, with a `redacted` count on each page, as described under [Reader and scrape](#reader-and-scrape). Crawls also accept `extract`, `highlights`, and `question` to run per-page structured extraction, passage selection, or page-grounded Q&A during the crawl; if extraction or Q&A fails on one page, that page keeps its content and records `extractError` or `answerError` while the crawl continues. Crawls accept the same sitemap, search, domain, and path controls as maps. Retrieve a bounded slice of crawl results with query parameters: ```text GET /v1/jobs/JOB_ID?offset=0&limit=25&includeContent=false ``` `offset` accepts 0 to 500, `limit` accepts 1 to 100, and `includeContent` chooses whether page content accompanies the page metadata. The response includes `pagination` with `offset`, `limit`, `total`, and `nextOffset`. Each job also caps returned content at two million characters and five MiB of serialized pages/errors. When that budget is reached, completed pages remain available and the job reports `truncated: true` with `truncationReason: "result_limit"`. Up to 20 jobs per client are retained; older finished jobs are removed to admit new work. Queued jobs recover after restart. Crawl resumes its committed frontier and extraction resumes committed page snapshots; unresolved page attempts are recorded as `outcome_unknown` and skipped. An ambiguous final model invocation, running Research jobs, and running catalog and backfill jobs remain failed rather than repeating metered work. Recovery preserves the original absolute deadline, backs off, and permits at most three restarts. Extraction checkpoints share the five-MiB result bound; oversized checkpoints fail explicitly. Recovery metadata is public, while its snapshots and intents remain private. These leases are single-writer restart bookkeeping. Jobs and monitors belong to the client identity associated with the key. A different client cannot inspect or cancel them. ## Change monitors ```text POST /v1/monitors GET /v1/monitors GET /v1/monitors/:id DELETE /v1/monitors/:id POST /v1/monitors/:id/check ``` Watch readable page text, or selected fields, on an interval. Supply a public HTTPS webhook to receive changes: ```json { "url": "https://example.com/products/desk", "intervalSeconds": 3600, "selectors": { "price": ".price", "availability": ".availability" }, "webhookUrl": "https://your-app.example/webhooks/vitna" } ``` The interval defaults to 3,600 seconds and accepts 60 seconds to 30 days. The first successful check establishes a baseline. Subsequent checks compare normalized text or selector values. A missing field can be a change; it is not treated as a price of zero. To score changes structurally instead of flat text diffs, pass `diffMode` (`prose`, `pricing`, `policy`, or `risk_factors`) and optional `minMateriality` (`0` to `1`). When the computed structural materiality score falls below `minMateriality`, the monitor records a `below_materiality` suppression entry, skips webhook delivery, and preserves its baseline; when it meets or exceeds the threshold, the event carries typed hunks, numeric deltas, and a SHA-256 `diff.attestation` digest. To suppress minor noise on a page monitor, set `meaningfulChange: true` (default 5% changed-token ratio), `{ "minChangedRatio": 0.1 }`, or `{ "prompt": "..." }` (requires a configured language model). When a change is judged not meaningful, the monitor records a `suppressed` reason in history, sends no webhook, and keeps the previous baseline intact so gradual drift still crosses the threshold. To watch search results for a query instead of a single URL, create a web monitor with `kind: "web"` (or omit `url` and pass `query`), along with optional `maxResults` (1 to 20, default 10), `domains`, `excludeDomains`, `ranking`, `goal`, `dedup` (`"story"` default or `"url"`), `maxAgeDays` (1 to 3,650), `minSnippetCharacters` (1 to 1,000), `summarize` (`true` for cited bullet summaries), and `schema` (for cited structured event payloads): ```json { "kind": "web", "query": "vitna release notes", "domains": ["example.com"], "goal": "pricing change or new enterprise plan", "dedup": "story", "maxAgeDays": 14, "summarize": true, "intervalSeconds": 3600, "webhookUrl": "https://your-app.example/webhooks/vitna" } ``` Web monitors report `added` and `removed` result URLs against the previous snapshot. With `dedup: "story"`, syndicated reprints with matching canonical URLs, normalized headlines, or 3-word shingle Jaccard similarity at or above 0.6 are suppressed across up to 200 seen stories without firing webhooks (`suppressedCount` is recorded in check history). When `goal` is set, alerts fire only when a newly added result matches the goal (judged by the configured language model when available, or deterministically by query-term match otherwise). Setting `summarize: true` or `schema` attaches cited `summary`, `data`, and `citations` fields to `monitor.changed`. List or inspect monitors to see their current snapshot, recent changes, check times, and webhook delivery state. The manual `/check` route takes an empty JSON object and runs a check immediately. `DELETE` stops the monitor. Scheduling requires the service process to remain running. Readiness selectors and retry settings are saved with each monitor. History is bounded to 20 checks. Webhook failures retain the same event ID across up to five delivery attempts. An event that fails all five leaves the queue, so a later successful delivery reports `delivered` again; the loss stays visible as `webhook.failedEvents`, a running count, and `webhook.lastFailure`, which names the event, its attempts and the last error. ## Screenshots and PDFs ```text POST /v1/screenshot POST /v1/pdf ``` Capture a browser view: ```json { "url": "https://example.com/", "format": "png", "fullPage": true, "width": 1440, "height": 960 } ``` Screenshots support `png` and `jpeg`. The default viewport is 1,440 by 960 pixels; width accepts 320 to 4,096 and height accepts 240 to 4,096. PDFs use Chromium print output and need only a URL for the default layout. On `/v1/pdf`, you can also bound long pages with `pageRanges` (page numbers or ranges between 1 and 1,000, such as `"1-3"` or `"1-2,5"`), `scale` (`0.1` to `2`), and `printBackground` (boolean, default `true`) to keep artifacts under the 10 MiB capture limit: ```json { "url": "https://example.com/", "pageRanges": "1-2", "scale": 0.75, "printBackground": false } ``` Both routes return an artifact in a JSON envelope: ```json { "artifact": { "contentType": "image/png", "base64": "BASE64_ENCODED_BYTES", "bytes": 12345 } } ``` This envelope is illustrative. Decode `base64` into bytes and save them with the returned content type. The playground provides a download button for live artifacts. ## Linked documents ```text POST /v1/documents ``` Download the PDFs a page links to, as the files they are: ```json { "url": "https://investors.example.com/financials", "maxFiles": 10 } ``` The page is read in the browser, as for Read, and every document link on it is listed. A link counts when its path ends in `.pdf` or its `type` attribute is `application/pdf`; a link whose text, title, label or class says "PDF" ("Annual report (PDF, 2 MB)") counts too and is listed after those, since it can lead to a page rather than a file. Links in embedded viewers (`iframe`, `embed`, `object`) count, fragments are dropped, and a link back to the page itself does not count. Each file is then fetched through the same browser and network path as the page, with its `country`, `session`, `network` and `language`, so a site that serves the page to the browser serves its files too. A `url` that is itself a document is the one file. | Field | Default | Meaning | |---|---|---| | `kinds` | `["pdf"]` | Which documents to find: any of `pdf`, `docx`, `xlsx`, `pptx` | | `maxFiles` | 10 | How many of the listed links to download, 1 to 20, in the order listed | | `download` | true | false lists the links without downloading any | | `timeoutMs` | 30,000 | The whole request's deadline, the page and every file together | The browser fields (`render`, `waitFor`, `actions`, `country`, `session` and the rest of [Browser controls](#browser-controls)) apply to the page; files take its network and identity but not its `waitFor` or `actions`. ```json { "url": "https://investors.example.com/financials", "found": [ { "url": "https://cdn.example.com/2025-annual-report.pdf", "text": "2025 Annual Report" }, { "url": "https://investors.example.com/static-files/0f1c", "text": "Q3 results (PDF)" } ], "documents": [ { "url": "https://cdn.example.com/2025-annual-report.pdf", "text": "2025 Annual Report", "kind": "pdf", "contentType": "application/pdf", "filename": "2025-annual-report.pdf", "bytes": 2481503, "sha256": "9f2c...", "base64": "JVBERi0xLjcK..." } ], "errors": [{ "url": "https://investors.example.com/static-files/0f1c", "code": "not_a_document" }], "truncated": false } ``` This response is illustrative. `found` lists up to 500 links in the order they are tried. `documents` holds each file downloaded, with `finalUrl` when a redirect served it from elsewhere, and `errors` each link that did not give one: `not_a_document` when it served a page, `document_kind_excluded` when it served a kind not asked for, `documents_too_large` when it would take the answer past 32 MiB, or the code a Read of it would fail with, such as `target_http_error` or `document_too_large` for a file over 16 MiB. A link in `found` but in neither list was not reached, and `truncated` is then true: `maxFiles`, the byte budget or the deadline stopped the downloads, or the page had more than 500 document links. The downloads already made are always returned; only a page that cannot be read fails the request. The page and each file attempted count against the daily page quota, and each is metered as a browser read. Files are read one at a time, so ten files take about ten times as long as one; ask for fewer, or a longer `timeoutMs`, on a slow site. ## Browser API ```text GET /v1/browser (WebSocket upgrade) ``` Drive a real Chromium with Puppeteer, Playwright or any Chrome DevTools Protocol (CDP) client. Each connection launches one browser for that session and removes it, with its profile, when the connection closes. Scripts, clicks, typing, forms and screenshots are the client's to send. The endpoint is off unless the operator sets `VITNA_WEB_BROWSER_API=true`. Connect with a `vitna_web_` key in the `Authorization` header. The website relay does not carry WebSockets, so connect to the Web service's own origin: ```js import { chromium } from "playwright"; const browser = await chromium.connectOverCDP("wss://web.example.com/v1/browser", { headers: { Authorization: `Bearer ${process.env.VITNA_WEB_KEY}` }, }); const context = await browser.newContext(); const page = await context.newPage(); await page.goto("https://example.com/"); await browser.close(); ``` Puppeteer's `connect` takes the same endpoint as `browserWSEndpoint` and the header in `headers`. The relay is tested with Playwright's `connectOverCDP` and with a raw CDP client. Add `?country=US` to select an operator proxy route that serves that country, as page reads do, and `?network=residential` (or `isp`, `datacenter`) to use only routes of that kind. A session has no retries to step up through, so without `network` it uses the cheapest configured route. Add `?session=NAME` (1 to 64 letters, numbers, underscores or hyphens) to keep a route and its provider address from one session to the next, as `session` does for page reads: the provider is asked for the same address whenever that key opens a session of that name, and an unnamed session is given a fresh one. Whether the address is held is the provider's to decide. No other parameter is accepted. The browser starts before the handshake completes, so every refusal is an ordinary HTTP answer with the JSON error body: | HTTP | Codes | |---|---| | 400 | `invalid_request`, `country_unavailable`, `network_unavailable` | | 401 | `unauthorized` | | 404 | `not_found` when the Browser API is not enabled | | 426 | `upgrade_required` for a request that is not a version 13 WebSocket upgrade | | 429 | `session_limit`, `service_busy`, `daily_limit`, `rate_limited`, with `Retry-After` | | 503 | `browser_unavailable`, `service_unavailable` | ### What a session may ask of its browser Every command passes an allowlist before it reaches the browser. A refused command is answered with a CDP error, `-32601` for a method that is not available and `-32602` for parameters that are not, and the browser never sees it. The relay refuses what would reach past the session: - **This machine's files**: `DOM.setFileInputFiles`, `Page.handleFileChooser`, `Input.dispatchDragEvent` carrying files, `Network.loadNetworkResource` and `DOM.getFileInfo`. A navigation by `Page.navigate`, `Target.createTarget` or a rewritten `Fetch.continueRequest` must be to an `https:`, `http:` or `data:` URL, or `about:blank`. File uploads are therefore unavailable. - **The network guard**: `Target.createBrowserContext` with `proxyServer`, `proxyBypassList` or `originsWithUniversalNetworkAccess`, and `Target.setRemoteLocations`. - **The machine itself**: `Tethering`, `SystemInfo`, `Extensions`, `PWA`, `Cast`, `Memory` and `Tracing`, and every `Browser` method except `getVersion`, `close`, the window and permission methods, and `setDownloadBehavior`, which is rewritten as below. - **The relay**: `Target.exposeDevToolsProtocol` and `Target.sendMessageToTarget`. Attach with `flatten: true`, as Puppeteer and Playwright do. Downloads are rewritten rather than refused: `Browser.setDownloadBehavior` and `Page.setDownloadBehavior` are forwarded with `behavior: "deny"`, because clients ask for downloads into a directory on their own machine. A domain or method not on the allowlist, including one added in a later Chromium, is refused until it is reviewed. The relay keeps one browser session for itself. It denies downloads in every new browser context before the context's id reaches the client, and it closes any page past the page limit. Its session id is refused in a client's commands and its events are not forwarded. Network access is the same as for page reads: public HTTPS destinations on port 443 through the guarded proxy, each address checked and pinned, within the operator's `VITNA_WEB_ALLOWED_HOSTS`. Secure WebSockets are carried; QUIC and unproxied UDP are disabled. A plain HTTP page is answered with 403 by the guard itself. ### Session limits and close codes | Environment | Default | Meaning | |---|---|---| | `VITNA_WEB_BROWSER_API` | `false` | `true` enables `/v1/browser` | | `VITNA_WEB_BROWSER_API_SESSIONS` | 2 | Sessions open at once in this process, 1 to 20 | | `VITNA_WEB_BROWSER_API_SESSIONS_PER_CLIENT` | 1 | Sessions open at once for one client, at most the process limit | | `VITNA_WEB_BROWSER_API_SESSION_MS` | 300,000 | Whole-session limit, 10,000 to 3,600,000 | | `VITNA_WEB_BROWSER_API_IDLE_MS` | 120,000 | Time without a client command, 5,000 to 3,600,000 | | `VITNA_WEB_BROWSER_API_MAX_BYTES` | 104,857,600 | Network transfer through the guard, both directions, 1 MiB to 10 GiB | | `VITNA_WEB_BROWSER_API_MAX_PAGES` | 10 | Pages open at once; the relay closes the newest past it, 1 to 100 | A session also allows 10 browser contexts, 2,000 commands awaiting the browser, 4 MiB per client message and 64 MiB per browser message. Output a client leaves unread is capped at 64 MiB. The connection closes with: | Code | Reason | |---|---| | 1000 | The client closed the connection, or sent `Browser.close` | | 1001 | `service_shutdown` | | 1007 | A message that was not valid UTF-8 or JSON | | 1008 | A message that was not a command, or `client_not_reading` | | 1009 | A client message over 4 MiB | | 1011 | The browser failed while the session ran | | 4001 | `session_timeout` | | 4002 | `idle_timeout` | | 4003 | `transfer_limit` | | 4004 | `browser_exited`, a browser that exited unasked | | 4005 | `output_too_large`, a browser message over 64 MiB | A session is one request against the client's per-minute and daily limits, or one shared-quota admission. Its browser time, from launch through cleanup, is recorded as browser milliseconds and priced at `VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD`, like every other browser operation. A session through a residential or ISP route also records its transferred bytes for per-GB pricing. The session's entry in `/v1/usage` and its receipt carry the operation `browser`, the close reason as its outcome, `closeCode`, `network`, `browserProfile`, `commands`, `refusedCommands` and `transferredBytes`. `GET /v1/status` reports `capabilities.browserApi` and, when enabled, `limits.browserApi`, whose `execution` says where sessions run: `{"mode":"local"}`, or the workers and their combined capacity. Its `browserProfile` is the profile the calling key's sessions launch with. ### Browser profiles A session's browser is `native` unless the operator grants its account the `cloakbrowser` profile, the Unblocker's profile of that name: an operator-installed [CloakBrowser](../unblocker/overview.md#browser-profiles-and-live-checks) executable, launched with the same fingerprint flags and a 1920 by 947 desktop window. A named session keeps its fingerprint as it keeps its address, so it returns as the same device; an unnamed one draws a new fingerprint. ```dotenv VITNA_WEB_BROWSER_API_CLOAKBROWSER_CLIENTS=["acct_ID","pilot"] VITNA_WEB_BROWSER_API_CLOAKBROWSER_PATH=/opt/cloakbrowser/chrome ``` The grant lists client ids, a static key's name or an account's id, 1 to 1,000 of them. Like a proxy-access grant it is the operator's check on the customer: confirm who the customer is and what the traffic is for before adding one. Public signup never adds an account to it. Every other key's sessions stay native, on `VITNA_WEB_BROWSER_PATH` or the bundled Chromium. The path is set here when sessions run on this machine, and on each worker otherwise (below); startup refuses a path without grants, or grants without a path to launch them. The CloakBrowser [binary license](https://github.com/CloakHQ/CloakBrowser/blob/main/BINARY-LICENSE.md) is the operator's to review. The relay does not solve challenges. A profile changes how the browser presents itself, and the network guard and allowlist are the same for every profile. Verification pages, CAPTCHAs and interstitials reach the client as they would reach any browser, and what to do with them is the client's. This differs from page reads, where the Unblocker waits on a recognised Cloudflare challenge for its client. By default, sessions launch on the Web process's own machine, including when page reads use owned browser workers. The executable is `VITNA_WEB_BROWSER_PATH`, or Playwright's bundled Chromium installed by `npm run web:install` and `npm run web:browser`; a browser channel alone is not accepted. The browser runs with its sandbox on, so the host must support it, on Linux through unprivileged user namespaces or the setuid sandbox. A reverse proxy must pass the upgrade through: `deploy/vitna-web.nginx.conf.example` has a `location = /v1/browser` block that does. ### Sessions on owned workers To run sessions on machines you operate, list them in `VITNA_WEB_BROWSER_API_WORKERS`: ```dotenv VITNA_WEB_BROWSER_API=true VITNA_WEB_BROWSER_API_SESSIONS=8 VITNA_WEB_BROWSER_API_WORKERS=[{"id":"browser-a","origin":"https://browser-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","sessions":8}] ``` Each worker is the `npm run web:worker` process described under [owned browser workers](#owned-browser-workers), with `VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS` set to the sessions it may hold (1 to 20; unset or `0` leaves it off). It uses its own `VITNA_WEB_BROWSER_PATH` and `VITNA_WEB_ALLOWED_HOSTS`, with the same rule that a channel alone is not accepted. `sessions` defaults to 2, up to 20 per worker and 20 workers, and `VITNA_WEB_BROWSER_API_SESSIONS` may not exceed their total. These workers are separate from `VITNA_WEB_BROWSER_WORKERS`, so a machine can serve Browser API sessions without page reads, or both. The Web process keeps the client's connection, the command allowlist and every limit above; the worker runs the browser and its network guard. They share one connection per session, an HTTP upgrade to `/v1/cdp` on the worker's server authenticated with its operator token, which carries the DevTools messages and the guard's transfer count. The operator proxy route a session selects is sent over that connection, credentials included, so give workers the same trust as the Web host. A worker answers the upgrade only once its browser is ready, so a worker that is full or cannot launch is passed over for the next, most free sessions first; a session already running is never moved. With every worker full, `/v1/browser` answers 429 `service_busy`. A worker closes a session's browser when its connection drops, when the worker stops, or 30 seconds past `VITNA_WEB_BROWSER_API_SESSION_MS` if the Web process never ends it. Its `/healthz` reports `browserApi` with its sessions, the profiles it can launch and how many are active. A worker offers the `cloakbrowser` profile only with its own `VITNA_WEB_BROWSER_WORKER_CLOAKBROWSER_PATH`, an absolute path, and its native sessions keep using `VITNA_WEB_BROWSER_PATH`. Grants stay on the Web process. A worker asked for a profile it cannot launch answers 422 `profile_unavailable` and is passed over like a full one; when no worker offers the account's profile, `/v1/browser` answers 503 `browser_unavailable`. ## Proxy access ```text GET /v1/proxy GET /v1/proxy/keys POST /v1/proxy/keys DELETE /v1/proxy/keys/KEY_ID ``` Datacenter proxies from Vitna's own egress gateways, for your own HTTP clients. The operator grants access account by account, with a monthly allowance. The account owner then mints proxy keys with the owner token. The key is shown once: ```bash curl -X POST https://web.example.com/v1/proxy/keys \ -H "Authorization: Bearer $VITNA_WEB_OWNER_TOKEN" \ -H "Content-Type: application/json" -d '{"label":"crawler"}' ``` The response carries `username` (the account), `proxyKey`, and the key's `id` and `label`. Connect through any gateway in `GET /v1/proxy` as an HTTPS proxy: ```bash curl -x "https://ACCOUNT:vitna_proxy_KEY@egress-de.example.com:8791" https://example.com/ ``` `GET /v1/proxy` returns `access` (`granted`, `suspended`, or `not_granted`), `username`, `endpoints` (each gateway's host, port, and country), `limits` (`monthlyBytes`, `concurrency`, `tunnelsPerMinute`), the month's `usage.bytes`, and the keys without their secrets. Revoking a key takes effect at every gateway within about a minute, including on tunnels already open. Proxy routes take the owner token, never an API key, and are relayed by the website like the other account routes. Only HTTPS destinations are reachable: CONNECT to port 443 of a public address. A refusal names its reason in an `X-Vitna-Error` header, listed in the [gateway reference](../unblocker/overview.md#run-an-owned-egress-gateway). Bytes in both directions are metered and settled hourly into the account's statement as `proxyBytes`. They are priced per decimal GB at `VITNA_WEB_BILLING_PROXY_GB_MICRO_USD`. Settled and still-open hours both count against the month's allowance, which gateways enforce within about a minute of each other. The operator configures the grants and gateways: ```dotenv VITNA_WEB_PROXY_ACCESS={"acct_ID":{"monthlyGb":50,"concurrency":10,"tunnelsPerMinute":600}} VITNA_WEB_PROXY_GATEWAYS=[{"id":"egress-de","tokenHash":"GATEWAY_TOKEN_HASH","endpoint":"egress-de.example.com:8791","country":"DE"}] VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000 ``` | Grant field | Default | Meaning | |---|---|---| | `monthlyGb` | Required | Decimal GB per UTC month, 1 to 100,000 | | `concurrency` | 10 | Tunnels open at once, 1 to 1,000 | | `tunnelsPerMinute` | 600 | Tunnels started per minute, 1 to 100,000 | | `suspended` | `false` | `true` stops the account's access and closes its tunnels | A grant names an account this service holds, and public signup never creates one. A grant is the operator's check on the customer: confirm who the customer is and what the traffic is for before adding one. Changes take effect on restart, and gateways see them within a minute. Each gateway's `tokenHash` comes from `npm run egress:gateway-key` on that gateway, and its `endpoint` is the public host and port customers connect to. ## Product and company data ```text POST /v1/product POST /v1/company ``` Both accept a URL and the common browser options. Extraction reads published JSON-LD, microdata/RDFa, supported embedded application data, and page metadata, with bounded fallbacks for explicit labels and supported factual prose. These routes run locally without model configuration. Product data contains `name`, `description`, `price`, `currency`, `availability`, `sku`, `brand`, `images`, `offers`, and `variants`. Published `url`, `gtin`, and `mpn` can also be returned. Records are matched using page identity, canonical URLs, identifiers, and primary-entity references. Related records for the same product can contribute offers and variants; unrelated product cards do not contribute a price to the selected product. When offers publish different prices, `price` is `null` and `priceRange` can contain observed `low` and `high` bounds with `currency`. A published single bound remains a single bound. Prices in different currencies are not combined into a range. Offers preserve their own `price`, `currency`, `availability`, and `url`, plus published `lowPrice`, `highPrice`, `offerCount`, or `seller`. A variant price becomes the selected price only when the page identifies that variant; otherwise variant prices remain in `variants` and `offers`. On a Shopify storefront, recognized by Shopify's own page markers, a product URL (`/products/<handle>`, under an optional locale or collection) also reads the store's own record from `/products/<handle>.json`. That record sits beside the page's as `platform`: `name: "shopify"`, `productId`, `handle`, `vendor`, `productType`, `tags`, `options`, `priceRange`, and every variant with its `sku`, `barcode`, `price`, `compareAtPrice`, `currency` and `options`. Its values fill only the fields the page left empty: `brand` from the vendor, `currency`, `images`, and the `sku` and `gtin` of a product with one variant. The page's own price and availability stand. When the page's published data cannot be resolved, for example a graph past the 25,000-node bound, the store's record answers alone, with one offer per variant; when the store's record cannot be read, the page's record stands as on any other site. The record is a plain public HTTPS read with no browser and no further page quota. Company data contains `name`, `description`, `url`, `industry`, `locations`, `contactLinks`, `socialLinks`, and `products`. Published `legalName`, `foundingDate`, `employees`, `email`, and `telephone` can also be returned. Multiple distinct contacts produce `emails` or `telephones` arrays. A scalar contact is returned when the page identifies a primary contact or publishes only one value. Public contact links and `sameAs` social profiles are retained; a directory's own footer contacts are not assigned to an external company it profiles. An unresolved choice between companies returns `extraction_ambiguous` (HTTP 422). Unknown baseline scalar fields are `null` and baseline lists are empty. Additional fields appear only when supported by published values. Prices, dates, employee counts, locations, and contacts are not estimated from other entities, approximate prose, or missing data. ```json { "url": "https://example.com/", "render": true } ``` Available fields depend on what the page publishes. These endpoints do not fill missing company facts or product details with invented values. ## Store catalogs ```text POST /v1/catalog ``` List a Shopify or WooCommerce store's products from the store's own catalog: ```json { "url": "https://shop.example.com/", "maxProducts": 250 } ``` `url` is any page of the store; its origin is the store. `maxProducts` accepts 1 to 1,000 and defaults to 100. The store's home page says which platform it runs on, through Shopify's or WooCommerce's own page markers or a WordPress REST link. When the home page names neither, as on a headless storefront, each platform's catalog is tried in turn, Shopify first, and each attempt counts as a page. The response gives: - the `store` origin, after following at most two redirects (such as a `myshopify.com` address, or a bare domain, to the store's own address); - `platform`, `shopify` or `woocommerce`; - the store's `currency`; - `products`; - the catalog `pages` read; - `truncated`, when more products remain; - on WooCommerce, the store's `total` product count. Each product carries `id`, `handle`, `url`, `title`, `description`, `vendor`, `tags`, `options`, `images`, `priceRange`, `available` (whether any variant is), and its `variants`. Each variant has `id`, `title`, `sku`, `price`, `compareAtPrice`, `currency`, `available` and `options`. **Shopify.** The catalog is `/products.json`, read a page of 100 at a time. The `currency` comes from the storefront's own `Shopify.currency` script, and is `null` on a headless storefront without one. Every product has at least one variant. Products add `productType`, `publishedAt` and `updatedAt`, and variants add `barcode`. **WooCommerce.** The catalog is the Store API that WooCommerce publishes under the WordPress REST root, read up to 100 products a page in product id order. The root is `/wp-json/`, or the one the home page names, such as `/?rest_route=/` or a subdirectory install. A root on another origin is not followed. Each page's variable products have their variations read beside it, 100 at a time and up to 250 per product, so each variant carries its own SKU, price and stock. A variant whose attribute accepts any value names no option for it. A simple product has no `variants`; its `sku`, `price` and `available` are its own. Prices are converted from the Store API's minor units, and `vendor` is the product's brand. Products add: | Field | Value | |---|---| | `sku` | The product's own SKU | | `type` | `simple`, `variable`, `grouped` or `external` | | `categories` | Category names | | `attributes` | The product's attributes that do not define variants, each with its `name` and `values` | | `price`, `compareAtPrice` | The product's one price and its regular price when on sale; `null` when its variations are priced differently | | `currency` | The store's currency code | | `onSale`, `onBackorder` | As the store reports them | | `lowStockRemaining` | Where the store shows a low-stock count | | `rating` | `average` and `count`, where the product has reviews | The catalog is read with plain public HTTPS requests through the same guard as crawl discovery, with no browser. Every catalog read counts against the page quota: one per 100 Shopify products, and on WooCommerce one per page of products plus one per 100 variations. The request counts once against the request limits. A WooCommerce page may decode to 32 MiB; its compressed transfer stays within the 2 MiB bound of every public read. A catalog request must finish within the service's request deadline, 30 seconds by default. WooCommerce stores often serve the Store API uncached, and a page of 100 products can take several seconds. For a large catalog on a slow store, set `async: true` to read it as a background job: ```json { "url": "https://shop.example.com/", "maxProducts": 1000, "async": true, "jobTimeoutMs": 600000 } ``` The response is a job receipt (HTTP 202) with `kind: "catalog"`. Poll `GET /v1/jobs/:id`: the job's `progress` counts each catalog read as it happens, and its `result` is the catalog answer described above. `jobTimeoutMs` (300,000 by default, up to 600,000) and `webhookUrl` work as on other [background jobs](#background-jobs), and both require `async: true`. A job's result is stored whole, within five MiB. A catalog job therefore stops before the product that would take its result past that bound, and reads no more pages. It reports `truncated: true` with `truncationReason: "result_limit"`, as a crawl job does. Products measured on live stores average about 4 KB, so 1,000 products usually fit. A catalog cut short by `maxProducts` names no `truncationReason`. A service restart fails a running catalog job with `job_interrupted`; send it again. A queued catalog job survives a restart. A catalog dataset reads the same catalog on a schedule, with its 30-minute run deadline. The store's `robots.txt` is read first, under the agent name `VitnaWeb`. A store that disallows its catalog to that agent, or to every agent, is refused with `catalog_disallowed` (403). The catalog is `/products.json` on Shopify and the Store API path on WooCommerce. The store's `Crawl-delay`, up to two seconds, is kept between reads. These errors can also be returned: - `catalog_unsupported` (422): the site publishes neither catalog. - `catalog_unavailable` (502): a later WooCommerce page or variation read fails. A partial catalog is not returned. - `destination_blocked` (400): with `VITNA_WEB_ALLOWED_HOSTS` set, every read must stay in scope, including the redirects it follows. Requests identify themselves as `VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview)`. ## Careers ```text POST /v1/careers ``` A company's open roles from its own job board on Greenhouse, Lever, Ashby, Workable, Recruitee or Personio, read from each platform's public job board API: ```json { "url": "https://jobs.lever.co/acme", "maxJobs": 200, "descriptions": true } ``` `url` is the board (`job-boards.greenhouse.io/NAME` or `boards.greenhouse.io/NAME`, `jobs.lever.co/NAME` or `jobs.eu.lever.co/NAME`, `jobs.ashbyhq.com/NAME`, `apply.workable.com/NAME`, `NAME.recruitee.com`, `NAME.jobs.personio.de` or `.com`), one of its job pages, or a careers page that links or embeds exactly one board. `maxJobs` accepts 1 to 1,000 and defaults to 100. `descriptions`, true by default, includes each posting's description as plain text that keeps its lines; false leaves it out. The response gives the `board` (`platform`, `name`, an EU Lever board's or a Personio board's `region`, and its public `url`), `jobs`, the `pages` read, `truncated` when more postings remain, and, for Greenhouse, the board's own `total`. Each job carries the same fields on every platform, `null` or empty where the board publishes nothing: | Field | Meaning | |---|---| | `id` | The board's id for the posting | | `title`, `url`, `applyUrl` | The posting, its page, and where to apply | | `company` | The company name, where the board gives one (Greenhouse, Workable, Recruitee, Personio) | | `department`, `departments`, `team` | The board's own grouping | | `locations`, `country` | Every location the posting names | | `workplaceType`, `remote` | `remote`, `hybrid` or `onsite` as the board states it, and whether the role is remote | | `employmentType` | The board's own words, such as `FullTime` or `Permanent` | | `publishedAt`, `updatedAt` | ISO timestamps, where published | | `compensation` | Lever's or Recruitee's published salary range (`min`, `max`, `currency`, `interval`), or Ashby's `summary` where the company shows pay on its postings | | `description` | The posting's text, when `descriptions` is true | | `platform` | `greenhouse`, `lever`, `ashby`, `workable`, `recruitee` or `personio` | Boards are read with plain public HTTPS requests through the same guard as store catalogs, with no browser, and the API host's `robots.txt` is read first under the agent name `VitnaWeb`. Every page read counts against the page quota: one for a Greenhouse, Ashby, Workable or Recruitee board, one per 50 postings on Lever (with Lever's `Crawl-delay` kept between them), and one for a careers page that has to be read to find its board. A Personio board is one read, or up to three with descriptions, since Personio's feed carried its description sections only when a language was named (measured 2026-10-07): it is read as it stands, then in English, then in German, until one carries them. With `VITNA_WEB_ALLOWED_HOSTS` set, the board's API host must be in scope as well as the `url`. Postings an Ashby board marks unlisted are left out, as are Recruitee offers not published. Workable says only whether a role is remote, so its other roles have no `workplaceType`. A renamed Recruitee company's board redirects to its new name, which is followed. A whole board's response may decode to 32 MiB; its compressed transfer stays within the 2 MiB bound of every public read. | HTTP | Codes | |---|---| | 403 | `job_board_disallowed` when `robots.txt` disallows the board's API, which Personio's does for a company it does not know | | 404 | `job_board_not_found` for a board that does not exist | | 422 | `job_board_unsupported` when no board is found at `url`; `job_board_ambiguous` when a careers page links more than one, naming each | | 502 | `job_board_unavailable` when the board answers with anything but its postings | The platforms document these APIs as public (retrieved 2026-10-07): the [Greenhouse Job Board API](https://docs.greenhouse.io/job-board.html), the [Lever Postings API](https://github.com/lever/postings-api), Ashby's [public job posting API](https://developers.ashbyhq.com/docs/public-job-posting-api), Recruitee's offers API (`NAME.recruitee.com/api/offers/`), and Personio's [XML job feed](https://developer.personio.de/docs/retrieving-open-job-positions); Workable's board widget API (`apply.workable.com/api/v1/widget/accounts/NAME`) is the one its own job widget reads. SmartRecruiters publishes a posting API too, but its API host's `robots.txt` disallows every agent but LinkedIn's, so it is not read. A careers page rendered only by script names no board in its HTML; send the board's own URL instead. ## Investor Relations (`POST /v1/ir`) ```text POST /v1/ir ``` Discover and classify Investor Relations presentations, press releases, financial reports, earnings transcripts, and webcasts from a company URL or IR portal via Q4 feeds and rendered IR pages: ```json { "url": "https://investor.nvidia.com/", "kinds": ["presentation", "press_release", "financial_report"], "maxItems": 25 } ``` ## Document parsing (`POST /v1/parse`) ```text POST /v1/parse ``` Parse a public HTTPS URL or base64-encoded PDF, DOCX, XLSX, or PPTX document into structured pages, typed blocks, extracted tables, Markdown, and character citation offsets: ```json { "url": "https://example.com/report.pdf", "format": "both", "maxPages": 100 } ``` ## Claim verification (`POST /v1/verify`) ```text POST /v1/verify ``` Deterministically verify a factual claim against supplied URLs or native search results, returning per-segment support status, verbatim quotes with character offsets, content hashes, and a SHA-256 attestation digest: ```json { "claim": "Example Domain is maintained for illustrative examples.", "urls": ["https://example.com/"], "maxSources": 3 } ``` ## Structural change detection (`POST /v1/diff`) ```text POST /v1/diff ``` Compare two public URLs (`beforeUrl`, `afterUrl`) or two inline text snapshots (`beforeText`, `afterText`) across `prose`, `pricing`, `policy`, or `risk_factors` modes with materiality scores and a SHA-256 attestation digest: ```json { "beforeText": "Starter is $29/mo.", "afterText": "Starter is $39/mo.", "mode": "pricing" } ``` ## Entity alt-data dossiers (`POST /v1/entity`) ```text POST /v1/entity ``` Assemble a company dossier in parallel across requested modules (`profile`, `careers`, `ir`, `infra`, `filings`, `news`): ```json { "domain": "nvidia.com", "ticker": "NVDA", "modules": ["profile", "careers", "ir", "infra", "filings", "news"] } ``` ## Enrichment matrices (`POST /v1/grid`) ```text POST /v1/grid ``` Enrich an `entities` by `columns` table synchronously (up to 10 entities, or asynchronously with `async: true` for up to 100 entities) using `entity`, `search`, `verify`, or `source` column strategies with cited source URLs, quotes, and attestation digests. Pass `discover` (with `query`, optional `entityType`, `count`, `criteria`, `domains`, `excludeDomains`, `queries`, `index`, and `asOf`) and optional hard `budget` controls (`maxSearches`, `maxVerifications`, `maxDurationMs`) to discover and verify matching entities in one call, with or without explicit `entities` or `columns`: ```json { "discover": { "query": "AI inference hardware startups", "entityType": "company", "count": 10, "criteria": ["founded after 2022", "builds custom silicon"] }, "budget": { "maxSearches": 6, "maxVerifications": 20 }, "columns": [{ "id": "org", "strategy": "entity", "field": "profile.name" }] } ``` Pass the same `discover`, `budget`, `entities`, and `columns` definition inside `source.grid` on `POST /v1/datasets` to maintain it as a recurring living webset with incremental diffs, webhooks, and CSV/JSONL export. ## Datasets ```text POST /v1/datasets GET /v1/datasets GET /v1/datasets/DATASET_ID DELETE /v1/datasets/DATASET_ID POST /v1/datasets/DATASET_ID/run GET /v1/datasets/DATASET_ID/records?run=latest&format=jsonl GET /v1/datasets/DATASET_ID/changes?after=RUN ``` A dataset is a collection you define once and Vitna keeps current. You give it a source, the field that identifies a record, and how often to run. Each run keeps a snapshot of the records and what changed since the run before. ```json { "name": "Competitor prices", "source": { "recipe": "RECIPE_ID", "urls": ["https://shop.example.com/p/1", "https://shop.example.com/p/2"] }, "key": "url", "ignoreFields": ["checkedAt"], "intervalSeconds": 86400, "webhookUrl": "https://hooks.example.com/vitna" } ``` | Source | Fields | Records | |---|---|---| | A recipe over pages | `recipe`, `urls` (1 to 100) | One per page: its `url` and the recipe's fields | | A recipe over a site | `recipe`, `url`, `maxPages` (1 to 100, default 20), `includePaths`, `excludePaths` | One per page the site map finds | | A store catalog | `catalog` (a Shopify or WooCommerce store's URL), `maxProducts` (1 to 1,000, default 250) | One per product, as `/v1/catalog` returns it | | A careers board | `careers` (a board or careers page URL), `maxJobs` (1 to 1,000, default 250), `descriptions` (default true) | One per posting, as `/v1/careers` returns it, keyed by `id` | | Vitna's market record | `market`: `model`, `tooling`, `extraction`, `speech`, `retrieval`, or `compute` | One per figure on the class's tape, with an `id` naming its sku, seller, meter and any band | | A grid (living webset) | `grid`: `discover`, `budget`, `entities`, `columns`, `concurrency` (same contract as `POST /v1/grid`, without task fields) | One flat record per matched or supplied entity (`id`, `name`, `domain`, `ticker`, `url`, optional `verification`, and each column id) | `key` names the field that identifies a record from one run to the next. It defaults to `url` (`id` for a market, a careers board, or a grid); records without it are skipped and counted. When a `grid` run hits a hard `budget` cap or a column cell error, the run is marked `partial`, preserves previously discovered records, and removes none. `ignoreFields` lists fields whose changes don't count, such as timestamps. `intervalSeconds` is 900 to 2,592,000, a day by default. The first run starts when the dataset is created. `POST .../run` runs it now; a dataset that is already running answers 409 `dataset_running`. `GET /v1/datasets/DATASET_ID` lists the runs, newest first. Each gives its `status`, the counts of `records`, `added`, `changed`, and `removed`, the `pages` read, and the first 100 page `errors` with their total. A run's status is one of: - `succeeded`: it read every page. - `partial`: it missed a page or stopped at a limit. It keeps the records it could not reach as they were and removes nothing. A record leaves the dataset only on a run that read its whole source. - `failed`: its source could not be read at all. The previous records stay. `GET .../records` returns a run's records as JSONL (`format=jsonl`, the default) or CSV (`format=csv`). CSV has one column per field, in the order fields first appear. Nested values are written as JSON, and text a spreadsheet would run as a formula gets a leading `'`. `run` is `latest` (the default) or a run number. `GET .../changes?after=RUN` returns what each run after `RUN` added, changed, and removed, oldest first, whole runs at a time, up to 10,000 changes. Added and changed entries carry the record as it now is; removed entries carry the key. Read on with `after` set to the response's `next`; `more` says further runs follow. `resync: true` means runs after yours have been pruned, so read the records afresh. With `webhookUrl`, each run sends a `dataset.run.completed` event with its counts, retried with backoff up to five times. Runs use the same browser slots, page quota, and accounting as API calls, one run at a time per service. A run that a restart interrupts starts again at the next tick. The limits: | Limit | Value | |---|---| | Datasets per key | 5 | | Records per run | 5,000, each up to 16 KiB | | Size per run | 8 MiB | | Runs kept | The last `retainRuns` (1 to 30, default 10), within 32 MiB | The latest run is always kept, and so is the run the next one is compared against. ### Published datasets ```text GET /v1/published GET /v1/published/SLUG GET /v1/published/SLUG/records?run=latest&format=jsonl GET /v1/published/SLUG/changes?after=RUN ``` The operator can publish datasets that every key can read. Vitna's market record is the main example: each class's prices, kept current, with a feed of what moved. Published datasets take the same `records` and `changes` parameters. No key can run or remove them, and reading one counts as a request like any other. ```dotenv VITNA_WEB_PUBLISHED_DATASETS=[{"slug":"market-model","name":"AI model prices","source":{"market":"model"},"ignoreFields":["confirmedAt","fetchedAt","evidenceSha256","observationId"],"intervalSeconds":3600}] ``` Each entry is a dataset definition with a `slug` (up to 48 lowercase letters, numbers, and hyphens) and no webhook. A published dataset keeps its runs across restarts while its source and key stay the same. It starts afresh when either changes, and is removed when the configuration drops it. A market source reads the record's public tape at `https://vitna.ai/api/v1/markets/{class}/tape`; `VITNA_WEB_RECORD_ORIGIN` names another origin. Ignoring the confirmation and evidence fields, as in the example, means a price that is re-confirmed unchanged doesn't count as a change. ## Firehose ```text GET /v1/firehose ``` A continuous stream of freshly crawled web pages: every news page in Common Crawl's news crawl (CC-NEWS), about 300,000 a day in many languages, read by cursor. Common Crawl publishes a news file every one to two hours. The service reads each file whole and keeps every HTTP 200 HTML page's title and text, in the order it read them, for a retention window of 72 hours by default. ```text GET /v1/firehose?after=0&limit=100&domain=reuters.com,de&language=en,de ``` | Parameter | Default | Meaning | |---|---|---| | `after` | 0 | The cursor: records after this one. Send the previous answer's `next` | | `limit` | 100 | Records per answer, 1 to 1,000 | | `domain` | Omitted | Up to 20 domains, separated by commas. A domain covers its subdomains, and a top-level domain such as `de` or a suffix such as `co.uk` covers every site under it | | `language` | Omitted | Up to 10 language codes, such as `en` or `de`, from the page's own `lang` attribute or `Content-Language` header | | `text` | `true` | `false` leaves each page's text out | ```json { "records": [ { "id": 4102, "url": "https://www.example.com/news/rates", "host": "www.example.com", "fetchedAt": "2026-10-07T14:58:25.000Z", "language": "en", "title": "Rates rise", "text": "The bank raised rates by a quarter point.", "htmlBytes": 280039, "digest": "sha1:NQ5RS4U4IVNJT3QEUFJYXWG45ZAZ2XDJ", "source": { "crawl": "CC-NEWS", "file": "crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz", "offset": 951, "length": 58004, "url": "https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz" } } ], "next": 4102, "more": true, "missed": false } ``` Read on by sending `next` as `after`. - `more` is true while further records already match. - When `more` is false, the cursor has caught up and `next` holds its place, so polling with it returns only the pages read since. - `missed: true` means records past the cursor were pruned before they were read. Each record's `fetchedAt` is when Common Crawl fetched the page. `language` is `null` when the page names none, and `digest` is the payload's SHA-1 as Common Crawl records it. `source` names the news file and the `offset` and `length` of the page's WARC record in it, so the page's HTML is one ranged read away: ```bash curl -r 951-58954 https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz | gunzip ``` The firehose is off until the operator sets `VITNA_WEB_FIREHOSE_PATH` to its store's SQLite file. Until then it answers `firehose_not_configured` (503), and `firehose_empty` (503) while the store holds nothing yet. `GET /v1/status` reports `capabilities.firehose` and `limits.firehoseRecords`. While the firehose is on, it also reports `firehose`: the store's `files`, `records`, its newest `latestFile`, and the `from` and `to` of its pages' fetch times. Each record returned adds one `firehoseRecords` to usage and to an account's statement. The ledger rate is `VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD`, and the operator estimate rate for static clients is `VITNA_WEB_FIREHOSE_RECORD_USD`. A read counts once against the request limits and uses no page quota. Common Crawl publishes the crawl under its terms of use; the pages themselves remain their publishers'. ### Monthly crawl backfill ```text POST /v1/firehose/backfill ``` A backfill reads one domain's pages from a Common Crawl monthly crawl (CC-MAIN), in the firehose's record shape, so a feed can start with a site's recent history. It is always a background job: the request answers HTTP 202 with a job of `kind: "backfill"`. Poll `GET /v1/jobs/:id`; its `result` holds the pages. ```json { "domain": "example.org", "path": "/news/", "language": "de", "maxPages": 200 } ``` | Field | Default | Meaning | |---|---|---| | `domain` | Required | A registered domain, such as `example.org`, with its subdomains | | `path` | Omitted | A path prefix, such as `/news/`, read on the domain itself and its `www` host | | `crawl` | The newest | A monthly crawl, such as `CC-MAIN-2026-39` | | `language` | Omitted | One language code, such as `de`, as Common Crawl's own detection reports a page's languages | | `maxPages` | 100 | Pages read, 1 to 500 | | `jobTimeoutMs` | 600,000 | The job's deadline, up to 600,000 milliseconds | | `webhookUrl` | Omitted | A public HTTPS destination told when the job finishes | Common Crawl's URL index lists the domain's captures in URL order. A URL captured more than once in the crawl is read once, from its latest capture. Up to `maxPages` pages are read, each with one ranged request, and each record is checked against the index before it is kept: its URL, its length and its payload digest. Aim a backfill at a section with `path`, since URL order otherwise starts wherever the site's addresses sort first. A record's `language` is still the one the page declares, which can differ from the index's detection. The result gives: - `crawl` and `domain`, and `path` or `language` when they were asked for; - `records`, each with `source.crawl` naming the monthly crawl, and no `id`, since a backfill's pages are not in the stream; - `captures`, the index rows read (up to five for each page asked for); - `pages`, the records read, and `skipped`, those that failed a check; - `truncated`, when more pages exist than were read. A backfill can also stop early, keeping the pages it already read, with a `truncationReason`: `result_limit` when the job's five-MiB result is full, or `commoncrawl_blocked` or `commoncrawl_unavailable` when Common Crawl stops answering. Many publishers block Common Crawl's crawler, and a backfill of such a domain finds nothing (`captures: 0`). Every Common Crawl request from every backfill takes its turn at one a second by default, so 100 pages take about two minutes. The host shares Common Crawl's allowance with Vitna Search's archive imports. - `VITNA_WEB_FIREHOSE_BACKFILL_RATE` raises the pace, up to 4 a second. - A busy index (a 429 or a 5xx) is asked again twice, 10 and 20 seconds later. - A 403 stops every backfill for 20 minutes. New ones answer `commoncrawl_blocked` (503) until then, and `GET /v1/status` reports `firehoseBackfill.blockedUntil`. Each record returned is one `firehoseRecords` unit. Backfills are off until the operator sets `VITNA_WEB_FIREHOSE_BACKFILL=true`, and answer `firehose_backfill_not_configured` (503) until then. They do not need the stream's store. `GET /v1/status` reports `capabilities.firehoseBackfill`, `limits.backfillPages` and, while backfills are on, `firehoseBackfill` with its `ratePerSecond`. A service restart fails a running backfill with `job_interrupted`. ### Structured topic events ```text GET /v1/firehose/events ``` Aggregates normalized public-registry and SEC filing events across up to eight structured topics (`cisa-kev`, `federal-register`, `fda-drug-recalls`, `fda-device-recalls`, `treasury-debt`, `sec-8k`, `sec-13d`, `sec-form4`) with a 5-minute in-memory TTL cache per topic: ```text GET /v1/firehose/events?topic=cisa-kev,sec-8k&since=2026-10-01&limit=20 ``` | Parameter | Default | Meaning | |---|---|---| | `topic` | All 8 topics | Comma-separated or repeated topic names | | `since` | Omitted | Earliest publication date formatted `YYYY-MM-DD` | | `limit` | 20 | Maximum events returned across selected topics, 1 to 100 | ### Ingestion The Web service only reads the store. `platform/web/firehose-run.mjs` fills it, run on a timer (`deploy/web-api/firehose.timer`) as the Web service's own user, so both can open the file. Each run: 1. Lists the current month's news files on `data.commoncrawl.org`, and the previous month's while the retention window reaches into it. 2. Reads up to `VITNA_FIREHOSE_FILES_PER_RUN` files it does not yet hold, oldest first. 3. Prunes pages past the window or the text budget, oldest file first. A file is one request: about 1 GB and 26,000 pages, measured on 2026-10-07, read in about two minutes. That keeps far inside Common Crawl's request limits. - **A 403 stops the run** at once, with no further request, and the run exits with status 2 so the block shows in the journal. - **A 429, a 5xx or a dropped connection** stops the run until the next one. - **A run that stops part-way keeps the pages it read** and resumes at the next record with a ranged read, so no page is stored twice. - **One run at a time:** a lease in the store keeps a second run from starting while one is still going. | Setting | Default | Meaning | |---|---|---| | `VITNA_FIREHOSE_PATH` | Required | Absolute path of the store's SQLite file; the Web service's `VITNA_WEB_FIREHOSE_PATH` names the same file | | `VITNA_FIREHOSE_RETAIN_HOURS` | 72 | Hours of pages kept, by when their file was crawled, from 1 to 720 | | `VITNA_FIREHOSE_MAX_TEXT_GB` | 6 | Text kept at most, in GiB; past it the oldest files' pages go first | | `VITNA_FIREHOSE_MIN_FREE_GB` | 10 | A run reads no file while the disk has less free space than this | | `VITNA_FIREHOSE_FILES_PER_RUN` | 2 | Files read in one run, from 1 to 24 | Pages average 4.2 KB of text, so 72 hours of news is about 4.5 GB. The raw CC-NEWS crawl stream does not yet carry: - the monthly crawl as a stream, only as a backfill of one domain at a time; - each page's HTML in the answer; - delivery by webhook or to a storage bucket. Structured public-registry event topics are served separately at `GET /v1/firehose/events`. ## Research ```text POST /v1/research ``` Search and read bounded page sets, then optionally return extractive findings: ```json { "query": "What does this company offer?", "maxSources": 3, "answer": true } ``` Discovery uses the local Vitna index by default; populate that index as described below, or supply your own `urls` array. `maxSources` defaults to 3 and accepts 1 to 20. With `answer: true`, the native engine selects observed source passages using lexical matches and returns numbered citations. Set `answer` to `false` to retrieve page content without an answer: ```json { "query": "What does this company offer?", "urls": ["https://example.com/"], "maxSources": 3, "answer": false } ``` The response includes the query, source page content, an answer when requested, and page errors. Native passage answers begin with "Extractive findings from supplied sources". Supported comparisons and arithmetic begin with "Grounded comparison" or "Grounded calculation" and show observed operands, units, formulas, and numbered citations. Supported arithmetic includes sum, average, difference, ratio, percentage change, minimum, and maximum over named entities and explicit compatible attributes. Conflicting facts, units, billing periods, missing facets, and unsupported qualifiers refuse instead of guessing. `generation` carries `engine: "vitna-native"`, `mode: "extractive"`, `planning: "intent_decomposition"`, and `reasoning: "bounded_rules_v1"`. Optional `analysis` contains validated coverage and finite calculations with existing source IDs. This is a finite rule engine; general reasoning still requires a separately operated self-hosted model. Unsupported requests return HTTP 422 `native_research_unsupported`. Supplied URLs bypass indexed discovery. Disabling Search or the answer engine can return `search_not_configured` or `model_not_configured` when the request needs that engine. ### Multiple searches and structured results Use search rounds to refine discovery after earlier pages have been read: ```json { "query": "Compare available products and pricing for this company.", "maxSources": 10, "maxQueries": 6, "maxRounds": 3, "maxPages": 20, "planQueries": true, "answer": true, "async": true, "jobTimeoutMs": 300000 } ``` | Field | Default | Meaning | |---|---|---| | `query` | Required | Question or research brief, up to 1,000 characters | | `maxSources` | 3 | Readable source budget, from 1 to 20 | | `maxQueries` | `maxRounds` | Search-query budget, from 1 to 10 | | `maxRounds` | 1 (`2` with `followLinks`) | Search/read rounds, from 1 to 3 | | `maxPages` | `maxSources` | Page-attempt budget, from 1 to 40 | | `followLinks` | `false` | Follow query-relevant links discovered on read pages across rounds (same host when `urls` are supplied) | | `planQueries` | `false` | Bounded query planning; deterministic source-based variants in native mode | | `schema` | Omitted | Supported JSON Schema for `data`; requires `answer: true` | | `stream` | `false` | Server-Sent Events progress and final result | | `async` | `false` | Submit a durable background job | | `jobTimeoutMs` | 300,000 | Background deadline, up to 600,000 milliseconds; requires `async: true` | | `webhookUrl` | Omitted | Public HTTPS completion destination; requires `async: true` | The input remains one `query`. The returned `queries` array records the searches actually executed. Up to 20 supplied `urls` bypass search discovery; they cannot be combined with `planQueries: true`. With `followLinks: true`, research scores unseen links found on each read page against the query terms and follows top-ranked candidates in subsequent rounds (scoped to the seed hosts when `urls` are supplied), bounded by `maxRounds`, `maxPages`, and `maxSources`. The response `budget` reports actual rounds, searches, page attempts, sources, model calls, context characters, limits, and the stopping reason. Provide a schema to receive validated JSON in `data` alongside the answer and collected pages. Native mode extracts supported literal values; an optional self-hosted model can generate schema-shaped data: ```json { "query": "List the products this company sells.", "urls": ["https://example.com/"], "answer": true, "schema": { "type": "object", "properties": { "products": { "type": "array", "items": { "type": "string" } } }, "required": ["products"], "additionalProperties": false } } ``` ### Timelines and multi-attribute comparisons Native Research can order observed historical events, select explicit earliest/latest events and calculate elapsed days between two cited dates. It accepts full ISO calendar dates, timezone-qualified ISO instants and unambiguous month-name/day/year dates. Year-only dates, numeric slashes, implicit timezones, mixed day/instant precision and crawl metadata are refused. Conflicting dates for the same named event remain ambiguous; requesting a year cannot resolve that contradiction. Structured responses expose `analysis.timeline` with at most 24 ordered events, normalized dates, precision and source IDs. Elapsed calculations use cited epoch-millisecond operands and an explicit day result. Multi-attribute comparisons expose at most 24 `analysis.comparisons` cells with entity, attribute, observed display value and source ID. Explicit date filters apply before selecting historical numeric records; absent or conflicting cells are refused rather than silently filled. All emitted analysis must pass the finite public response contract. The preserved native evaluation remains 44/44 cases. A separate authored timeline/comparison group passes 40/40, including 22 refusal diagnostics, in `outputs/web-native-evaluation/report-v3.json`. These fixtures exercise the bounded deterministic grammar, not open-ended general reasoning. ### Stream progress Set `stream: true` on a synchronous research request. The response content type is `text/event-stream`. Progress event names include `started`, `round`, `planning`, `search`, `source`, `source_error`, `generating`, and `completed`. Events include a monotonic sequence number and measured progress. The final `result` event contains the complete JSON response: ```text event: source data: {"type":"source","sequence":3,"url":"https://example.com/"} event: result data: {"query":"What does this company offer?","answer":null,"mode":"sources","sources":[],"meta":{}} ``` This wire example is illustrative. A stream failure uses an `error` event with `error.code` and `error.message`. Inspect that event even if the HTTP status was 200. The synchronous stream deadline is at most 60 seconds and may be shorter under the deployment settings. Disconnecting cancels the request. The playground displays progress, the final result, and stream errors; cURL users should include `-N` to see events as they arrive. ### Background jobs Set `async: true` for research, extraction or a [store catalog](#store-catalogs) that needs a longer deadline. A [firehose backfill](#monthly-crawl-backfill) is always a job. The create response uses HTTP 202 and includes `id`, `kind`, `status`, timestamps, `jobTimeoutMs`, `progress`, `result`, and `error`. Poll `GET /v1/jobs/:id`. Completed research, extraction, catalog or backfill output is in `result`; failed jobs carry `error`. `DELETE /v1/jobs/:id` cancels active work. `async` and `stream` cannot both be true. Async jobs can use an optional completion webhook. Store the job ID even when using a webhook, so progress and delivery outcomes remain inspectable. Job scheduling and callbacks require the service process to keep running. ## Sources ```text GET /v1/sources GET /v1/sources/:provider POST /v1/sources/find POST /v1/sources/run ``` Sources are official data providers, federal agencies and registries with public APIs, that an application or agent can discover, read the contract of, and call with the same key as every other Web API. Each provider offers capabilities, and each capability publishes a contract: its `input` and `output` as JSON Schema, its `price`, and the `terms` its data comes under. Reading the list, a provider's contracts, or a `find` is free. A call through `/v1/sources/run` is metered once it has returned data. Sources are off until the operator sets `VITNA_WEB_SOURCES=true`. `GET /v1/status` lists the enabled providers in `capabilities.sources` (empty while they are off), and every sources route answers `sources_not_configured` (503) until then. | Provider | Capabilities | Data | |---|---|---| | `federal-register` | `documents/search`, `documents/get` | Rules, proposed rules, notices and presidential documents, with agencies, dockets, comment deadlines and effective dates | | `ecfr` | `sections/search`, `sections/text` | The Code of Federal Regulations as in force today or on a past date, and a section's full text | | `usaspending` | `awards/search`, `agencies/list` | Federal contracts, grants and other awards, and each agency's budget authority and obligations | | `sec-edgar` | `companies/lookup`, `filings/recent`, `filings/search`, `facts/concept` | Companies' CIKs, their recent EDGAR filings, full-text filing search, and XBRL financial facts | | `openfda` | `drugs/labels`, `drugs/reactions`, `recalls/search`, `devices/events`, `drugs/recalls`, `devices/recalls` | Drug labeling, adverse event counts, food/drug/device enforcement recalls, and MAUDE medical device events and CDRH recalls | | `clinicaltrials` | `studies/search`, `studies/get` | Registered clinical studies, their status, phase, eligibility and outcomes | | `cisa-kev` | `catalog/search`, `catalog/summary` | CISA Known Exploited Vulnerabilities catalog search, ransomware campaign filter, and vendor summary | | `fred` | `series/observations` | St. Louis Fed FRED macroeconomic time series observations with period changes | | `finra` | `short-volume/ticker`, `short-volume/top` | Daily Reg SHO short sale volume by ticker and top shorted symbols | | `treasury` | `debt/daily`, `cash/tga`, `rates/average` | Treasury Debt to the Penny, TGA operating cash balance, and average interest rates | | `nyfed` | `rates/sofr`, `rates/effr`, `operations/rrp` | New York Fed daily SOFR, EFFR reference rates, and overnight Reverse Repo operations | | `bls` | `series/observations`, `series/latest` | Bureau of Labor Statistics CPI, unemployment, payroll, and price time series | | `courtlistener` | `courts/list`, `opinions/search`, `dockets/search` | Free Law Project U.S. court directory, judicial opinions, and RECAP federal court dockets | | `fcc` | `filings/search`, `proceedings/search` | FCC ECFS telecommunications docket filings and rulemaking proceedings | | `eia` | `energy/inventory`, `electricity/demand` | Weekly U.S. petroleum/natural gas inventories and hourly RTO electricity grid demand | | `dol-h1b` | `employers/approvals`, `lca/search` | USCIS H-1B Employer Data Hub approval/denial counts and DOL OFLC LCA wage filings | ### Find a capability Describe what you need, and optionally narrow it to a `category` (`government`, `finance`, `health` or `security`) or a `provider`: ```json { "query": "food recalls for listeria", "limit": 3 } ``` `limit` accepts 1 to 20 and defaults to 5. Each result is a whole contract, ranked by how well its title, keywords and description match the query, with its `score`: ```json { "query": "food recalls for listeria", "results": [ { "score": 9, "provider": "openfda", "capability": "recalls/search", "title": "Search FDA recalls", "description": "Recall enforcement reports for drugs, foods or medical devices: ...", "input": { "type": "object", "properties": { "product": { "type": "string", "enum": ["drug", "food", "device"], "default": "food" } }, "required": [], "additionalProperties": false }, "output": { "type": "object", "properties": { "total": { "type": "integer" } }, "required": ["total", "recalls"], "additionalProperties": false }, "example": { "provider": "openfda", "capability": "recalls/search", "options": { "product": "food", "query": "listeria", "classification": "Class I", "limit": 5 } }, "price": { "resource": "sourceRequests", "unit": "call", "microUsd": null, "discovery": "free" }, "available": true, "terms": { "license": "CC0 1.0 Universal (public domain dedication)", "url": "https://open.fda.gov/license/", "attribution": "Source: openFDA, U.S. Food and Drug Administration.", "notices": ["Do not present this data as endorsed by the FDA."], "retrieved": "2026-10-07" } } ] } ``` The schemas are shortened here. Without a `query`, `find` lists every capability in catalog order. `GET /v1/sources` lists the providers, and `GET /v1/sources/:provider` returns one provider with all of its contracts. Contracts are closed: an option the input schema does not name is refused with `invalid_options` (400), and the message names the option and the rule it broke, such as `options.title must be at most 50.` Every object in an output schema is closed too, and a call whose mapped result does not match its output schema fails with `source_contract` (502) rather than reaching the caller in an undeclared shape. ### Call a capability Send the provider, the capability and its options. `example` in every contract is a body that works: ```json { "provider": "ecfr", "capability": "sections/search", "options": { "query": "emergency exits", "title": 14, "limit": 5 } } ``` ```json { "provider": "ecfr", "capability": "sections/search", "data": { "total": 101, "page": 1, "pages": 21, "sections": [ { "citation": "14 CFR 25.807", "title": "14", "part": "25", "section": "25.807", "type": "Section", "heading": "Emergency exits.", "partHeading": "Airworthiness Standards: Transport Category Airplanes", "excerpt": "types of exits are defined as follows: (1) Type I. This type is a floor-level exit with a ...", "inForceFrom": "2016-12-30", "inForceUntil": null, "url": "https://www.ecfr.gov/current/title-14/section-25.807" } ] }, "source": { "name": "Electronic Code of Federal Regulations", "publisher": "Office of the Federal Register (National Archives) and the U.S. Government Publishing Office", "url": "https://www.ecfr.gov/api/search/v1/results?query=emergency+exits&per_page=5&page=1&date=current&hierarchy%5Btitle%5D=14", "retrievedAt": "2026-10-07T05:00:00.000Z", "license": "U.S. Government work, not subject to copyright in the United States (17 U.S.C. 105)", "licenseUrl": "https://www.copyright.gov/title17/92chap1.html#105", "attribution": "Source: the Electronic Code of Federal Regulations (eCFR), Office of the Federal Register.", "notices": ["The eCFR is authoritative but unofficial. Legal research should be verified against the official annual CFR, the daily Federal Register and the List of CFR Sections Affected."], "modifications": "Vitna returns the fields this contract names, under its own field names, with markup removed. A text field is cut at a word boundary at the length its description states (4,000 characters unless it says otherwise)." } } ``` Like every response it also carries `meta`. `source.url` is the upstream request the data came from, with any key removed. `dataAsOf` appears when the provider states how current its data is: openFDA's last update date, and ClinicalTrials.gov's processing timestamp exactly as its API writes it, with no time zone. Show `attribution` and `notices` wherever you display the data. ClinicalTrials.gov's terms ask redistributors to name it as the source, show the processing date, keep the data current and state modifications, which is what `dataAsOf` and `modifications` are for. Contact names, emails and phone numbers from study records are never returned. ### Errors | Code | Status | Meaning | |---|---|---| | `invalid_options` | 400 | The options break the contract; the message names the option | | `source_not_found` | 404 | No such provider or capability, or the provider has no such record | | `source_rejected` | 400 | The provider refused the query, in its own words where it gives them | | `source_rate_limited` | 429 | The provider is limiting requests; `Retry-After` says when to retry | | `source_capacity` | 429 | Vitna has used the provider's daily allowance; it resets at 00:00 UTC | | `source_busy` | 429 | Too many calls to that provider are already waiting | | `source_unavailable` | 502 or 503 | The provider could not be reached or answered with an error (502), or it needs operator configuration (503) | | `source_response_too_large` | 502 | The answer was over 2 MiB; narrow the request | | `source_contract` | 502 | The provider answered in a shape that does not match the contract | ### Pace and allowances Each provider has its own lane: its own concurrency, a minimum spacing between requests, and a daily ceiling where the provider publishes one. Vitna shares a provider's per-address allowance among all of its customers, so these are set below the providers' published limits (all read 2026-10-07): - SEC EDGAR allows 10 requests a second and requires every client to declare a user agent naming its operator and a contact address. Vitna sends 5 a second, and the provider is unavailable (`source_unavailable`, 503, with an `unavailableReason` in its contracts) until the operator sets `VITNA_WEB_SOURCES_CONTACT`. - openFDA allows 240 requests a minute and, without a key, 1,000 a day per address. Vitna keeps to 900 a day without a key, and 110,000 a day once the operator sets `VITNA_WEB_SOURCES_OPENFDA_KEY`, against openFDA's 120,000. - CourtListener allows keyless access to `/courts/` (`courts/list`), while `opinions/search` and `dockets/search` use an API token supplied via `VITNA_WEB_SOURCES_COURTLISTENER_KEY`. - FCC ECFS and EIA v2 work out of the box with `DEMO_KEY` (900/day ceiling) and scale when `VITNA_WEB_SOURCES_FCC_KEY` or `VITNA_WEB_SOURCES_EIA_KEY` is set. - `dol-h1b` reads USCIS H-1B Data Hub employer CSVs directly for `employers/approvals`, and uses `VITNA_WEB_SOURCES_DOL_KEY` for DOL OFLC `lca/search`. - FederalRegister.gov and eCFR.gov allow automated access only through their APIs and publish no rate limit. Vitna sends 2 requests a second to each. - ClinicalTrials.gov publishes no rate limit. Vitna spaces its requests 1.2 seconds apart. - USAspending.gov publishes no rate limit for its search API. Vitna sends 4 requests a second, 2 at a time. Requests identify themselves as `VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview)`, followed by the operator's contact address for SEC EDGAR. Reference lists (the SEC's ticker map, the eCFR title index, USAspending's agency list, ClinicalTrials.gov's processing date) are cached for an hour or a day once they load; a failed load is never cached. ### Metering `GET /v1/sources`, `GET /v1/sources/:provider` and `POST /v1/sources/find` are free. They count once against the request limits like any request. A `POST /v1/sources/run` that returns data adds one `sourceRequests` to usage and to an account's statement; a call that fails is not metered. The ledger rate is `VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD`, and every contract's `price.microUsd` shows it (`null` while it is unset). The operator estimate rate for static clients is `VITNA_WEB_SOURCE_REQUEST_USD`. One call is one capability. Batches of calls, feedback that credits a call, and a terms acceptance step for providers whose terms need one are not yet available. None of the current providers asks for a separate acceptance step. ## MCP ```text POST /v1/mcp ``` The Web API is also an MCP server, so an agent can find and call data sources, read pages and, where the operator offers it, search Google through the tools its client already understands. It uses the same `vitna_web_` key or org key as every other route, sent as a bearer token: ```bash claude mcp add --transport http vitna https://web.vitna.ai/v1/mcp \ --header "Authorization: Bearer $VITNA_WEB_KEY" ``` A client configured with JSON takes the same URL and header: ```json { "mcpServers": { "vitna": { "type": "http", "url": "https://web.vitna.ai/v1/mcp", "headers": { "Authorization": "Bearer vitna_web_..." } } } } ``` Connect to the API host itself. The website relay at `/api/web` does not forward `/v1/mcp` or MCP's headers. `GET /v1/status` reports the endpoint and the protocol versions it speaks in `capabilities.mcp`, and unauthenticated `GET /.well-known/mcp.json`, `GET /.well-known/agent-card.json`, `GET /openapi.json`, and `GET /llms.txt` expose the machine-readable schema and full tool catalog before signup. | Tool | What it does | Route it calls | |---|---|---| | `find_sources` | Ranks source capabilities for a task, each with its input schema and a working example | `POST /v1/sources/find` | | `describe_source` | A provider's contracts, or one capability's, with output schemas and terms | `GET /v1/sources/:provider` | | `run_source` | Calls one capability and returns its data with its source | `POST /v1/sources/run` | | `read_page` | Reads one public page as markdown or text, cut at 20,000 characters unless `maxCharacters` says otherwise; also accepts `onlyMainContent`, `includeTags`, `excludeTags`, `highlights`, `question`, `extract`, `injection` and `redact` | `POST /v1/read` | | `batch_scrape` | Reads up to 20 public pages in one call with shared reader, DOM filter, `highlights`, `question`, `extract`, `injection` and `redact` options; per-URL errors are isolated in `errors` | `POST /v1/batch` | | `extract_data` | Pulls fields from one page, up to 20 pages, or pages discovered from a starting URL, by selectors, a JSON Schema, a `prompt` (where a language model is configured), or a saved recipe; waits for the result | `POST /v1/extract` | | `map_site` | A site's URLs from its links and optionally its sitemaps, without reading them | `POST /v1/map` | | `crawl_site` | Starts a background crawl and returns its job id: 25 pages of 20,000 characters each, for up to 5 minutes, unless asked otherwise; accepts per-page DOM filters, `highlights`, `question`, `extract`, `injection` and `redact` | `POST /v1/crawl` | | `get_crawl` | A crawl's status and a slice of its pages, 25 at a time and without their text unless `includeContent` asks | `GET /v1/jobs/:id` | | `cancel_crawl` | Stops a queued or running crawl; pages already read stay readable | `DELETE /v1/jobs/:id` | | `research_web` | Runs bounded multi-page research across search hits or supplied `urls`, with optional `followLinks` and structured `schema` output; waits for the result | `POST /v1/research` | | `list_open_roles` | A company's open roles from its Greenhouse, Lever, Ashby, Workable, Recruitee or Personio board: the first 25 postings without their text unless `maxJobs` or `descriptions` says otherwise | `POST /v1/careers` | | `diff_pages` | Deterministic structural change detection and redlines across two URLs or inline text snapshots with materiality scoring and SHA-256 attestation | `POST /v1/diff` | | `parse_document` | Parses PDF, DOCX, XLSX or PPTX into structured pages, typed blocks, tables, markdown and character citation offsets | `POST /v1/parse` | | `verify_claim` | Deterministically verifies a factual claim against supplied URLs or search discovery with verbatim evidence offsets | `POST /v1/verify` | | `entity_dossier` | Builds a live company alt-data dossier across web profile, ATS roles, IR releases, DNS/SaaS infra, SEC filings and news | `POST /v1/entity` | | `enrich_grid` | Evaluates a synchronous entities-by-columns enrichment matrix across entity, search, verify and official source strategies | `POST /v1/grid` | | `discover_grid` | Discovers and verifies up to 10 entities against explicit `criteria` with verbatim quotes and `sha256:` attestations under hard `budget` controls, with optional `columns` enrichment | `POST /v1/grid` | | `investor_relations` | Discovers Investor Relations presentations, earnings releases, financial reports, transcripts and webcasts | `POST /v1/ir` | | `list_firehose_events` | Reads normalized CISA KEV, Federal Register, FDA, Treasury, SEC 8-K, SC 13D and Form 4 event feeds | `GET /v1/firehose/events` | | `capture_screenshot` | Renders a page in Chromium and returns a PNG or JPEG screenshot | `POST /v1/screenshot` | | `capture_pdf` | Renders a page in Chromium to PDF with `pageRanges`, `scale` and `printBackground` controls | `POST /v1/pdf` | | `list_linked_documents` | Discovers or downloads PDF, DOCX, XLSX and PPTX files linked from a page | `POST /v1/documents` | | `search_index` | Ranked pages from Vitna's own index, with optional `domains`, `excludeDomains`, `maxAgeDays`, `asOf` snapshot cutoff, `ranking`, `effort`, inline `contents`, and one-call live `scrape` | `POST /v1/search` | | `google_search` | Google's results page for a query, in SerpApi's field names unless `shape` asks for Vitna's own | `POST /v1/serp` on the search service | The source tools are listed only while sources are enabled, `search_index` and `discover_grid` only where a search engine is configured, and `google_search` only where the operator names a search service in `VITNA_WEB_SERP_URL`. `read_page`, `batch_scrape`, `list_open_roles` and the crawl tools ask for less than their routes do by default, because a whole page, job board or crawl can be more than a model should take in one result. A crawl tool's job deadline is the exception: it defaults to five minutes, where the route's defaults to one page's 30 seconds. `extract_data` does not offer the route's background mode; it waits for the result within the request deadline. Every tool is read-only except `crawl_site` and `cancel_crawl`, which start and stop a background crawl. A tool call is the REST request it stands for, sent with the caller's key, so it counts against the same request and page limits, is metered the same way (`find_sources` and `describe_source` are free, and a `run_source` call is one `sourceRequests` once it returns data), and appears in `/v1/usage` under the route's own operation. `google_search` is the exception, because its route is not this service's. The call goes to the search service's `POST /v1/serp` with the caller's own org key, `vitna_key_...` (see [one key](../one-key.md)), so the search service admits it, counts it against the org's search limits, applies its deadline, cache and `fresh` captures, and logs it, exactly as a direct request. It does not count against this service's request limit or appear in `/v1/usage`. A `vitna_web_` key covers the Web APIs only and is not sent on: the tool answers `platform_key_required`. Its arguments are SerpApi's, mapped onto the route's fields: | Argument | Route field | Allowed values | |---|---|---| | `q` | `query` | Required; 1 to 1,000 characters | | `gl` | `country` | Two-letter country code, such as `us` | | `hl` | `language` | Language tag, such as `en` or `pt-BR` | | `start` | `offset` | 0 to 190; 0 unless given | | `num` | `maxResults` | 1 to 10; 10 unless given | | `extras` | `extras` | `true` adds People Also Ask questions, related searches and ads | | `no_cache` | `fresh` | `true` captures a new page rather than reuse the key's recent answer | | `device` | None | `desktop` only. Vitna captures Google's desktop results page and has no mobile or tablet capture, so the tool refuses either as `invalid_arguments` | | `shape` | `shape` | `serpapi` unless given, or `vitna` for the route's default fields | The result is the route's answer as `structuredContent`, in the shape described under [SerpApi-shaped response](../search/serp.md#serpapi-shaped-response). When Google does not answer and the search service hands the search to Bing, the result says so: `meta.engine` is `bing`, and in SerpApi's shape `search_metadata` gives `bing_url` instead of `google_url`. A search can take up to the search service's deadline, about a minute on the hosted service, so give the client's tool-call timeout at least that. Results carry the route's response as `structuredContent` and as JSON text. When a route refuses a call, the tool result has `isError: true` and its text gives the route's code and message, such as `invalid_options: options.title must be at most 50.` or, for a rate limit, the number of seconds to wait. Arguments that break a tool's input schema are refused the same way, as `invalid_arguments`. An unknown tool or a malformed message is a JSON-RPC protocol error. ### Protocol versions The endpoint speaks protocol `2026-07-28` and the earlier `2025-11-25`, `2025-06-18` and `2025-03-26`, and keeps no session for any of them: - A `2026-07-28` request carries `io.modelcontextprotocol/protocolVersion` and `io.modelcontextprotocol/clientCapabilities` in `params._meta`, and mirrors its version, method and tool name into the `MCP-Protocol-Version`, `Mcp-Method` and `Mcp-Name` headers. A header that is missing or does not match the body is refused with `-32020` (400), missing metadata with `-32602` (400), and an unknown method with `-32601` (404). `server/discover` reports the supported versions and the server's identity. - An earlier client opens with `initialize` and is answered with the version it asked for, or `2025-11-25` when it asked for one this server does not speak. Later requests name the version in `MCP-Protocol-Version`; a request without the header is read as `2025-03-26`. No `Mcp-Session-Id` is issued, and one sent by the client is ignored. - A version the server does not speak is refused with `-32022` (400), whose `data.supported` lists the versions it does. Each POST carries one JSON-RPC message; batches are refused. Responses are always a single JSON object, never an SSE stream, so `GET` and `DELETE` answer `405`. A request carrying an `Origin` header is refused with `403` unless the operator lists that origin in `VITNA_WEB_MCP_ORIGINS` (comma-separated), so only the browser origins the operator names can call it from a web page. ## Capabilities and usage ```text GET /v1/status GET /v1/usage ``` Use the same bearer key to inspect the configured capabilities and limits, or your client usage in the current UTC day. The status response indicates which engines and search discovery are configured, and `capabilities.documents` lists the document kinds read (`limits.documentBytes` and `limits.documentCharacters` bound them). Inspect `engines`, `searchIndex`, and Research `generation` metadata to distinguish native extractive behavior from a selected self-hosted model. The usage response includes request and page-attempt counts, limits, and recent request outcomes. Resource counters include browser milliseconds, render attempts, Search calls, answer/extraction-engine calls, the bytes moved through residential and ISP routes (`residentialBytes`, `ispBytes`), and the bytes an account's own proxy keys moved (`proxyBytes`), and the firehose records returned (`firehoseRecords`). `searchRequests` and `modelRequests` also count native in-process invocations; these names do not imply an external network request. Static client keys have no usage statement. API accounts can retain a metered usage ledger; payment collection is disabled. ### Operator cost estimates Optional rates let an operator estimate measured resource usage: ```dotenv VITNA_WEB_BROWSER_SECOND_USD=0.001 VITNA_WEB_SEARCH_REQUEST_USD=0.005 VITNA_WEB_MODEL_REQUEST_USD=0.01 VITNA_WEB_RESIDENTIAL_GB_USD=5 VITNA_WEB_ISP_GB_USD=2 VITNA_WEB_PROXY_GB_USD=3 VITNA_WEB_SOURCE_REQUEST_USD=0.002 VITNA_WEB_FIREHOSE_RECORD_USD=0.0002 ``` These rates are illustrative configuration values, not Vitna prices. Each accepts a nonnegative USD rate per measured browser second, dispatched Search/model-engine invocation, source call that returned data, firehose record returned, or decimal GB (10^9 bytes) moved through a residential or ISP route or on an account's proxy keys. `/v1/usage` returns `resources` and a `cost` object with `estimatedUsd`, `rates`, `basis: "operator_rates"`, and `complete`. When all rates are unconfigured, `estimatedUsd` is `null`. With only some rates configured, the estimate covers those resources and `complete` is false when an observed resource has no rate. It does not imply that the unpriced resources cost zero. The estimate covers only the configured rate model. It is not a payment charge, a provider invoice, or a measurement of hosting and proxy bandwidth costs. Use provider billing records to reconcile external spend. ### Account usage statements Activated accounts retain idempotent resource observations and daily totals in the Vitna SQLite ledger. Browser durations, render attempts, Search calls, answer-engine calls, source calls that returned data, firehose records returned, bytes through residential and ISP routes, and bytes on the account's proxy keys are attributed to the account. Ledger pricing uses separate nonnegative integer microdollar rates, with one USD equal to 1,000,000 microdollars: ```dotenv VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD=1000 VITNA_WEB_BILLING_SEARCH_REQUEST_MICRO_USD=5000 VITNA_WEB_BILLING_MODEL_REQUEST_MICRO_USD=10000 VITNA_WEB_BILLING_RESIDENTIAL_GB_MICRO_USD=5000000 VITNA_WEB_BILLING_ISP_GB_MICRO_USD=2000000 VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000 VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD=2000 VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD=200 ``` These are illustrative operator values. Browser observations round up to a whole millisecond, and each event charge rounds up to a whole microdollar. Route bytes are priced per decimal GB for every attempt that moved them, delivered or not, because the operator's provider bills them that way. The monthly `/v1/invoices/YYYY-MM` owner route returns usage, observed event counts, priced totals, rate completeness, and daily entries. Its `kind` is `usage_statement` and `paymentCollection` is `false`. `pricedMicroUsd` covers resources with a configured rate; `totalMicroUsd` stays `null` when an observed resource has no rate or accounting is incomplete. A current month remains `accruing`. Check `complete` and `billingStatus` before treating a statement as settled. Failed or pending accounting can mark the statement incomplete even when an API operation returned successfully. The service retains pending observations for recovery; it does not silently report a complete bill for unrecorded usage. Ledger records have a bounded 400-day retention window; statements outside a fully retained month return `invoice_unavailable`. These statements do not charge a card, debit a prepaid balance, reconcile hosting bills, or deliver customer payment collection. ## Native engines and optional endpoints The default backends are in process: ```dotenv VITNA_WEB_SEARCH_BACKEND=native VITNA_WEB_MODEL_BACKEND=native VITNA_WEB_SEARCH_RERANK=true ``` Native retrieval uses Vitna's SQLite FTS5 index with title-weighted BM25 ranking and an optional local lexical reranking pass. Its file defaults to `search.sqlite` under `VITNA_WEB_DATA`; override it with `VITNA_WEB_SEARCH_DB`. Starting the service opens that index but does not crawl the web or add example documents. Empty or unrelated indexes can return no matches. Populate selected public HTTPS sources explicitly: ```bash node --env-file=.env.web platform/web/native-search-crawl.mjs --seed https://example.com/ --max-pages 50 --max-requests 100 ``` Repeat `--seed` for up to 100 seed URLs, or use `--catalog file.json` with the existing Vitna Search source catalogue format. Seed URLs cannot contain query strings. Crawls are bounded by `--max-requests`, `--duration-ms`, and `--timeout-ms`, respect the source scope, and add or refresh selected sources without removing other sites from the index. This crawler fetches public pages; it does not execute JavaScript. Use browser APIs for rendered pages. With configuration already in the environment, the equivalent script is available as: ```bash npm run web:index -- --seed https://example.com/ ``` To refresh an explicit source list in the service, configure exactly one of `VITNA_WEB_SEARCH_SEEDS` (a JSON array of public HTTPS seed URLs) or `VITNA_WEB_SEARCH_CATALOG` (a local Search catalog file): ```dotenv VITNA_WEB_SEARCH_SEEDS=["https://example.com/docs/"] VITNA_WEB_SEARCH_REFRESH=true VITNA_WEB_SEARCH_REFRESH_MS=300000 VITNA_WEB_SEARCH_REFRESH_REQUESTS=30 VITNA_WEB_SEARCH_REFRESH_DURATION_MS=120000 ``` Configured seeds or a catalog enable maintenance unless refresh is explicitly `false`. With neither supplied, startup makes no refresh requests. Due sources are selected by their oldest live page and catalog `refreshHours`, with bounded requests, duration, source batches, robots rules, conditional fetches, and the existing crawl lease. The default batch has three sources and seed sources have twenty pages each. `searchIndex.maintenance` exposes safe run summaries. Query text cannot enlarge this operator-defined source scope. Native Web search returns at most 20 candidates per query. It supports `site:hostname` and `-site:hostname` filters. Retrieval and reranking remain lexical; they do not establish semantic understanding or a global web index. Native Search applies bounded evidence admission, with `relevanceMode` set to `balanced` by default or explicitly to `strict`. Balanced mode can retain useful partial wording but may miss unsupported constraints; strict mode requires all normalized content terms and can lose paraphrases. Native Research and schema extraction use no external model endpoint. They support observed facts, finite comparison rules, and compatible numeric calculations, and refuse missing or ambiguous evidence. They do not provide general language-model reasoning. For a separate Vitna Search process, select the service backend explicitly: ```dotenv VITNA_WEB_SEARCH_BACKEND=service VITNA_WEB_SEARCH_URL=http://127.0.0.1:8789 VITNA_WEB_SEARCH_KEY=YOUR_SEARCH_CLIENT_KEY VITNA_WEB_SEARCH_MODE=index VITNA_WEB_SEARCH_RERANK=true ``` Web discovery uses indexed search. The separate Search service retains its own SERP contract, but the Web engine does not select a paid provider or SERP fallback. The service adapter returns up to ten results per query; larger source budgets use multiple queries or supplied URLs. The Search service applies its own default retrieval: hybrid where its [semantic retrieval](../search/overview.md#semantic-retrieval) is configured, keyword otherwise. The MCP tool `google_search` is offered only when the operator names a Search service whose SERP route it calls, whichever backend serves index search: ```dotenv VITNA_WEB_SERP_URL=http://127.0.0.1:8789 ``` Each call carries the caller's own org key there, never `VITNA_WEB_SEARCH_KEY`, so the URL must be https, or http on a loopback address. That Search service must have SERP retrieval enabled and accept org keys, which takes the same `VITNA_PLATFORM_VERIFY_URL` and `VITNA_PLATFORM_SERVICE_TOKEN` there as here. For language-based extraction or synthesized research answers, select an operator-owned compatible model runtime. `MODEL_URL` is its full chat completions endpoint; model name and optional private key are separate: ```dotenv VITNA_WEB_MODEL_BACKEND=self-hosted VITNA_WEB_MODEL_URL=http://127.0.0.1:8000/v1/chat/completions VITNA_WEB_MODEL_KEY=YOUR_RUNTIME_KEY VITNA_WEB_MODEL=YOUR_MODEL_NAME ``` The server requests JSON output, validates extraction against the schema, and checks that research answers reference the pages supplied to the model. Runtime credentials stay on the server. Setting endpoint variables requires the corresponding explicit backend; native mode does not silently call them. Each backend also accepts `disabled` to turn off that engine. ## Deployment boundaries The suite is an operator-configured service. Keep its client hash map, invitation hashes, and optional model/Search credentials in the service environment. Expose it through a TLS reverse proxy when remote access is needed. Website relay configuration belongs in the website environment. The data directory configured with `VITNA_WEB_DATA` stores crawl jobs, monitors, quotas, and recent usage. Keep that directory across process restarts and run one service writer per directory. State is stored in `state.sqlite` using SQLite WAL transactions. A legacy `state.json` is imported once into a new database and retained as a recovery backup. Stop the service before copying the complete data directory for a filesystem backup, so the database and its journal files are captured consistently. The default request limits are 30 requests per minute, 1,000 requests per day, and 2,000 page reads per day, with two concurrent operations. Configure them using `VITNA_WEB_RPM`, `VITNA_WEB_DAILY_LIMIT`, `VITNA_WEB_DAILY_PAGES`, and `VITNA_WEB_CONCURRENCY`. `VITNA_WEB_TIMEOUT_MS` sets the default browser deadline, initially 30,000 milliseconds. The browser service restricts requests to public HTTPS destinations and checks redirect destinations and resolved addresses. Rendering does not guarantee that a website will be reachable or that every delayed field will be ready. Choose a content selector with `waitFor` where appropriate. See [Unblocker setup](../unblocker/overview.md) for browser installation and proxy configuration, and [Search setup](../search/overview.md) for the separate search service. The deployment package includes `deploy/vitna-web.env.example`, `deploy/vitna-web.service`, `deploy/vitna-web.nginx.conf.example`, and `deploy/vitna-web.Dockerfile`. These prepare a deployment; they do not publish a hosted endpoint. Browser settings use `VITNA_WEB_BROWSER_PATH`, `VITNA_WEB_BROWSER_CHANNEL`, and `VITNA_WEB_BROWSER_PROFILE`. Its `auto` reads in Chromium and moves a challenged read, page or document, to Camoufox, and it and `camoufox` take `VITNA_WEB_CAMOUFOX_PYTHON`, `VITNA_WEB_CAMOUFOX_FINGERPRINTS` and `VITNA_WEB_CAMOUFOX_OS`, with the meanings the [Unblocker's](../unblocker/overview.md#the-auto-profile) settings of those names have; owned browser workers ignore all four. A service whose `HOME` does not last, as a systemd runtime directory does not, also needs `XDG_CACHE_HOME` set to the cache Camoufox's browser was fetched into; it reaches the fingerprint helper and no browser. The hosted Web API reads the hosted Unblocker's install: `deploy/web-api/install-camoufox.sh` gives `/opt/vitna-camoufox` the Web API user's group, starts Camoufox as that user under its unit's confinement, and prints the lines its env file needs (`deploy/web-api/OVH-RUNBOOK.md`, The Web API's auto browser profile). Each request's line in the service's log names the host it read, the engine that served the read (`chromium`, `camoufox` or `cloakbrowser`) and its attempts, with `challenged` when a challenge sent the read on; a request that read more than once, as `/v1/documents` does, also lists each read in `reads` (up to 20, a failed one with its `code`) and counts them in `readCount`. A job's reads are not on the line of the request that started it. vitna.ai keeps the host, engine and attempt count from an org's receipt, as it does from the Unblocker's. `/v1/usage` lists the requests alone. Proxy routes use `VITNA_WEB_PROXIES` or labelled `VITNA_WEB_PROXY_ROUTES`, and an optional exact-host scope uses `VITNA_WEB_ALLOWED_HOSTS`. For labelled routes, configure private operator credentials: ```dotenv VITNA_WEB_PROXY_ROUTES=[{"id":"us-primary","country":"US","proxy":"https://user:password@proxy.example:8443"}] ``` The operator supplies and labels these routes. Country selection does not provision a proxy network or verify the provider location. Residential and ISP accounts from a provider the operator holds take the same list, with credential templates; see [proxy routes](../unblocker/overview.md#proxy-routes). Keep proxy and integration credentials in private service configuration. The supplied systemd unit and Docker image run one process with two browser slots by default. Configure queue, page, request, and job limits for your machine, and verify representative workloads before increasing them. SQLite persistence and larger budgets do not by themselves establish coordinator high availability or a hosted service. Owned browser workers can distribute rendering, as described under [owned browser workers](#owned-browser-workers). The selected Vitna Search backend reads its populated index. The web suite does not provision a global web index. Owned CONNECT gateways can supply egress routes; see [owned egress setup](../unblocker/overview.md#run-an-owned-egress-gateway). ## Operators: running the service This section is for the operator of a Web API service and needs a checkout of Vitna's private repository. Nothing in it is needed to use the hosted pilot at `https://web.vitna.ai`, and it is not an offer of a self-hosted deployment. Until 2026-10-11 it opened this page as "Run locally", and the invitation, signup, recovery, quota, worker, metrics and benchmark material joined it from the customer half of this page the same day. The web suite uses the browser runtime installed for Vitna Unblocker. It has its own client keys and service process. ```bash npm run web:install npm run web:browser npm run web:demo ``` The demo uses a real browser against local illustrative pages. It does not contact a target website or a paid model. To start your own service, mint a client key: ```bash npm run web:key -- pilot ``` Keep the printed token private. Put the printed client hash, rather than the token, into an ignored `.env.web` file: ```dotenv VITNA_WEB_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"} VITNA_WEB_PORT=8790 VITNA_WEB_DATA=.web-data ``` Start the service with that configuration: ```bash node --env-file=.env.web platform/web/start.mjs ``` When the settings are already in the environment, use: ```bash npm run web:start ``` The default service address is `http://127.0.0.1:8790`. Browser operations use the installed Chromium runtime. Native engines make no model or search provider call at startup. Invite activation and owner key management are available through the account console when the website relay is configured. Customer payment collection and open self-service registration remain unavailable. ### Issuing invitations Static keys from `web:key` remain supported. For controlled account activation, create an operator invitation: ```bash npm run web:invite -- --client customer-01 --display-name "Customer application" ``` The command prints a one-time `activationToken` and an `invites` array. It does not contact the service or activate an account. Put only the returned hashed `invites` array in `VITNA_WEB_INVITES`, then start the service. Keep the activation token private. An account client identifier must differ from every static client identifier in `VITNA_WEB_CLIENTS`. The CLI defaults to a 24-hour invitation, a 90-day API-key expiry, and a 366-day owner-token expiry, measured from invitation creation. Options `--expires-hours`, `--key-days`, and `--owner-days` can shorten those windows; the maximum invitation lifetime is 720 hours. ### Optional public signup Public signup is disabled by default. Enable it explicitly with `VITNA_WEB_SIGNUP_ENABLED=true` and a private, stable 32-byte hexadecimal `VITNA_WEB_SIGNUP_SECRET`. `GET /v1/accounts/signup` reports safe availability metadata. The account console at `/apis/account` offers signup when enabled. Request a short-lived challenge with `POST /v1/accounts/signup/challenge` and an empty JSON object. The response specifies a SHA-256 proof over the UTF-8 bytes of `challenge + ":" + nonce`, with 18 leading zero bits and a bounded integer nonce. Submit the challenge, nonce and optional label to `POST /v1/accounts/signup`. This proof adds computational friction; it does not verify a human, email address or identity. No email or payment service is involved. Account creation, signup counters and a hash-only replay receipt commit in one SQLite transaction. The successful response returns the API key and owner token once. Repeating the same successful signup returns public account metadata with `duplicate:true` and `credentialsUnavailable:true`; it cannot recover the credentials. Keep the original response privately. Default issuance limits are three challenges per minute and 20 per UTC day per network bucket. Signup allows three accounts per network bucket per UTC day, 100 per coordinator per day and at most 1,000 retained accounts. IPv4 /24 and IPv6 /64 buckets are HMAC-hashed before storage. Failed proof attempts, retained network counters and replay receipts also have separate bounds. Explicit positive integer envs can lower or adjust coordinator capacity: `VITNA_WEB_SIGNUP_SIGNUPS_PER_DAY` (1 to 100,000) and `VITNA_WEB_SIGNUP_MAX_ACCOUNTS` (1 to 1,000). The OVH pilot ran with 20 signups per UTC day and at most 100 accounts until the owner closed signup on 2026-10-11. These limits apply across network buckets and count existing native accounts; the account cap does not reset at midnight. Omitted variables preserve the defaults. Configuring capacities while signup is disabled, or using noncanonical/out-of-range integers, rejects startup. Set them together with explicit enablement. The server uses its transport peer address. Headers supplied by a caller cannot change that address. Behind a reverse proxy, configure exact peer IPs in `VITNA_WEB_TRUSTED_CLIENT_PROXIES` and have that proxy overwrite `X-Vitna-Client-IP` with its observed client address. Missing or malformed identity from a trusted proxy fails closed. For website relays, also configure exact normalized egress IPs in `VITNA_WEB_TRUSTED_CLIENT_RELAYS` and the dedicated token SHA-256 in `VITNA_WEB_CLIENT_RELAY_TOKEN_HASH`. The relay sends `X-Vitna-Relay-Client-IP` plus `X-Vitna-Relay-Token`; the API checks its trusted socket, nginx-observed relay address and token before using the visitor IP. A configured relay without an authenticated assertion fails closed. The Next relay additionally requires a distinct nginx ingress token authenticated by `VITNA_WEB_RELAY_INGRESS_TOKEN_HASH`, and its private outbound `VITNA_WEB_RELAY_CLIENT_TOKEN`. It requires HTTPS beyond literal loopback. Nginx must overwrite ingress identity and establish trusted Cloudflare peers before reading Cloudflare identity. Caller `X-Forwarded-For` or `CF-Connecting-IP` headers alone are never trusted. ### Offline operator recovery If the owner token is lost or expired, or the activation response was lost after the invitation was redeemed, recover access through the operator's local account CLI. Stop the Web service first. The CLI opens the same single-writer SQLite data directory and refuses its live-writer lock; do not bypass that lock. Rotate the owner token by client identity: ```bash npm run web:account -- --data .web-data owner:rotate --client customer-01 ``` The result contains a new `ownerToken` and `ownerExpiresAt`. Save the token privately, then restart the service so its credential caches reflect the change. The old token is invalidated. This recovery requires filesystem access to the data directory; it does not require the lost owner token or an HTTP admin credential. If `--data` is omitted, the CLI uses `VITNA_WEB_DATA`, then `.web-data`. Other offline commands inspect account/key metadata or replace and revoke an API key: ```bash npm run web:account -- --data .web-data accounts:list npm run web:account -- --data .web-data keys:list --client customer-01 npm run web:account -- --data .web-data key:rotate --client customer-01 --key REPLACE_WITH_KEY_UUID npm run web:account -- --data .web-data key:revoke --client customer-01 --key REPLACE_WITH_KEY_UUID ``` Lists contain public metadata and hide credential hashes. `key:rotate` returns the replacement `apiKey` and key metadata once. Stop the service before these commands and restart it after mutations, preserving the single-writer boundary and refreshing cached credentials. ### Shared quota admission The owned quota authority coordinates admission across Web and Unblocker API processes over authenticated HTTP. Mint a separate operator credential and start a single authority with explicit authoritative service limits: ```bash npm run quota:key npm run quota:start ``` The authority requires `VITNA_QUOTA_TOKEN_HASH` and `VITNA_QUOTA_SERVICES`; its private durable directory defaults to `.quota-data`, port to 8793 and host to loopback. `deploy/vitna-quota.env.example` documents service and per-client limit configuration. Non-loopback listening requires configured TLS certificate and key files. API processes explicitly select `VITNA_WEB_QUOTA_BACKEND=shared`, `VITNA_WEB_QUOTA_URL` and `VITNA_WEB_QUOTA_TOKEN`; standalone Unblocker uses the corresponding `VITNA_UNBLOCKER_QUOTA_*` variables. Partial configurations are rejected. Authority limits override local quota settings. Each admitted API request and each browser attempt reserves its own atomic, idempotent operation. Retries consume page-attempt quota. Status/usage reads are free, scoped by service/client, and report the authority's limits and counters. Web resource timings and recent-operation lists remain scoped to their coordinator. Unreachable authorities fail closed before browser or search dispatch; uncertain admissions are never automatically replayed or failed over. Deduplication IDs contain bounded timestamps and are retained for 24 hours. The same ID with a changed admission conflicts; an expired ID cannot become a new charge after pruning. A denied operation stays denied on replay even after quota rollover. New logical work needs a new ID. Backwards clocks cannot reset consumed windows. Counters/events, queues and HTTP bodies are bounded, and quota changes plus deduplication commit in one SQLite transaction. The authority requires a dedicated private directory and rejects linked storage, reused account state and foreign database schemas before opening them for writes. Existing POSIX ownership/modes must be private; Windows operators must configure private ACLs. Run one writer per authority directory. This adds shared admission, not full coordinator or authority high availability. Account credentials, usage statements, jobs and monitors still belong to one API coordinator; separate API directories do not replicate that state. Shared quotas do not make public signup accounts portable across coordinators. Session workers also retain their own writer directories and stable routing. ### Owned browser workers The default `VITNA_WEB_BROWSER_BACKEND=native` executes Chromium in the Web process. To separate browser execution onto machines you operate, run `npm run web:worker` on each worker and explicitly select the worker backend: ```dotenv VITNA_WEB_BROWSER_BACKEND=workers VITNA_WEB_BROWSER_WORKERS=[{"id":"worker-a","origin":"https://worker-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","concurrency":2}] ``` Workers require `VITNA_WEB_BROWSER_WORKER_TOKEN_HASH`. Generate its SHA-256 hash with `npm run web:worker:hash`, reading the token from private `VITNA_WEB_BROWSER_WORKER_TOKEN` configuration. Use a distinct operator token, not a customer API key. Nonlocal worker binds require the worker TLS key and certificate files. Loopback HTTP is supported for local development. ```bash npm run web:worker:hash npm run web:worker ``` `deploy/vitna-browser-worker.env.example`, the systemd unit, and Dockerfile provide the full configuration and deployment templates. The pool supports up to twenty fixed workers, bounded queues, and explicit per-worker concurrency. Tenant/session hashing keeps persistent sessions on one worker while the configured worker list and order remain stable. Session cookies and localStorage stay in that worker's RAM and expire or disappear on worker restart. Clearing a session uses a separate bounded control queue. The pool does not replay a failed browser invocation on another worker. Without that replay, a read through workers does not step up a network after a block; name `network` to use a dearer tier. Workers report the network and transferred bytes of each attempt, delivered or not, so the coordinator prices residential and ISP traffic as it would its own. Status reports `engines.browser` and whether execution is local or distributed. The API coordinator and SQLite directory still require one writer; this provides browser distribution without coordinator high availability. The same worker process can also hold Browser API sessions: set `VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS` on it and list it in the Web process's `VITNA_WEB_BROWSER_API_WORKERS`. See [sessions on owned workers](#sessions-on-owned-workers). ### Operator metrics Generate a separate operator token with `npm run web:metrics:key`. Retain the token privately and configure only its SHA-256 hash as `VITNA_WEB_METRICS_TOKEN_HASH`. Authenticated `GET /metrics` returns Prometheus text; use the token as a bearer credential. This route rejects queries and bodies and is disabled when no hash is configured. Customer keys cannot scrape it. ```bash npm run web:metrics:key ``` Counters and histograms cover HTTP outcomes and full response transfer, current requests, bounded queues, browser duration, engine invocations, resource bytes, and ledger pricing observations. Labels use fixed operation, method, status, and service values; URLs and customer identifiers are excluded. Counters reset on process restart. `priced_micro_usd_total` includes known partial prices; `unpriced_events_total` tracks incomplete pricing instead of treating it as zero. These are process observations, not a replacement for the durable account ledger. `deploy/vitna-api-alerts.yml` supplies starter availability, error-rate, latency, queue, and metric-cardinality rules. Load it into your Prometheus configuration and configure your own alert delivery. The file creates no monitoring service or notification destination. ### Benchmark locally Run a repeatable baseline after installing Chromium: ```bash npm run web:benchmark -- --iterations 10 --concurrency 2 ``` The runner measures every Web API product, both screenshot formats, and the existing JS Unblocker through real local HTTP requests and real Chromium. Each operation uses a fresh durable store. Pages and assets are fulfilled locally, with delayed JavaScript, article content, product variants, and company data. No external model, search provider, webhook, or proxy is used. For a separate public-site coverage sample, run: ```bash npm run web:benchmark -- --only coverage --report outputs/web-benchmark/public-coverage.json ``` This makes two sequential Reader requests each to Python, React, MDN, GitHub, Hacker News, and an arXiv abstract, using automatic content readiness. Each response must contain a specified content marker, a minimum content length, and no truncation. It is a small diagnostic across six public sites, not a representative sample of web coverage. The JSON report and readable HTML table are saved under `outputs/web-benchmark/`. Use `--report path/to/report.json` to choose another location. One warmup per operation is excluded. The measured sample count accepts 1 to 50 and client concurrency accepts 1 to 4; the service keeps its default two browser slots. Latency includes browser startup, cleanup, durable writes, and full JSON transfer and decoding. Successful latency and failure counts are separate, and success requires verified output rather than only an HTTP 200 response. Crawl latency includes completion of three pages and polling every 100 ms. Monitor measures unchanged selector checks against independent monitors; baseline creation and deletion are excluded. Research measures collection of one supplied source, without search or generation. A ten-request sample is an initial baseline: its nearest-rank p95 equals its slowest observation. To include two sequential reader requests to each of three public training pages (static quotes, JavaScript quotes, and a book catalog), add `--live`: ```bash npm run web:benchmark -- --iterations 10 --concurrency 2 --live ``` To repeat selected measurements, use `--only reader,crawl` with operation IDs from the report, or `--only public` to run only the six public samples. The report also measures synchronous storage-write time so slow jobs can be distinguished from browser and network delays. Monitor setup is excluded, and render-attempt concurrency includes startup and cleanup. Public samples remain separate from the controlled results. This benchmark does not establish representative website coverage, competitor performance, hosted capacity, or external model and search-provider latency. `npm run web:load` exercises actual local HTTP servers with a controlled five-millisecond fixture renderer, bounded concurrency, output validation, overload rejection, and a quota restart check. Its saved latency and throughput measure the API transport and admission path, not Chromium or production website capacity. `npm run search:relevance` separately evaluates authored relevance and abstention fixtures. Keep these scopes separate when comparing performance. ```bash npm run web:load npm run search:relevance ``` --- <!-- https://vitna.ai/docs/unblocker/overview.md --> # Vitna Unblocker Hosted alpha. Vitna Unblocker is a browser-backed web access API: send a public HTTPS URL and receive the rendered HTML, the readable text and the links, or a validated JSON feed, with a receipt of every attempt. It is hosted at https://unblock.vitna.ai, takes an alpha key issued on request, and is relayed at https://vitna.ai/api/v1/unblock. Billing is not active. The product page is [vitna.ai/unblocker](https://vitna.ai/unblocker). An alpha key starts `vitna_unblock_` and is issued once a person has checked the fit of one workload: request it at [vitna.ai/unblocker#alpha-access](https://vitna.ai/unblocker#alpha-access). An org's platform key (`vitna_key_`) works too, at the hosted endpoint and through the relay; see [One key for the hosted APIs](../one-key.md). The prices on the product page are proposed and nothing can be bought yet. The console's Plan page counts each org's delivered pages per UTC month, from the receipts of requests made with its platform key, in the unit those prices name. Managed proxy capacity and image CAPTCHA solving are not offered. ## Hosted access Send each request to the hosted endpoint with the key as a bearer token: ```bash curl https://unblock.vitna.ai/v1/unblock \ -H "Authorization: Bearer $VITNA_UNBLOCKER_KEY" \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/","render":true,"output":"json","timeoutMs":30000}' ``` The website relays the same request, with the same key, at `https://vitna.ai/api/v1/unblock`. The relay forwards the key unchanged, refuses a body over 8 KiB, and does no browser work itself; the product page's live playground sends through it. Signed in to the console, [vitna.ai/console/unblocker](https://vitna.ai/console/unblocker) runs the same request with the org's platform key: it checks each field against the service's own rules before sending, shows the page and every attempt it took, prints the request as curl, JavaScript and Python, and links the run to its record in Activity. Authenticated `GET /v1/status` reports the limits that apply to your key, what it has used of them, and that billing is not enabled. It renders nothing and consumes no allowance: ```bash curl https://unblock.vitna.ai/v1/status \ -H "Authorization: Bearer $VITNA_UNBLOCKER_KEY" ``` Each key has a per-minute and a daily allowance, counted across the hosted service's processes. A request past either is refused with `429` and `rate_limited` or `daily_limit`, and a `Retry-After` header gives the seconds until the allowance resets. Refused pages count against the allowance as delivered ones do. When every browser slot is in use the answer is `429` with `service_busy`, also with `Retry-After`. The hosted service runs the `native` browser profile; see [Browser profiles and live checks](#browser-profiles-and-live-checks). Keep the key on your server, never in a page a browser loads or in a repository. The service stores only its SHA-256 hash, so a lost key cannot be read back: ask for a new one through the contact form at [vitna.ai/start#contact](https://vitna.ai/start#contact), which is also where to ask for help. ## Request The service serves this endpoint, hosted at `https://unblock.vitna.ai`: ```text POST /v1/unblock ``` The website relays to it. It forwards the caller's key; it has no shared service key and performs no browser work itself: ```text POST /api/v1/unblock ``` The relay is an operator setting, `VITNA_UNBLOCKER_SERVICE_URL`; without it the website endpoint returns 503 with `service_not_configured`. The product playground defaults to an explicitly illustrative example; its Live API mode uses this relay. ```bash curl https://unblock.vitna.ai/v1/unblock \ -H "Authorization: Bearer $VITNA_UNBLOCKER_KEY" \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/","render":true,"output":"json","timeoutMs":30000}' ``` | Field | Default | Meaning | |---|---|---| | `url` | Required | Public HTTPS URL, at most 4,096 characters; query strings are supported | | `render` | `true` | Execute JavaScript; false opens the page with JavaScript disabled | | `output` | `json` | `json` returns HTML, text, and links; `html` or `text` returns that string | | `waitFor` | Omitted | CSS selector to wait for until visible; requires JavaScript rendering | | `timeoutMs` | 30,000 | Total deadline across attempts, capped by the deployment; 1,000 to 60,000 | | `retries` | 1 | Additional attempts, from 0 to 2 | | `country` | Omitted | Two uppercase letters selecting a configured proxy route that serves that country | | `network` | `auto` | `auto` starts on the cheapest configured route and steps up a tier after a block; `datacenter`, `isp`, or `residential` uses only routes of that kind | | `session` | Omitted | 1 to 64 letters, numbers, underscores, or hyphens; sticky proxy selection and identifier for optional persistence | | `persistSession` | `false` | Boolean; restores and saves cookies and localStorage for the authenticated client and `session`; requires a session identifier | | `actions` | Omitted | Up to 50 ordered scroll, wait, click, write or press objects; requires `render: true`; uses the bounds below | | `language` | Browser default | BCP 47 locale, at most 35 characters, such as `en-US` | | `viewport` | Desktop default | `{ "width": 1440, "height": 960 }`; width 320 to 4,096, height 240 to 4,096 | | `headers` | Omitted | Only `accept-language` up to 200 ASCII characters and `dnt` equal to `0` or `1` | Without a selector, rendering waits for meaningful visible content after DOM readiness and any recognized browser verification. The browser allows an initial 500-millisecond grace period, then requires 200 milliseconds of stable content, within five seconds or the remaining request deadline. A page that still presents an empty or loading state returns `target_incomplete`, which can be retried within the request budget. Set `waitFor` for pages whose useful content appears later. A `waitFor` selector that never becomes visible fails with `selector_timeout` before the deadline: the wait stops up to 1.5 seconds short (a quarter of what remains, on a short budget) so the response names the selector rather than `target_timeout`. A retry waits within what is left of `timeoutMs`, not a fresh budget. HTML is the browser's serialized DOM, including content added by scripts, rather than the original response bytes. Text extraction prefers main/article content and removes scripts, navigation, and hidden attributes. It does not infer a schema or run an LLM. Links are resolved public HTTPS URLs. `meta.delivered` confirms an accepted readable snapshot. It does not establish that every application-specific data field or delayed item is ready. Published text inside ASP.NET form wrappers is retained. Text in input controls, navigation, and hidden attributes remains excluded. A secondary loading widget does not prevent delivery when the page already contains substantial published content; an empty or loading-only content region still fails readiness. Automatic readiness also waits for in-flight, same-origin Q4 GET feeds matching `/feed/*.svc/Get*` before starting the stable-content interval. This remains within the existing five-second readiness deadline and does not wait for general network idleness. Use `waitFor` for slower archives or application-specific completion. For `application/json` and `application/*+json` destinations, `output: "json"` returns `{ "html": "", "text": "original response body", "links": [], "data": ... }`. The envelope's `contentType` identifies the destination MIME type. `output: "text"` returns the JSON text. This reads and validates the decoded navigation response, not Chromium's JSON viewer DOM. Malformed JSON returns `target_invalid_json`; the same two-MiB body limit, HTTP status, challenge, network, and deadline checks apply. An unsuccessful status is judged before the body is parsed, so an error body that is empty or not JSON still returns `target_blocked` or `target_http_error`. A body nested more than 512 levels deep is delivered as `text` without `data`. HTML output, selectors, actions, and captures are unsupported for JSON feeds. An operator integrating `createBrowserRenderer` can set `blockedResourceHosts` to at most 50 lowercase DNS hostnames. Exact hosts and their subdomains are excluded before optional resources consume the 400-request budget. This is an internal factory option, unavailable in public API requests, and defaults to an empty list. Main navigations and recognized verification resources remain allowed. For IR collection, operators can use observed chart and analytics hosts to avoid spending the page budget on unrelated widgets; required issuer scripts and feed hosts must remain available. The standalone service reads the JSON array from `VITNA_UNBLOCKER_BLOCKED_RESOURCE_HOSTS`, defaulting to `[]`. For example, `["tradingview-widget.com", "s.tradingview.com", "s3.tradingview.com"]` excludes the optional chart embeds observed on GROW's IR page. Do not apply that profile to work that requires those charts. To reduce unnecessary waiting, set `waitFor` to a selector that identifies the content you need. This replaces the automatic content-readiness wait and returns as soon as that content is visible after verification. Choose populated content rather than a container that also exists during loading. For the public ScrapingCourse challenge demos, the verified heading is `#challenge-title`: ```json { "url": "https://www.scrapingcourse.com/cloudflare-challenge", "render": true, "output": "text", "waitFor": "#challenge-title" } ``` The selector controls when the snapshot is taken. It does not shorten a destination's verification or deliberate JavaScript delay. ### Browser actions Actions run in order after navigation and recognized verification, before `waitFor` or the automatic content-readiness check. They dismiss a consent banner, press a "load more" button, open collapsed answers, type into a search box, and scroll a feed, so the snapshot holds what those steps reveal. A sequence holds at most 50 actions and shares the request's deadline, less a short reserve for the read that follows it. | Action | Fields | Bounds | |---|---|---| | Scroll by pixels | `{ "type": "scroll", "y": 2000, "x": 0 }` | Integer `y` from -20,000 to 20,000; optional integer `x` from -4,096 to 4,096, default 0 | | Scroll a screen | `{ "type": "scroll", "direction": "down" }` | `up` or `down` by one viewport; with `selector`, scrolls that element by its own height, for a feed in a scrollable box | | Fixed wait | `{ "type": "wait", "milliseconds": 250 }` | Integer milliseconds from 1 to 10,000; total fixed waits at most 30,000 milliseconds | | Selector wait | `{ "type": "wait", "selector": "#loaded" }` | Waits until the element is visible, for at most 10 seconds | | Click | `{ "type": "click", "selector": "#accept" }` | Waits up to 10 seconds for the first match to be visible and able to receive the click; `"all": true` clicks every match, up to 20 | | Write | `{ "type": "write", "text": "Q3 2026", "selector": "#q" }` | Types 1 to 1,000 characters; with `selector` it clicks that field first, otherwise it types into the focused element; a tab or line break types as Tab or Enter | | Press | `{ "type": "press", "key": "Enter" }` | One character, or `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `Insert`, `Home`, `End`, `PageUp`, `PageDown`, an arrow key (`ArrowDown`), `Space` or `F1` to `F12`, after at most three distinct `Control`, `Shift`, `Alt` or `Meta` modifiers, as in `Control+K` | Selectors are CSS, up to 200 characters, with no control characters. Scrolling moves at most 100,000 pixels in all, with a `direction` step counted as 4,096. A wait supplies exactly one of `milliseconds` or `selector`. Unknown fields and action types are refused. Screenshot and scrape steps, which capture the page partway through a sequence, belong to the Web API's [read and scrape](../web/overview.md#browser-controls), which return their captures; this endpoint refuses them. Actions keep every guard navigation has: DNS and address checks, the resource and transfer limits, and the method guard. **Only GET requests leave the browser**, so a click can follow a link, open a tab or submit a search form that uses GET, but a form that POSTs, a sign-in or a purchase fails with `navigation_blocked` and never reaches the destination. The API runs no caller-provided JavaScript. This illustrative request accepts a consent banner, searches, and waits for the results: ```json { "url": "https://example.com/filings", "render": true, "output": "json", "actions": [ { "type": "click", "selector": "#accept-cookies" }, { "type": "write", "selector": "input[name=q]", "text": "Q3 2026" }, { "type": "press", "key": "Enter" }, { "type": "wait", "selector": "#results" } ], "timeoutMs": 30000 } ``` An action that runs out of time returns `action_timeout`; invalid CSS returns `invalid_selector`; an action the page cannot take returns `action_failed`. A failure during the sequence, including a navigation a step caused, names that step, counted from 0: ```json { "error": { "code": "navigation_blocked", "message": "...", "statusCode": null, "action": { "index": 1, "type": "click" } } } ``` Every output travels in a JSON envelope. This example is illustrative: ```json { "url": "https://example.com/", "finalUrl": "https://example.com/", "statusCode": 200, "title": "Example Domain", "output": "text", "content": "Example Domain Illustrative page content.", "meta": { "requestId": "illustrative-request-id", "delivered": true, "rendered": true, "durationMs": 1500, "attempts": [{"attempt": 1, "outcome": "delivered", "route": "direct", "network": "direct", "browser": "chromium"}], "network": {"requests": 1, "blockedRequests": 0, "transferredBytes": 1200, "connectionFailures": 0}, "challenge": {"detected": false, "cleared": false, "waitedMs": 0, "verificationRequests": 0, "interactions": 0} } } ``` The receipt carries measured duration, request count, and encrypted tunnel bytes. Bytes include protocol overhead and both directions. `delivered` is a delivery observation; the alpha has no billing implementation. ## Refusals and retries Recognized challenge pages receive up to 12 seconds per attempt for the page's JavaScript verification to complete. Detection uses Cloudflare's `cf-mitigated: challenge` response header or specific challenge titles and markup. The service keeps the same session while verification runs, then checks the latest main-page response and rendered content. A visible Cloudflare verification checkbox can receive at most two interactions during the wait. The service does not interact with arbitrary page forms or solve image puzzles. `render: false` refuses a challenge without waiting. Ordinary 403 and 429 responses fail delivery checks and may be retried within the configured attempt budget. A challenge that remains after the wait returns `target_challenge`. `meta.challenge` on delivered responses reports whether a challenge was detected and cleared, the measured wait, and the number of verification requests and widget interactions. Challenge attempts include these fields in their attempt entries, including failed attempts. A successful verification is counted as delivered only after the final HTTP status and content pass delivery checks. Transient network/proxy failures, incomplete content, and target 5xx responses are retryable. Each attempt uses a new browser. Configured routes rotate unless `session` selects a sticky route. A block or challenge moves the next attempt up a network tier when the operator configured dearer routes; see [Proxy routes](#proxy-routes). Under the [auto profile](#the-auto-profile) a challenge also moves the next attempt from Chromium to Camoufox. Each attempt entry names its `browser`. All attempts share one deadline. | HTTP | Codes | |---|---| | 400 | `invalid_request`, `invalid_browser_options`, `country_unavailable`, `network_unavailable`, `destination_blocked` before admission, `session_owner_required` | | 401 | `unauthorized` | | 404 | `not_found` for an unknown path | | 405 | `method_not_allowed` | | 413 | `invalid_request` for an oversized request body | | 415 | `unsupported_media_type` | | 429 | `rate_limited`, `daily_limit`, `page_limit` (shared quota page attempts), `service_busy`, `session_busy`, `session_capacity` | | 499 | `request_cancelled` | | 500 | `internal_error` when the service cannot prepare a response | | 502 | `target_blocked`, `target_challenge`, `target_http_error`, `target_unreachable`, `target_incomplete`, `proxy_unavailable`, `destination_blocked`, `navigation_blocked`, `action_failed`, `selector_timeout`, `invalid_selector`, `request_limit`, `transfer_limit`, `page_too_large`, `target_empty`, `target_content_type`, `target_invalid_json`, `session_state_too_large`, `session_cleared` | | 503 | `browser_unavailable`, `sessions_unavailable`, `session_cleanup_unavailable`, `session_store_unavailable`, `session_decryption_failed`, `session_store_invalid`, `quota_unavailable`, `service_unavailable` while shutting down; website relay: `service_not_configured`, `service_unavailable` | | 504 | `target_timeout`, `action_timeout`, `session_cleanup_timeout`, `upload_timeout` for a request body not received in time | API failure responses include the request ID and attempt outcomes after admission. A challenge is never intentionally returned as a delivered page. Header detection is explicit; title and markup detection is heuristic. The alpha cannot promise detection of every anti-bot system, image CAPTCHA solving, or success on every site. Outcomes depend on the browser runtime and route. Cloudflare does not support automated browsers for solving production challenges; passing a demonstration target is not a general success guarantee. ## Sessions and network scope Every attempt has a fresh browser and loopback CONNECT proxy. `session` keeps proxy selection stable while the operator's route list stays unchanged. `persistSession` defaults to false, so a session identifier alone leaves each attempt's cookies and storage fresh. Explicit `persistSession: true` restores and saves cookies and localStorage for the authenticated client and session identifier after successful delivery and browser/network cleanup. Keys mapped to the same client identifier share that client's session namespace. State is RAM-only: 15-minute sliding TTL after each successful save, 100 sessions total, 20 per client, and 256 KiB of serialized state per session in the standalone startup. Expired state is discarded when the session store is next accessed. Restart loses this state. No idle browser is retained. sessionStorage, IndexedDB, the live DOM, open connections, and page JavaScript state are not restored. Failed or cancelled attempts preserve the previous saved snapshot. An oversized replacement returns `session_state_too_large`. Concurrent same-session persistence returns `session_busy`; exhausted total or per-client capacity returns `session_capacity`. To reuse a page's own cookies and localStorage, send the same authenticated client, session identifier, and explicit persistence option on each request: ```json { "url": "https://example.com/catalog", "render": true, "output": "json", "session": "catalog-01", "persistSession": true, "waitFor": "main" } ``` `DELETE /v1/sessions/SESSION_ID` authenticates with the API key and clears only the caller's saved session. It accepts no query parameters or request body and consumes admission quota. Clearing an active persistent session cancels its attempt and waits for browser cleanup before returning: ```bash curl -X DELETE https://unblock.vitna.ai/v1/sessions/catalog-01 \ -H "Authorization: Bearer $VITNA_UNBLOCKER_KEY" ``` ```json { "session": "catalog-01", "cleared": true } ``` An absent or expired session returns `cleared: false`. The cancelled render cannot save late state. If a clear request is cancelled before dispatch, it does not change the session. If it is cancelled while waiting for active cleanup, the cleanup still settles and the previous committed snapshot is retained when it has not expired; retry the clear request to remove it. Callers can set the two bounded headers listed above. Credentials, cookies, proxy URLs, arbitrary headers, and browser scripts cannot be supplied. The page can issue public HTTPS GET requests for scripts, styles, and data. After observing a main-page challenge, the service also allows verification POSTs from fetch/XHR and verification images under `/cdn-cgi/challenge-platform/` on that page's origin, `challenges.cloudflare.com`, and its subdomains. There are at most 20 verification POSTs/images per attempt, with POST bodies capped at 128 KiB. A Cloudflare challenge can also hand back the main page through a URL-encoded verification POST to the same origin and path, preserving application query parameters. Only token-shaped verification fields are accepted, with at most eight unique fields. This requires an observed `cf-mitigated: challenge` main-page response. Other form navigations return `navigation_blocked`. Handoff requests share the verification count and body limits. These requests count toward the overall resource budget and retain all DNS, HTTPS, host scope, and transfer checks. Application POSTs, other methods, ordinary images, fonts, media, downloads, and WebSockets are blocked; service workers are disabled. Workflows that require application POST data loading are outside this alpha contract. An unavailable secondary resource increases `meta.network.connectionFailures` without replacing a usable page's result. A failed main navigation still fails delivery. Private DNS answers and transfer budget violations fail the attempt. Public DNS addresses are validated and pinned into the actual TCP connection. IP literals, internal names and addresses, credential-bearing URLs, HTTP, custom ports, and private redirects are refused. The browser retains its native TLS, certificate verification, and original hostname/SNI. A Chromium sandbox is enabled. Deploy the service as its own unprivileged process with the supplied memory and CPU limits. ### Encrypted restart persistence and shared quotas Named sessions remain in RAM by default. Pair `VITNA_UNBLOCKER_SESSION_DATA` with an exact 32-byte hexadecimal `VITNA_UNBLOCKER_SESSION_KEY_HEX` to retain encrypted cookies/local storage across restarts. Use a separate private directory from `VITNA_UNBLOCKER_DATA`; requests still require `persistSession:true`. AES-256-GCM binds each stored state to its tenant, session, expiry and byte bounds. A key that opens no stored row fails startup closed and changes nothing. Modified ciphertext is never used or overwritten: when the key opens other rows, startup moves the damaged row aside unchanged under a `browser-session-quarantine:v1:` key and logs the count, and at runtime it fails only its own session, which its owner can still delete. Each request decrypts only its own session's row. Deletion, TTL and per-client/global limits survive restart. Storage closes after physical browser cleanup, preserving successful state. Existing POSIX directories/files must be privately owned by the service user; Windows deployments must supply private ACLs. Directory/data links and hardlinked databases are rejected. Retain the operator key separately from a consistent database backup. Rotation and replicated session storage are not implemented. For multiple standalone API processes, explicitly configure `VITNA_UNBLOCKER_QUOTA_BACKEND=shared`, `VITNA_UNBLOCKER_QUOTA_URL` and `VITNA_UNBLOCKER_QUOTA_TOKEN` to use the owned quota authority. Its service limits govern both API requests and each browser attempt, including retries. Status reports shared limits and counters; unavailable admission fails closed without automatic retries or local fallback. One authority writer coordinates quota over durable SQLite, while browser concurrency and sessions remain local. See [shared Web quota setup](../web/overview.md) and `deploy/vitna-quota.env.example`. Local quotas remain the default. ## Proxy routes Direct connections work without a proxy account. An operator can configure HTTP or HTTPS CONNECT proxies in the service's private environment: ```dotenv VITNA_UNBLOCKER_PROXIES=["https://user:password@proxy.example.com:8443","http://user:password@proxy2.example.com:8080"] ``` To expose country selection, label the configured routes: ```dotenv VITNA_UNBLOCKER_PROXY_ROUTES=[{"id":"us-primary","country":"US","proxy":"https://user:password@proxy.example.com:8443"},{"id":"gb-primary","country":"GB","proxy":"https://user:password@proxy2.example.com:8443"}] ``` The request then selects one of those labels: ```json { "url": "https://example.com/", "country": "US", "session": "price-watch-01", "language": "en-US", "viewport": { "width": 1440, "height": 960 }, "headers": { "dnt": "1" } } ``` The lists accept up to 20 proxy URLs and 100 entries in all. A country request uses only routes that serve that country and fails with `country_unavailable` when none does. Without a session, selection cycles through the eligible configured routes. With a session, its identifier deterministically selects the same eligible route while that list stays unchanged. Vitna does not verify the provider's physical location or supply a proxy network. ### Residential and ISP provider routes A route can also come from a residential, ISP, or datacenter proxy account the operator holds. Give it a `kind` and credential templates in place of fixed credentials: ```dotenv VITNA_UNBLOCKER_PROXY_ROUTES=[{"id":"dc-us","country":"US","proxy":"https://user:password@proxy.example.com:8443"},{"id":"residential","kind":"residential","countries":"*","proxy":"http://gateway.provider.example:7000","session":{"username":"customer-ACCOUNT[-country-{country}]-session-{session}","password":"PASSWORD"}}] ``` | Field | Meaning | |---|---| | `kind` | `datacenter` (the default), `isp`, or `residential` | | `country` | The one uppercase two-letter country a route serves | | `countries` | `"*"` or a list of uppercase codes the provider can target; requires `{country}` or `{COUNTRY}` in a template | | `session` | `username` and `password` templates, each 1 to 512 characters | In a template, `{session}` becomes a token for the attempt: a fresh one each time, so each attempt can leave from a new address, or, when the request names a `session`, one derived from the client and that session, so the provider can keep its address. The request's session name never reaches the provider. `{country}` and `{COUNTRY}` become the requested country in lower and upper case. A `[...]` segment is kept only when every placeholder in it has a value, which is how a template serves requests that name no country. A template with `{country}` outside a segment serves only requests that name one. Vitna assumes no provider syntax beyond this: write the username format your provider documents into the template. Attempts start on the cheapest network configured. A block, challenge, or HTTP error from the target moves the next attempt up a tier, from datacenter to ISP to residential, within the request's `retries`; other failures retry on the same tier. A request can name a tier with `network`, and fails with `network_unavailable` when no route of that kind serves it. A route whose proxy refuses the connection rests for a minute while the others serve. Each attempt entry names its `route` and `network`. Credentials are percent-encoded URL credentials. They stay inside the connector and never enter receipts or API responses. Proxy providers must support CONNECT to a pinned public IP; TLS still uses the target's original hostname. Network access and proxy capacity remain the operator's infrastructure. ## Run an owned egress gateway Run Vitna's CONNECT server on an operator-controlled machine. It needs no proxy provider API. Keep the password private and create a credential hash from environment variables: ```bash npm run egress:hash ``` The command reads `VITNA_EGRESS_USERNAME` and `VITNA_EGRESS_PASSWORD`, then prints only SHA-256 of `username:password`. The username uses 1 to 64 letters, numbers, dots, underscores, or hyphens; the password is 16 to 256 characters without control characters. Store the hash in an ignored egress configuration file, rather than its plaintext password: ```dotenv VITNA_EGRESS_OPERATORS={"browser":"REPLACE_WITH_PRINTED_CREDENTIAL_HASH"} VITNA_EGRESS_HOST=127.0.0.1 VITNA_EGRESS_PORT=8791 ``` ```bash node --env-file=.env.egress platform/unblocker/egress-start.mjs ``` With settings already in the environment, use: ```bash npm run egress:start ``` Configure a browser service's private upstream URL using that username and URL-encoded password. The examples are placeholders: ```dotenv VITNA_UNBLOCKER_PROXY_ROUTES=[{"id":"owned-us","country":"US","proxy":"https://browser:URL_ENCODED_PASSWORD@egress.example.com:8791"}] # The Web suite uses the same route shape. VITNA_WEB_PROXY_ROUTES=[{"id":"owned-us","country":"US","proxy":"https://browser:URL_ENCODED_PASSWORD@egress.example.com:8791"}] ``` The gateway binds to loopback HTTP by default. For HTTPS CONNECT, configure both `VITNA_EGRESS_TLS_KEY_FILE` and `VITNA_EGRESS_TLS_CERT_FILE` with the operator's PEM files. Match the upstream URL scheme to the configured listener. Operators authenticate with Basic `Proxy-Authorization`, which plain HTTP carries readable, so startup refuses any `VITNA_EGRESS_HOST` other than `127.0.0.1`, `::1`, or `localhost` unless both TLS files are configured; the supplied Dockerfile binds loopback for the same reason. A regular website reverse proxy may not support forwarding CONNECT tunnels. Only authenticated CONNECT to public destinations on port 443 is accepted. DNS results must all be public; the connection pins the checked public address. Private targets and other ports are refused. Browser requests already pin their upstream CONNECT destination to a public IP, so an egress hostname-only allowlist does not match those tunnels. If restricting the gateway, list the explicit public IPs the browser will use, or leave the allowlist empty for public destinations. Canonical IPv6 entries use brackets. `GET /healthz` reports only process health; other ordinary HTTP methods return `405`. The gateway does not terminate the target website's TLS. | Environment | Default | Meaning | |---|---|---| | `VITNA_EGRESS_ALLOWED_HOSTS` | All public destinations | Comma-separated exact public hostnames or IPs | | `VITNA_EGRESS_CONCURRENCY` | 32 | Active CONNECT tunnels | | `VITNA_EGRESS_QUEUE_LIMIT` | 64 | Waiting authenticated tunnels | | `VITNA_EGRESS_CONNECTION_LIMIT` | 128 | Accepted client sockets, including incomplete headers | | `VITNA_EGRESS_MAX_BYTES` | 20,971,520 | Per-tunnel bytes in both directions | | `VITNA_EGRESS_TUNNEL_TIMEOUT_MS` | 60,000 | Whole tunnel deadline | | `VITNA_EGRESS_CONNECT_TIMEOUT_MS` | 10,000 | DNS and connection deadline | | `VITNA_EGRESS_QUEUE_TIMEOUT_MS` | 5,000 | Maximum queue wait | | `VITNA_EGRESS_HEADER_TIMEOUT_MS` | 5,000 | Request header deadline | Place owned nodes in the regions your workloads need and label their routes in the browser configuration. Country labels describe the operator's routing choice; Vitna does not verify location or provision regional machines, IP capacity, or a residential network. Limits apply to each gateway process. The supplied `deploy/vitna-egress.env.example`, `deploy/vitna-egress.service`, and `deploy/vitna-egress.Dockerfile` prepare an operator deployment; they do not provision or publish a gateway. ### Sell proxy access from a gateway A gateway can also carry customers' own traffic on proxy keys their accounts mint, once it is joined to the Web coordinator that holds those accounts. Mint the gateway's token, give the gateway the token and the coordinator's origin, and give the coordinator the token's hash: ```bash npm run egress:gateway-key ``` ```dotenv # The gateway VITNA_EGRESS_ACCOUNTS_URL=https://web.example.com VITNA_EGRESS_GATEWAY_TOKEN=vitna_gateway_PRINTED_TOKEN VITNA_EGRESS_HOST=0.0.0.0 VITNA_EGRESS_TLS_KEY_FILE=/run/secrets/egress.key VITNA_EGRESS_TLS_CERT_FILE=/run/secrets/egress.crt # The Web coordinator VITNA_WEB_PROXY_GATEWAYS=[{"id":"egress-de","tokenHash":"PRINTED_HASH","endpoint":"egress-de.example.com:8791","country":"DE"}] ``` Customers sign in as `ACCOUNT:vitna_proxy_...`; accounts and grants are covered in [proxy access](../web/overview.md#proxy-access). The gateway asks the coordinator about a key and keeps the answer for 30 seconds. While the coordinator cannot be reached, during a deploy for example, it keeps an allowed answer for up to five minutes. Every minute it reports the bytes each account moved, as running totals for the hour. The coordinator settles each hour into the account's ledger 15 minutes after the hour ends. A gateway that stops loses at most its last minute of counting, which is never billed. Operator credentials keep working alongside customers', and `VITNA_EGRESS_OPERATORS` may be empty on a gateway that serves only customers. Customer tunnels share the gateway's `VITNA_EGRESS_CONCURRENCY` and `VITNA_EGRESS_CONNECTION_LIMIT`. Raise those limits for customer traffic, or run separate gateways so customers cannot crowd out your own browsers. For customers, the gateway enforces: - **HTTPS only.** CONNECT to port 443 of a public address, checked and pinned as for operators. There is no plain HTTP and no other port, so no mail. - **The account's grant.** Tunnels open at once, tunnels started per minute, and the month's bytes. When the allowance runs out, the gateway closes all of the account's open tunnels. - **Blocked hosts.** `VITNA_EGRESS_BLOCKED_HOSTS` lists hosts customers may not reach; each entry also covers every name under it. - **Tunnel limits.** Bytes, total lifetime and idle time, set by the `VITNA_EGRESS_CUSTOMER_*` settings below. - **Failed sign-ins.** After 30 failed sign-ins in a minute from one network (an IPv4 /24 or IPv6 /64), customer sign-ins from that network answer 429 until the minute ends, and the coordinator is not asked. Operators are never held back. - **Revocation.** Open tunnels are checked again every 30 seconds as their answers age, so a revoked key or a suspended account loses its tunnels within about a minute. A refusal names its reason in an `X-Vitna-Error` header: | HTTP | `X-Vitna-Error` | |---|---| | 403 | `proxy_access_suspended`, `proxy_access_not_granted`, `monthly_allowance_exhausted`, `destination_blocked` | | 407 | None: the account or key is not valid | | 429 | `concurrency_limit`, `tunnel_rate_limit`, `authentication_rate_limited` | | 503 | `accounts_unavailable` when the coordinator cannot answer | Each customer tunnel writes one JSON line to standard output, with `event` set to `customer_tunnel` and the fields `at`, `client`, `host`, `bytes`, `durationMs`, and `outcome`. It records the account, the destination host and the volume, which is what an abuse report needs. It never records content, URLs, or the key. Under systemd the journal keeps these lines; set its retention (`MaxRetentionSec=`) to match your policy. The gateway limits what a customer can do. It does not vet who the customer is: that check happens when you grant access. | Environment | Default | Meaning | |---|---|---| | `VITNA_EGRESS_ACCOUNTS_URL` | Unconfigured | The Web coordinator's HTTPS origin (loopback HTTP on one machine); with the token, serves customer proxy keys | | `VITNA_EGRESS_GATEWAY_TOKEN` | Unconfigured | This gateway's `vitna_gateway_` token | | `VITNA_EGRESS_BLOCKED_HOSTS` | None | Comma-separated lowercase hosts or IPs customers may not reach, each with every name under it | | `VITNA_EGRESS_CUSTOMER_MAX_BYTES` | 1,073,741,824 | Per customer tunnel, both directions | | `VITNA_EGRESS_CUSTOMER_TUNNEL_TIMEOUT_MS` | 600,000 | Whole customer tunnel deadline | | `VITNA_EGRESS_CUSTOMER_IDLE_TIMEOUT_MS` | 120,000 | Customer tunnel time without a byte | | `VITNA_EGRESS_SIGN_IN_FAILURES_PER_MINUTE` | 30 | Failed sign-ins per network per minute before 429 | ## Browser profiles and live checks The default `native` profile uses the installed or bundled Chromium runtime. An optional `cloakbrowser` profile uses an operator-installed CloakBrowser executable with its fingerprint flags and desktop viewport defaults. Requests can select a different viewport within the common limits. It retains the Chromium sandbox, certificate checks, guarded proxy, and request limits. The profile requires `VITNA_UNBLOCKER_BROWSER_PATH`; no vendor binary is bundled, downloaded at service startup, or substituted automatically. ```dotenv VITNA_UNBLOCKER_BROWSER_PROFILE=cloakbrowser VITNA_UNBLOCKER_BROWSER_PATH=/opt/cloakbrowser/chrome ``` This profile was locally verified with CloakBrowser 146 against the public Cloudflare and Antibot demonstration pages. The native Edge runtime on the same machine remained challenged. Other versions and deployment platforms need their own verification. CloakBrowser's [binary license](https://github.com/CloakHQ/CloakBrowser/blob/main/BINARY-LICENSE.md) requires a separate OEM/SaaS agreement for customer-facing browser API use; an internal test does not provision that agreement or a hosted service. ### The camoufox profile The `camoufox` profile renders in [Camoufox](https://camoufox.com/python), the Firefox build Search's Google collector already runs, and is the one profile measured to clear the challenge demonstrations without a licence agreement. On 2026-10-16, on one Windows desktop with no proxy, hardened Chromium that passed every Sannysoft test still held 0 of 6 challenge attempts, while Camoufox cleared 18 of 20 in a standalone trial and, through this service's own guarded proxy, 18 of 20 one at a time and 14 of 20 four at a time ([Measured on 2026-10-16](#measured-on-2026-10-16)). Camoufox's fingerprints come from its Python package, so the host needs a Python with that package and its browser fetched, pinned as Search pins them: ```bash python3 -m venv /opt/vitna-camoufox /opt/vitna-camoufox/bin/pip install "camoufox[geoip]==0.5.7" /opt/vitna-camoufox/bin/python -m camoufox fetch ``` ```dotenv VITNA_UNBLOCKER_BROWSER_PROFILE=camoufox VITNA_UNBLOCKER_CAMOUFOX_PYTHON=/opt/vitna-camoufox/bin/python VITNA_UNBLOCKER_CAMOUFOX_FINGERPRINTS=8 ``` `camoufox fetch` puts the browser in the cache directory of the user who runs it, so run it as the service's user, and give the service the same `XDG_CACHE_HOME` when its `HOME` does not last, as a systemd runtime directory does not. For the hosted service, `deploy/web-api/install-camoufox.sh` does both and checks the result under the unit's confinement (`deploy/web-api/OVH-RUNBOOK.md`, The auto browser profile). At startup the service runs `platform/unblocker/camoufox-options.py` once for that many fingerprints (1 to 32, 8 by default) and refuses to start if it cannot. Attempts take turns over them; a request that names a `session` keeps one, so a persisted session's cookies return to the browser that earned them. The helper runs with only the environment it needs to find Python and Camoufox's cache, and only Camoufox's own `CAMOU_CONFIG_n` and `CAMOU_PREFS_n` keys reach the browser: Camoufox's launch options copy the whole process environment, and the service refuses any other key. Firefox has no launch flags for what the Chromium profiles enforce with them, so the profile sets preferences that no fingerprint can override: every connection, loopback names included, goes through this attempt's guarded proxy, which resolves names, refuses private answers and pins the public address, and none falls back to a direct connection; DNS-over-HTTPS, prefetching, speculative connections and QUIC are off, and WebRTC carries no UDP outside the proxy. Printing a page to PDF and reading a document's response are Chromium's alone and are refused on this profile; the public Unblocker uses neither. It is slower: four at a time on one desktop, its median page took 3 to 8 seconds where Chromium's took about one, so it suits pages that sit behind a browser check rather than every page. The `camoufox` package is MIT-licensed by its package metadata (read 2026-10-16). The browser it fetches is a Firefox build with no licence file of its own; review its terms before selling access through it, as for any browser this service runs. `VITNA_UNBLOCKER_CAMOUFOX_OS` limits the fingerprints to a comma list of `windows`, `macos` and `linux`; unset, Camoufox chooses among all three. On a Windows host on 2026-10-16, Windows-only fingerprints cleared the challenge demos as often as mixed ones (18 of 20 each, one at a time) at half the median time, 4.7 seconds against 9.7, so measure a host with its own operating system before choosing. ### The auto profile `auto` reads every page in Chromium and moves a request to Camoufox only after a challenge has held it, so ordinary pages keep Chromium's speed and a page behind a browser check still clears: ```dotenv VITNA_UNBLOCKER_BROWSER_PROFILE=auto VITNA_UNBLOCKER_CAMOUFOX_PYTHON=/opt/vitna-camoufox/bin/python ``` Under `auto` the first attempt reads in Chromium. If it meets a challenge it does not wait one out, since Chromium has cleared none of the demonstrations it met; it ends at once with `target_challenge`, and the next attempt reads in Camoufox with the full challenge wait. The step uses the request's own `retries`, so it needs at least one (the default); with `retries: 0` a challenged page ends `target_challenge` after the Chromium attempt. A block that is not a challenge steps the network tier and stays in Chromium. A capture stays in Chromium, which alone prints it. The Web API's reads, which also take a document a URL serves, move as pages do. Firefox has no response interception, so Camoufox takes such a document from its navigation's own response, which Playwright keeps once the context accepts downloads; only those reads accept them. This service's own reads never take a document, so Camoufox still refuses one here as `target_content_type`. If Camoufox cannot be readied when the service starts, `auto` starts on Chromium alone and logs the reason in one line, so a broken Camoufox install costs the step up rather than the service; the `camoufox` profile refuses to start instead. On Linux, Camoufox's fingerprints carry the `FONTCONFIG_FILE` it writes for the operating system they claim, which the browser is given beside the `CAMOU_` keys. Each attempt entry says which browser it used: ```json { "attempts": [ { "attempt": 1, "outcome": "target_challenge", "statusCode": 403, "browser": "chromium" }, { "attempt": 2, "outcome": "delivered", "browser": "camoufox" } ] } ``` Run the explicit live check using the same browser and proxy settings as the service. It creates a temporary local API key, sends authenticated HTTP requests to a local process, verifies page content, and exits: ```bash npm run unblocker:check npm run unblocker:check -- outputs/unblocker-check.json npm run unblocker:check -- --extended outputs/unblocker-extended-check.json ``` The check contacts `https://quotes.toscrape.com/js/`, `https://www.scrapingcourse.com/cloudflare-challenge`, and `https://www.scrapingcourse.com/antibot-challenge`. It exits unsuccessfully if any target is not delivered with its expected content. Default unit tests use fixtures and do not contact these websites. The `--extended` suite adds JavaScript-disabled comparisons, JavaScript pagination, a 10-second rendering delay, AJAX quotes, AJAX products, two fresh attempts per challenge page, and Sannysoft fingerprint diagnostics. It verifies rendered item counts and readable content, excluding inline scripts from item counts. The product check waits for the last populated product name; the page's initial empty product cards are loading placeholders. The delayed-page control deliberately omits `waitFor` to exercise automatic readiness. Stable visible page content does not establish that delayed quote items are ready; a persistent empty or loading state fails with `target_incomplete`. The matching selector case waits for all 10 quotes. The AJAX quote check covers the initial batch; it does not scroll or click to collect further batches. Fingerprint diagnostics report each detection test separately in `detectionTests` and `detectionPassed`. The report's `passed` field describes delivery and content assertions, including the controls; it does not mean every fingerprint detector was passed or establish protection against other anti-bot systems. Reports omit API keys, raw page content, and fingerprint values. ## Comparing with Bright Data An explicit comparison runner sends the same four public pages through the local authenticated Vitna HTTP API and Bright Data's [Web Unlocker REST API](https://docs.brightdata.com/products/web-unlocker/send-your-first-request). It covers immediate JavaScript, a 10-second JavaScript delay, and the two challenge demonstrations. Two rounds alternate provider order, with at most eight Bright Data requests. These requests use your existing account and may be billable under its plan; the runner does not create a zone or change account settings. Set `BRIGHTDATA_API_KEY` and `BRIGHTDATA_UNLOCKER_ZONE` in an ignored local `.env.brightdata` file. Configure Vitna's browser settings in the environment as for the live check, then run: ```bash node --env-file=.env.brightdata platform/unblocker/benchmark.mjs outputs/unblocker-benchmark/comparison.json npm run unblocker:benchmark -- --vitna-only outputs/unblocker-benchmark/vitna-baseline.json ``` The second command runs only the Vitna baseline. Its report explicitly marks Bright Data as `not_run` and `comparisonComplete` as false. The default comparison refuses to send requests when either Bright Data credential is missing, and stops further provider requests on authentication, configuration, quota, or malformed-response errors. Timing covers the caller's entire request through the complete decoded response body. Both providers must return the expected readable content and populated item counts, with no remaining challenge. An HTTP 200 containing only JavaScript source or an empty loading page fails verification. Reports exclude credentials, provider zone identifiers, response headers, raw HTML, and cookies. Average latency includes only content-verified responses and must be read alongside success counts. Vitna uses an explicit `waitFor` selector and a fresh browser per request; the default comparison uses Bright Data's automatic behavior without custom headers. The provider's browser, retries, session reuse, proxy route, and cache state are unobserved. This small demo comparison includes local versus remote API transport overhead and does not establish production performance or isolate JavaScript challenge-solving time. The bulk runner uses ten pages from three public training-site domains: five JavaScript/AJAX pages, three static controls, and two challenge demos. Its default is four rounds (40 attempts per provider) with four requests in flight per provider. Both provider batches start together; each has its own bounded queue. Page order rotates each round. ```bash node --env-file=.env.brightdata platform/unblocker/bulk-benchmark.mjs outputs/unblocker-benchmark/bulk.json npm run unblocker:bulk -- --vitna-only --rounds 4 --concurrency 2 outputs/unblocker-benchmark/bulk-vitna.json ``` `--rounds` accepts 1–10 and `--concurrency` accepts 1–4. These cap each provider at 100 attempts; the default comparison uses at most 40 billable Bright Data requests. Failed account/configuration requests stop further scheduling; already-started requests settle. Interrupting the runner aborts in-flight calls and preserves a partial report. No failed request is retried. For a browser-rendering comparison, `--brightdata-render` sends the documented [`render: "true"` parameter](https://docs.brightdata.com/products/web-unlocker/features#force-javascript-rendering-with-a-browser) to Bright Data. `--javascript-only` selects the five JavaScript/AJAX targets for a smoke test. `--timeout-ms` accepts 1,000–60,000 and sets the same caller deadline for both APIs, plus Vitna's request and ephemeral service deadline. Without that option, the original 35-second caller and 30-second Vitna deadlines remain. The report records forced rendering and the deadlines. With credentials and browser settings already in the environment: ```bash npm run unblocker:bulk -- --brightdata-render --javascript-only --rounds 1 --timeout-ms 60000 outputs/unblocker-benchmark/smoke-rendered.json npm run unblocker:bulk -- --brightdata-render --rounds 10 --timeout-ms 60000 outputs/unblocker-benchmark/bulk-rendered-100.json ``` The smoke uses at most five Bright Data requests; the full run uses at most 100. Run the smoke before the larger workload. Forced rendering retains Bright Data's automatic readiness rather than Vitna's explicit selectors, and its browser, routes, cache, and internal retries remain unobserved. Each provider's batch reports content-verified throughput, p50 and p95 latency using nearest rank, failure counts, and results by static, JavaScript, and challenge category. Latency for verified pages and latency for all attempts are recorded separately. An empty JavaScript shell remains a content failure even if its API request returned quickly. The isolated benchmark service admits 120 requests per minute and uses the requested browser concurrency. These are benchmark configuration values; they do not change another running service's quotas or capacity. Repeating these ten pages tests consistency and resource contention within this workload; it does not expand the suite's coverage of production websites. ### Measured on 2026-10-16 The full Vitna-only run (`--vitna-only --rounds 10 --concurrency 4 --timeout-ms 60000`), on one Windows x64 desktop with the native Edge profile and no proxy, is the largest Unblocker run on record: [unblocker-bulk-100-20261016.json](../../deploy/web-api/evidence/unblocker-bulk-100-20261016.json). | Pages | Verified | Verified p50 | Verified p95 | |---|---|---|---| | Static, 30 | 30 | 0.89 s | 1.44 s | | JavaScript, 50 | 50 | 1.12 s | 11.5 s, the page that waits 10 seconds | | Challenge demos, 20 | 0 | | | | All, 100 | 80 | 1.07 s | 11.4 s | The 100 took 117 seconds with four browsers in flight, 0.69 verified pages a second. Every challenge attempt ended `target_challenge` after the 12-second wait. The extended live check the same evening ([unblocker-extended-check-20261016.json](../../deploy/web-api/evidence/unblocker-extended-check-20261016.json)) agrees: all four attempts at the two challenge demos stayed challenged, each after two widget interactions and ten verification requests, and Sannysoft flagged 4 of its 31 fingerprint tests. It also shows the limit of automatic readiness: the 10-second delayed page without `waitFor` was returned as delivered holding none of its 10 quotes, so name a selector for content that arrives late. The same run on the [camoufox profile](#the-camoufox-profile) that evening ([unblocker-bulk-camoufox-100-20261016.json](../../deploy/web-api/evidence/unblocker-bulk-camoufox-100-20261016.json)): | Pages | Native, verified | Camoufox, verified | Camoufox p50 | Camoufox p95 | |---|---|---|---|---| | Static, 30 | 30 | 30 | 3.33 s | 11.3 s | | JavaScript, 50 | 50 | 50 | 5.56 s | 20.9 s | | Challenge demos, 20 | 0 | 14 | 7.97 s | 28.5 s | | All, 100 | 80 | 94 | 5.69 s | 17.5 s | It took 220 seconds, 0.43 verified pages a second. Each of the six misses ended `target_challenge`, mostly in the first round, while four Firefox browsers were starting at once on one desktop. One at a time, as the hosted service runs (`VITNA_UNBLOCKER_CONCURRENCY=1`), the two demos ten times each cleared 18 of 20 at the default 12-second wait. The extended live check on the same profile ([unblocker-extended-check-camoufox-20261016.json](../../deploy/web-api/evidence/unblocker-extended-check-camoufox-20261016.json)) delivered all ten ordinary pages and cleared three of its four challenge attempts; the fourth ran out the 12-second wait. Sannysoft flagged only `Chrome (New)`, which a Firefox build has no reason to pass. The delayed page without `waitFor` was again delivered with none of its quotes. The [auto profile](#the-auto-profile) the next morning ([unblocker-bulk-auto-100-20261017.json](../../deploy/web-api/evidence/unblocker-bulk-auto-100-20261017.json)), same pages and settings with Windows fingerprints and the one retry its step needs (`--retries 1`), delivered all 100: static 30 of 30 at a 1.00-second median, JavaScript 50 of 50 at 1.21 seconds, and challenge demos 20 of 20 at 6.97 seconds. It took 89 seconds, 1.13 verified pages a second, faster than the native run's 117, because Chromium no longer waits out a challenge it cannot clear. The extended live check on it ([unblocker-extended-check-auto-20261017.json](../../deploy/web-api/evidence/unblocker-extended-check-auto-20261017.json)) cleared three of four challenge attempts at about 5.5 seconds each; the fourth ran out Camoufox's 12-second wait. Its Sannysoft row reads Chromium's four flags, rightly: that page sets no challenge, so auto never leaves Chromium for it. These are training pages on one machine. They say nothing about production websites, the hosted service's capacity, or a proxy or CloakBrowser profile, which these runs did not use. ## Configuration and status | Environment | Default | Meaning | |---|---|---| | `VITNA_UNBLOCKER_CLIENTS` | Required | JSON client identifier to SHA-256 token hash | | `VITNA_UNBLOCKER_HOST` | `127.0.0.1` | Listening interface | | `VITNA_UNBLOCKER_PORT` | 8788 | Listening port | | `VITNA_UNBLOCKER_DATA` | `.unblocker-data` | Standalone SQLite WAL directory for admission counters; separate from Web account/job state | | `VITNA_UNBLOCKER_METRICS_TOKEN_HASH` | Unconfigured | Lowercase 64-character SHA-256 hash of an operator metrics token; enables authenticated `GET /metrics` | | `VITNA_UNBLOCKER_RPM` | 20 | Per-client requests per UTC minute | | `VITNA_UNBLOCKER_DAILY_LIMIT` | 1,000 | Per-client requests per UTC day | | `VITNA_UNBLOCKER_CONCURRENCY` | 2 | Browser requests in flight per process | | `VITNA_UNBLOCKER_TIMEOUT_MS` | 30,000 | Maximum total deadline, up to 60,000 | | `VITNA_UNBLOCKER_CHALLENGE_WAIT_MS` | 12,000 | Maximum verification wait per attempt, 0 to 20,000; 0 disables waiting | | `VITNA_UNBLOCKER_PROXIES` | `[]` | Private ordered CONNECT proxy URLs | | `VITNA_UNBLOCKER_PROXY_ROUTES` | `[]` | Private route objects with a unique `id` and `proxy` URL, and optionally `kind`, `country` or `countries`, and `session` credential templates | | `VITNA_UNBLOCKER_ALLOWED_HOSTS` | All public hosts | Optional comma-separated exact hosts, including required asset hosts | | `VITNA_UNBLOCKER_BROWSER_CHANNEL` | Bundled Chromium | Optional installed browser channel, such as `msedge` | | `VITNA_UNBLOCKER_BROWSER_PATH` | Playwright default | Optional operator-selected executable path | | `VITNA_UNBLOCKER_BROWSER_PROFILE` | `native` | `native`, `cloakbrowser` (requires an installed executable path), `camoufox` or `auto`, Chromium first and Camoufox after a challenge (both require the Python below) | | `VITNA_UNBLOCKER_CAMOUFOX_PYTHON` | Unconfigured | `camoufox` and `auto`: a Python with the `camoufox` package installed and its browser fetched | | `VITNA_UNBLOCKER_CAMOUFOX_FINGERPRINTS` | 8 | `camoufox` and `auto`: fingerprints made at startup, 1 to 32 | | `VITNA_UNBLOCKER_CAMOUFOX_OS` | All three | `camoufox` and `auto`: comma list of `windows`, `macos`, `linux` the fingerprints may claim | | `VITNA_UNBLOCKER_SERVICE_URL` | Unconfigured | Website process only: destination for its relay | The request body is capped at 8 KiB. A page has at most 400 allowed intercepted resource requests, 20 MiB of tunnel transfer, and 2 MiB of serialized HTML. Chromium also bounds HTTP redirect chains. Startup stores admission counters in `VITNA_UNBLOCKER_DATA` (default `.unblocker-data`) with SQLite WAL transactions. Keep this volume across restart and use one writer per directory. Embedded service instances without a store use memory; independent replicas do not share quotas. The directory must differ from Web account/job state. Failures consume admission quota; retries remain one API request. Durable shared quotas are still required before selling usage across replicas. `GET /healthz` confirms the HTTP process; it does not launch a browser or verify a proxy account. Authenticated `GET /v1/status` reports configured limits, current concurrency, configured quota storage, and disabled billing. Receipts log client identity, target hostname, outcome, attempt count, and duration. Receipts omit URL paths/queries, keys, proxy credentials, and fetched content. No search index or raw HTML archive is written. Opted-in cookies and localStorage stay in the bounded RAM session store described above. ## Operator metrics The standalone server uses `platform/web/metrics.mjs` for authenticated Prometheus text metrics. Generate an operator credential locally: ```bash node platform/web/metrics.mjs --key ``` The command prints JSON containing `token` and `hash`. Keep the token private for your scraper and configure only its printed hash in the service: ```dotenv VITNA_UNBLOCKER_METRICS_TOKEN_HASH=REPLACE_WITH_PRINTED_LOWERCASE_SHA256_HASH ``` After restarting with that setting, use the printed token as a separate operator bearer credential: ```bash curl http://127.0.0.1:8788/metrics \ -H "Authorization: Bearer $VITNA_METRICS_TOKEN" ``` `GET /metrics` returns `text/plain; version=0.0.4` with `Cache-Control: no-store`. The endpoint accepts no query parameters or body. It returns 404 when no metrics hash is configured, 401 for missing or invalid operator credentials, and 405 for an authenticated method other than GET. Client API keys do not authorize metrics access. | Metric | Observation | |---|---| | `vitna_api_http_requests_total` | Completed responses by bounded method, operation, and status labels, including errors and metrics scrapes | | `vitna_api_http_duration_seconds` | Histogram of request time through response transfer, in seconds | | `vitna_api_http_in_flight` | Current HTTP requests | | `vitna_api_browser_active` | Browser requests in flight, sampled at HTTP request and response events | | `vitna_api_render_attempts_total` | Completed render attempts, including failed attempts | | `vitna_api_browser_milliseconds_total` | Measured render wall time including browser cleanup, not CPU time | | `vitna_api_transferred_bytes_total` | Tunnel bytes reported by delivered snapshots, including protocol overhead and both directions | | `vitna_api_resource_requests_total` | Allowed resource requests reported by delivered snapshots | Metrics carry `service="unblocker"`. Labels exclude client identities, tokens, URLs, hostnames, session identifiers, and fetched content. Series counts are bounded. Metrics are process-local and reset on restart; they are separate from the durable admission counters in `VITNA_UNBLOCKER_DATA` and do not aggregate independent replicas. Search, model, and priced-usage resource counters stay at zero for this standalone service. The service uses [Playwright Chromium](https://playwright.dev/docs/browsers) and [browser contexts](https://playwright.dev/docs/api/class-browsercontext). Challenge handling follows Cloudflare's documented [response header](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/) and [browser support limits](https://developers.cloudflare.com/cloudflare-challenges/reference/supported-browsers/). The comparable one-request product shape is described in [Bright Data's Web Unlocker documentation](https://docs.brightdata.com/products/web-unlocker/introduction). ## For operators: run the service yourself Operator reference. Everything from here down needs the Vitna repository, which is private, and is for the people who run the hosted service. A customer needs none of it: the hosted endpoint and an alpha key are under [Hosted access](#hosted-access). ### Run it locally The hosted API answers at `https://unblock.vitna.ai`. Since 2026-10-07 a browser that opens it, or its `/docs`, is sent to `/unblocker` and this reference on vitna.ai, where the product's pages live: the operator sets `VITNA_UNBLOCKER_PAGE_URL` and `VITNA_UNBLOCKER_DOCS_URL`, and a self-hosted instance with neither set serves its own playground. The browser service has its own pinned Playwright dependency. It does not share the website database, signing keys, or search index. ```bash npm run unblocker:install npm run unblocker:browser npm run unblocker:demo ``` Open `http://127.0.0.1:8788`. The demo opens a real Chromium session and executes an illustrative page's JavaScript through the authenticated API. The temporary demo key is filled into the local playground. No target site or paid proxy is contacted. To check that flow and exit: ```bash node platform/unblocker/demo.mjs --check ``` For live pages, mint a client key and keep its token private: ```bash npm run unblocker:key -- pilot ``` Put the printed hash, not the token, into an ignored `.env.unblocker`: ```dotenv VITNA_UNBLOCKER_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"} VITNA_UNBLOCKER_PORT=8788 ``` Start the live service and open its playground. Paste the token and enter a public HTTPS page: ```bash node --env-file=.env.unblocker platform/unblocker/start.mjs ``` The equivalent script, when the settings are already in the environment: ```bash npm run unblocker:start ``` The service binds to loopback by default. TLS termination belongs at a reverse proxy if it is exposed externally. `deploy/vitna-unblocker.service`, `deploy/vitna-unblocker.env.example`, `deploy/vitna-unblocker.nginx.conf.example`, and `deploy/vitna-unblocker.Dockerfile` prepare a standalone deployment. The hosted service at `https://unblock.vitna.ai` runs from `deploy/web-api/` instead, with its own unit, environment and runbook. --- <!-- https://vitna.ai/docs/runtime/overview.md --> # The runtime An optional process that applies an accepted decision per request, on your own infrastructure. It is the second way to consume a decision, beside the [config](../routing/config.md) a consumer loads into a variable it already reads. ## The distinction the whole thing rests on **Vitna's software may sit in your request path. Vitna's service may not.** The runtime loads a [signed policy](../routing/policy.md) from local disk at boot and answers every request from it. It makes no call to vitna.ai: not at boot, not per request, not on a timer. There is no endpoint that serves a policy to a process, and there will not be one. So Vitna can be entirely unavailable and every request keeps routing, because the answer to "where should this request go" was signed into a file before the request existed. ## What it is for A team with a gateway already has somewhere to put a decision, and config mode costs them no runtime footprint at all. A team without one has nowhere to load an environment fragment into. The runtime is the same decisions, applied. It also closes a gap the record has: a usage export read after the fact is scoped by whatever the exporting system could scope it by, which is rarely the lane. A [ledger](ledger.md) written at the moment of the request is lane-exact because the caller named the lane. ## What it will not do | It never | Because | |---|---| | Calls vitna.ai | A runtime that fetched its policy would put Vitna's availability inside yours. [Pushing the ledger](pushing.md) is a separate command an operator schedules, and the serving process holds no code path to vitna.ai with push enabled or without | | Substitutes a model | A lane that cannot be reached is an error you see, not a quiet reroute | | Estimates a token count | A provider that reported no usage is one whose usage the record does not know | | Serves what it cannot attribute | Every answer has a lane, a decision and a policy digest behind it, or is marked unrouted. [Relay](relay.md) marks every answer unrouted, because it has no policy and so no lane | | Sends your prompts anywhere but the provider | Your keys, your accounts, your traffic | | Retries, or serves past a chain | A second serve is a [fallback](../routing/fallbacks.md) a person registered against the lane's decision, walked in order and recorded per attempt; nothing is ever served that no chain names. This row said "never fails over" until R2 shipped chains, and went stale for a day | ## The pieces | Page | What it covers | |---|---| | [Relay: the runtime with no policy](relay.md) | The same runtime with no policy, serving every request as the model it names, for a developer with no register yet | | [Per-request routing with vitna-route](route.md) | Deterministic, cache-sticky per-request model and effort routing (`shadow` first, promoted by an Aperture-signed policy) | | [A lane on your own machine](local.md) | One command from a workload in your register to a runtime on localhost | | [Installing the runtime](installing.md) | npm, the container, and Deploy on a cloud account you own | | [Serving a lane](serving.md) | Starting it, the request it takes, and what it refuses | | [Redaction](redaction.md) | Lifting known credential formats out of a request before it leaves, and putting them back into the answer, where you turn it on | | [The response cache](cache.md) | Answering a repeat of a deterministic request from memory, where you turn it on, and writing it down as no provider's work | | [Putting a coding harness on a lane](harness.md) | One command that starts opencode or Crush against a lane, with no provider key in its environment | | [The ledger](ledger.md) | What one request records, and reading it back into the register | | [Measuring your unit profile](profile.md) | What one call on a lane consumes, measured from that ledger, locally | | [Pushing the ledger](pushing.md) | The opt-in command that sends closed days to the record | --- <!-- https://vitna.ai/docs/runtime/cache.md --> # The response cache A repeat of a deterministic request, answered from memory rather than by a provider. Off unless you turn it on, and never a saving. ```bash VITNA_CACHE_TTL_SECONDS=300 vitna-runtime vitna-relay --cache 300 ``` ## What it answers Only a request that asked for the most likely answer, once, and whole: - `temperature` set to `0`. An absent temperature is the provider's default, which samples, so a request that did not say `0` is never answered from memory. - One choice: `n` absent or `1`. - Not streamed. And only one the runtime reads as a chat completion. An Anthropic request that names a model is [passed to Anthropic as it was written](serving.md#the-other-dialect), and the cache leaves it alone as well. Zero is not a promise that a provider answers the same way twice. What the cache promises is narrower: an answer it gives is the answer a provider gave to exactly this request, within the last `VITNA_CACHE_TTL_SECONDS`. ## Exactly this request The key is the whole body the runtime would forward, in a canonical form in which key order does not matter and everything else does: the messages to the last character, the tool calls and their ids, `max_tokens`, `stop`, `tool_choice`, every field. For a lane it also holds the policy digest, the lane, and the caller's `x-vitna-cohort`, since those decide which attempt serves the request. So a reload that changes what a lane serves never answers with what the previous decision put there, and two callers on different sides of a traffic share never answer each other. ## A hit The answer is the one stored, with `x-vitna-cache: hit` on the response. Its [ledger](ledger.md) row says what happened and nothing more: | Field | On a hit | |---|---| | `serve`, `via` | `cache` | | `cache_hit` | `true` | | `outcome` | `served`, because the caller got an answer | | `units` | `0`, because no provider served it | | `inputTokens`, `outputTokens` and the cache parts | null, because nobody was asked anything | | `sku`, `decision`, `provider`, `model` | the serve whose answer it repeats, so the answer still resolves to the decision that authorized it | `vitna-runtime-push` and `vitna-runtime-profile` count hits apart from their figures and say how many, so a lane answered from memory neither reads as a sku that failed nor inflates what the sku did. `/status` counts hits, misses and entries. Nothing anywhere adds up what a hit would have cost: a price needs the date it applied on, and the runtime counts nothing in money. ## Asking for a fresh answer A caller's `cache-control` header is honoured: `no-cache` asks the provider even where an answer is stored, and replaces it; `no-store` keeps this request's answer out of memory. ## What it holds At most 1024 answers and 64 MB, the least recently used going first, in the serving process's memory. Nothing is written to disk, and a restart starts it empty. An answer larger than the whole allowance is not kept. Answers are kept after [redaction](redaction.md) has put a request's secrets back into them, so a cached answer may hold a secret the caller sent. It stays in the same process that already held it. --- <!-- https://vitna.ai/docs/runtime/harness.md --> # Putting a coding harness on a lane *An operator note.* The launcher is not in the published package: it runs from a checkout of Vitna's private repository, as does the local lane it looks for. Outside Vitna, point the harness at a runtime you [installed](installing.md) yourself, with the base URL [Serving a lane](serving.md) gives. A coding harness is the cheapest way to make a lane carry real traffic, because it is traffic somebody was going to generate anyway. One command starts one with its environment pointed at a runtime you are already running, and every turn of the session lands on the lane and leaves a [ledger](ledger.md) row. ```bash npm run harness:start -- opencode --lane deep-research ``` It needs a runtime answering first. [A lane on your own machine](local.md) is the shortest way to one and lands on the port this command already looks at: ```bash npm run runtime:local ``` [Serving a lane](serving.md) is the other way, and the launcher cannot tell them apart. It asks whatever is answering what it serves, rather than reading a policy a second time, so a lane it offers you is a lane that will answer. ## What it is, and what it is not It is a wrapper. It sets the environment a harness reads, and everything else you typed reaches the harness in the order you wrote it. It parses no harness flags, forwards no requests and sits in no request path. | Flag | Meaning | |---|---| | `--lane <alias>` | Which lane the session runs on. Required. | | `--url <base>` | The runtime. Default `http://127.0.0.1:8787`. | | `--print` | Shape the whole launch, print it, start nothing. | | `--list` | Name each harness and where it reads its configuration. | | `--context-window <n>`, `--max-output <n>` | Override what the harness is told the model can hold. | | anything else | The harness's, untouched. | A bare `--` ends the launcher's reading entirely, which is the escape for the day a harness ships a flag of its own called `--lane`. ## The harnesses Two, and a third is a decision rather than a default. Each entry is a promise to keep following somebody else's releases, so the list is short on purpose. | Harness | Install | Configured through | |---|---|---| | [opencode](https://opencode.ai) | `npm i -g opencode-ai` | `OPENCODE_CONFIG_CONTENT`, a whole config document in the environment | | [Crush](https://github.com/charmbracelet/crush) | `npm i -g @charmland/crush` | `CRUSH_GLOBAL_CONFIG`, a directory holding `crush.json` | | [Claude Code](https://claude.com/claude-code) | `npm i -g @anthropic-ai/claude-code` | its environment; no configuration file is written | Neither file on your disk is read, replaced or edited. opencode merges the document in the environment last, after its global config and the project's own; Crush is pointed at a temporary directory that is removed when the session ends. **Both are told the lane twice**, as the model and as the small model. A coding harness asks a cheap model for titles and summaries between turns, and a small model left pointing somewhere else is a request that never reaches the runtime and leaves no row, which makes a session look smaller than it was. ## Which session a row belongs to Every request a launched session makes says who it was for, so its rows can be totalled as one session rather than picked out by time and lane. The launcher tells each harness to send two headers the runtime records on every row ([the request](serving.md#the-request)): `x-vitna-agent`, which is `harness:` and the harness's id (`harness:opencode`, `harness:crush`, `harness:claude`), and `x-vitna-run`, a new id for each launch. The launcher prints both before the harness starts. | Harness | Told through | |---|---| | opencode | `headers` in the provider's `options`, which opencode hands to the provider it loads | | Crush | `extra_headers` on the provider | | Claude Code | `ANTHROPIC_CUSTOM_HEADERS`, one `Name: Value` per line | A real opencode 1.18.35 session and a real Claude Code 2.1.283 session, run against a real runtime with only the provider stubbed, left rows carrying both, the small model's included, and the provider was sent neither. Crush was read from its source at v0.100.0 rather than run. The colon keeps a harness apart from a routine: a routine's id is lower case letters, digits and hyphens, so a routine named `opencode` is never counted as the harness. ## What the harness never gets Two mechanisms, and the point of having both is that neither is sufficient. **Its environment has every provider credential removed.** Anything named `*_API_KEY`, `*_AUTH_TOKEN`, `*_API_TOKEN`, `*_SECRET_KEY`, `*_ACCESS_KEY` or `*_CREDENTIALS`, the AWS and Google credentials that are not shaped like that, and the whole of `VITNA_*`, which is where the signing key and the push key live. What was withheld is printed by name and never by value, the same rule `/status` follows for the keys the runtime does hold. The shape is deliberately wider than the model providers, and that has a cost worth knowing: a key for something that is not a model at all, a sandbox or a deployment target, matches it too, so a harness tool that wanted one finds it missing. That is the right way round for a launcher whose whole point is that the runtime holds the credential relationship, and it is not silent. Every launch prints the list. **And the harness is told to load no provider but this one.** That is `enabled_providers` for opencode and `CRUSH_DISABLE_DEFAULT_PROVIDERS` for Crush. This is the half that matters, because clearing the environment cannot reach a credential the harness already stored on disk under its own account. What it gets instead is `VITNA_LANE_KEY`, whose value says in its own words that it is not a credential. A runtime started without `VITNA_INBOUND_KEY` reads no inbound authorisation at all, so the value is arbitrary; sending nothing is not an option, because a harness with no key configured decides its provider is not set up and never asks it anything. That makes this command one for a runtime on loopback, which is where one listens by default. A runtime with an inbound key (see [the front door](serving.md#the-front-door)) answers this command's `/status` read with its version and nothing else, and refuses the placeholder on `/v1`, so the command says the runtime serves no lane. Carrying the key through is not built yet. ## Where the numbers in its config come from Both harnesses want a context window, an output ceiling and a price per million tokens. The window and the ceiling decide when the harness compacts its own history; the price is what it shows you at the end of a session. They come from the record. The launcher reads what the runtime says the lane resolved to and looks that model up in the catalogue by its exact id, so a session priced in a harness's status bar is priced from a figure with a date on it rather than from a table its author last updated at some point. **A lane routed direct will miss**, and the launcher says so rather than finding a lookalike. A direct lane sends the provider's own name for a model and the catalogue is keyed by OpenRouter's, and turning one into the other is a lookup the record makes rather than a rule anything may apply, which is what [Model ids](../routing/model-ids.md) is about. On a miss the window and the ceiling are the launcher's own defaults, named as defaults, and the prices are zero rather than a plausible number. Zero is visibly not a price. A guess is not visibly anything. ## Tools, and the one lane shape that will not work A coding harness sends `tools` on every turn. The OpenRouter and OpenAI direct routes forward the body as it arrived, so tool calling works on those today. The Anthropic direct route translates, and it refuses `tools` by name rather than dropping them, for the reason [Serving a lane](serving.md) gives: a caller who sent tools, had them dropped and got an answer produced without them has a wrong answer that looks like a right one. So **an Anthropic direct lane will refuse the first turn of any harness session.** The launcher warns when a lane routes direct. `VITNA_ROUTE_ANTHROPIC=openrouter` on the runtime moves it to the substrate, which speaks the dialect natively. ## Codex It cannot reach a lane, and it is named by the launcher rather than left out of it. Codex stopped speaking this dialect: `wire_api = "chat"` was removed and is now refused by name, the only value its provider config takes is `responses`, and the runtime serves `POST /v1/chat/completions` and has no `/v1/responses`. The removal was announced on 2025-12-09 for early February 2026, and a version without it is 0.84.0 (read from `openai/codex` at rust-v0.154.0 and its discussion 7782, retrieved 2026-09-10). It needs an inbound Responses dialect this runtime does not have. Pointing it at a vendor's own compatible endpoint would bypass the runtime and write no ledger row, which forfeits the whole point. Claude Code was in this section for the same shape of reason, needing an inbound Anthropic dialect. The runtime serves `POST /v1/messages` since 2026-09-10, so it is in the table above instead. ## Claude Code, and what running it found Its entry was checked against 2.1.267 by running it, which is worth saying because running it is the only thing that could have produced any of the following. Nothing here comes from a published description of that dialect. **It sends three fields nothing documents.** `thinking`, as `{"type":"adaptive"}` rather than a token budget, `context_management`, and `output_config`. All three were refused by name, which is the runtime working as designed and also a session that cannot make one request. **Two of them turn off together**, which is why the launcher sets `MAX_THINKING_TOKENS` to `0` and says so when it starts. The `context_management` this harness sends is an instruction to clear thinking blocks, so a session with no thinking has nothing to manage. **The third cannot be turned off**, so the runtime learned it. `output_config.effort` is a level, and the dialect the runtime forwards has the same three levels under the name `reasoning_effort`, so that one is a lookup rather than a guess. A level outside `low`, `medium` and `high` is still refused. **It puts a `system` message inside the messages array**, which the Anthropic dialect documents as carrying user and assistant turns only. It carries the agent-type listing. The runtime carries it in place. Extended thinking being off is the one thing a session on a lane does differently from a session against the vendor, and the launcher prints it every time rather than leaving it to this page. ## Adding a third One entry in `platform/harness/launch.mjs` naming the command, how it is installed, and what it is told. There is no discovery and no generic mode, and that is the design rather than an omission: a launcher that pointed any binary at a base URL would be claiming support for harnesses nobody has run. What costs the day is not the entry. It is finding out where that harness reads its configuration, what it calls a model, and whether it insists on a live model list. The runtime serves no `GET /v1/models`, so a harness that asks for one has to be told not to. Crush asks whenever its model list is empty or discovery is on, which is why its entry declares its models and sets `discover_models` to false. ## Windows The launcher resolves the command on `PATH` itself, honouring `PATHEXT`, so a harness that is not installed is said in the launcher's words with the install line rather than as a spawn error. A harness installed by npm is a `.cmd` shim rather than an executable there, and a shim has to be handed to the command interpreter, which parses the arguments a second time. The launcher builds that command line itself rather than letting a shell do it, so an argument carrying a space or a quote arrives whole. One thing it cannot carry: a `%`. There is no escape for one on a command line as there is inside a batch file, and it is expanded inside quotes as well as outside. An argument containing one is refused by name rather than delivered changed. ## What it refuses | Refusal | Why | |---|---| | A harness it does not know | It starts two, and each further one is an ongoing obligation rather than a default | | Codex | Named with the reason above, because it is the wrong answer to a reasonable question | | Nothing answering at the runtime's URL | With the line that starts one | | A lane the runtime does not serve | With the lanes it does serve | | A lane the runtime carries and cannot serve | With the runtime's own reason, usually a missing key | | One of its own flags given no value | Rather than guessing at one | | An argument a Windows command line would change | Named, rather than passed through wrong | ## Seeing it without starting anything ```bash npm run harness:start -- crush --lane deep-research --print ``` The whole launch: every variable added, every variable withheld by name, the config document as the harness would receive it, and the command line that would have run. Nothing is started and no harness needs to be installed. --- <!-- https://vitna.ai/docs/runtime/installing.md --> # Installing the runtime Getting from nothing to a served lane, on your laptop, your box, or your own cloud account. Everything below is the version where you hold the key: the published package, a signed policy and the key that signed it. To run it with no policy at all, serving every request as the model it names, start [Relay](relay.md) instead, which is the same package. ## The two things it needs, and where they come from A signed policy and the key that signed it. Both come from `/console/deployment`, signed in to your organisation: 1. An owner presses **Ask for a signed policy**. The signer runs every five minutes, so the policy is ordinarily signed within about six minutes of the ask; until then the page says the ask is waiting, and says so again if it has waited longer. Where no key is registered to sign policies yet, `GET /api/v1/keys` answers `"policyKey": null`, nothing is signed, and the page says so and offers no ask until one is. 2. Beside the signed policy, **Download policy.json** saves the file, exactly as it was signed. 3. The page prints the two lines to give the runtime, with the key that signed that file: ```bash VITNA_POLICY=/etc/vitna/policy.json VITNA_PUBLIC_KEY=MCowBQYDK2VwAyEA... ``` Pin the key the page prints beside the file, not the key over the daily roots on `/witness`, which signs the record and not your policy. Every key and what it signs is at [`GET /api/v1/keys`](../api/v1/verification/keys.md). A CI job with no session collects the same file with an org key from [`/api/v1/artifacts`](../api/v1/engine/artifacts.md). ## From npm ```bash npm install -g vitna-runtime ``` Or without installing anything: ```bash npx vitna-runtime ``` A global install puts seven commands on the path: `vitna-runtime`, which serves signed lanes; `vitna-relay`, which serves with [no policy](relay.md); `vitna-route`, which handles [per-request model and effort routing](route.md); `vitna-runtime-prove`, which checks one real serve against its ledger row; `vitna-runtime-lock`, which verifies a signed policy and lockfile in CI; `vitna-runtime-profile`, which reads the [ledger](ledger.md) and prints [what your traffic consumes](profile.md) reaching no network at all; and `vitna-runtime-push`, which [sends that ledger's closed days back](pushing.md) when, and only when, you run it. Started this way the runtime listens on `127.0.0.1` and asks callers for no key. To reach it from another machine, set `VITNA_HOST` and `VITNA_INBOUND_KEY` together: [the front door](serving.md#the-front-door) covers both. The package has **no dependencies**. It is plain `.mjs` under plain `node`, with no build step, because a gateway a customer trusts with every prompt is one a security review should be able to read in an afternoon. A test asserts that property rather than claiming it: the published file list is checked against the closure of every import from the entry points, so a dependency cannot arrive without failing a build. ## From the container ```bash docker run --rm -p 8787:8787 --stop-timeout 35 \ -e VITNA_POLICY=/etc/vitna/policy.json \ -e VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ -e VITNA_INBOUND_KEY="$VITNA_INBOUND_KEY" \ -e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \ -v "$PWD/policy.json:/etc/vitna/policy.json:ro" \ ghcr.io/convexityos/vitna-runtime:latest ``` The image is built by packing the same whitelist npm publishes, so what it holds is what the tarball holds. Nothing of the record or the site is in it. **The image refuses to start without `VITNA_INBOUND_KEY`.** A container port is reached from outside the container, so the image listens on every interface, and a runtime reachable from a network with no key would spend your provider keys for whoever found it. Generate one (`openssl rand -hex 32`), and send it as your SDK's API key: every `/v1` request carries it as `authorization: Bearer <key>`. A health check needs none. **`--stop-timeout 35` is what lets a restart drain.** `docker stop` sends `SIGTERM` and then `SIGKILL` ten seconds later unless told otherwise, and the runtime drains for up to 30 seconds (see [restarting](serving.md#restarting)). Compose spells it `stop_grace_period: 35s`, and Kubernetes `terminationGracePeriodSeconds: 35`. ## On your own cloud, which is Deploy Deploy is the runtime on infrastructure you own, provisioned by tooling Vitna publishes. Your keys stay in your account and Vitna is not in the request path, exactly as when you run the process yourself. It is not a hosted gateway, and [the modes table](overview.md) is exact about the difference. Save this beside your policy as `fly.toml` and change the app name. It is the recipe Vitna keeps working, printed whole here because nothing else needs to be downloaded: ```toml app = "my-vitna" primary_region = "iad" kill_timeout = 35 [build] image = "ghcr.io/convexityos/vitna-runtime:latest" [env] VITNA_PORT = "8787" VITNA_HOST = "0.0.0.0" # VITNA_LEDGER = "/data/ledger.jsonl" [http_service] internal_port = 8787 force_https = true auto_stop_machines = "suspend" auto_start_machines = true min_machines_running = 1 [[http_service.checks]] grace_period = "5s" interval = "30s" method = "GET" timeout = "5s" path = "/status" # [[mounts]] # source = "vitna_ledger" # destination = "/data" ``` `kill_timeout` is 35 because on SIGTERM the runtime drains for 30 seconds and then gives the requests it cut 2 more to write their ledger rows; Fly's default of five would cut every stream still open with no row at all. `VITNA_HOST` is every interface because Fly's proxy reaches the machine over its private network, which is what makes `VITNA_INBOUND_KEY` required below. The health check reads `/status` with no header, which answers `{ ok, version }` and nothing else. A ledger that must outlive a restart needs a volume: `fly volumes create vitna_ledger --size 1`, then uncomment `VITNA_LEDGER` and the mount. Then three commands: ```bash fly launch --copy-config --no-deploy --name my-vitna fly secrets set VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ VITNA_POLICY_B64="$(base64 -w0 policy.json)" \ VITNA_INBOUND_KEY="$(openssl rand -hex 32)" \ ANTHROPIC_API_KEY=... fly deploy --image ghcr.io/convexityos/vitna-runtime:latest ``` Then point your SDK's base URL at `https://my-vitna.fly.dev/v1`, give it the `VITNA_INBOUND_KEY` you set as its API key, and name a lane. **An app deployed from this recipe before 0.2.0 has no inbound key**, and because the recipe runs `:latest`, its next `fly deploy` boots a runtime that refuses to start until `fly secrets set VITNA_INBOUND_KEY=...` has run. Its health check needs no change: the recipe's check reads `/status` with no header, which still answers. The image runs anywhere a container runs. Fly is the one path documented end to end; a second platform is documentation rather than code. ## The policy as a secret A machine provisioned from a console has no filesystem to copy a file onto, so the document may also arrive base64 encoded in `VITNA_POLICY_B64`. **This is a delivery, not a fetch.** You put it there, no endpoint served it, and it is verified against the key you pinned by the identical path the file form takes. A policy signed by another key, or edited after signing, refuses to start here exactly as it does on a box. Exactly one of `VITNA_POLICY` and `VITNA_POLICY_B64` may be set. Both is refused at boot rather than ranked, because a precedence rule is a deploy that quietly serves the wrong document the day somebody forgets which one wins. A reload re-reads whichever one is in play. A secret that has gone missing or gone malformed refuses the reload and the running policy stands, which is the same fail-static rule a half-written file gets. ## Whether the readings come back, which is a decision to take here The runtime writes its [ledger](ledger.md) locally whatever you decide. Sending those readings to Vitna is a separate command, and nothing schedules it until you do: ```bash vitna-runtime-push ``` **That is a property of the software rather than a setting**, and [Pushing the ledger](pushing.md) is where it is argued. The serving process holds no code path that reaches vitna.ai, with telemetry or without, so the claim that the runtime never calls home survives push existing. It stays true whichever way you answer this. This page did not used to ask, which turned "off by default" into "never put to anybody". Those are different, and only the first was ever the commitment. **What answering yes buys.** The register can then state what it told a workload to run *and* what the workload was observed running, so a decision's prediction stops being an intent and becomes something [checkable against what actually served](../routing/outcome.md). It is the same reading [`aperture:usage:import`](../routing/usage-import.md) takes from a file you export by hand, arriving with less typing. **What it costs.** One org-scoped push key, and a timer you install. Vitna mints the key for your organisation and it is shown once: ask for one through the [contact form](https://vitna.ai/start#contact). The record keeps only its sha256, and [/console/settings](https://vitna.ai/console/settings) lists the keys your organisation holds. It goes in `VITNA_PUSH_KEY` on the box that will push, and authenticates that org's workloads only. **What goes, and how to see it before deciding.** Served units per lane per closed UTC day, mapped to workloads through the policy. No prompt, no response, no content of any kind. You do not have to take that from this page: ```bash vitna-runtime-push --dry-run ``` shapes and prints the exact payloads, holding no key and reaching no network. Read them, then answer. **What answering no costs**, stated so the choice is informed rather than cautious: the register holds what it decided and never learns what happened, so the outcome comparison above has nothing to read for that workload, and the evidence behind the next decision stays whatever somebody typed in by hand. ## Licence The runtime is Apache-2.0. The record, the collector and the emitting half of the engine are not published and are not covered by that grant. The published paths include the policy parser and its canonical form, which is deliberate: a policy signed with your own key verifies against whatever key your runtime pinned, so the executor runs with no Vitna in the picture at all. That is the architectural claim written as a licence term rather than a promise. --- <!-- https://vitna.ai/docs/runtime/ledger.md --> # The ledger One line per request, appended to the file `VITNA_LEDGER` names. It is yours, on your disk, and it moves only when you move it. ## A row ```json { "at": "2026-08-24T09:14:02.118Z", "model": "claude-opus-4-8", "units": 1, "lane": "deep-research", "sku": "anthropic/claude-opus-4-8", "provider": "anthropic", "via": "direct", "decision": "154", "policy": "ecb93b2a5c50...", "policyAgeSeconds": 3120, "routed": true, "serve": "primary", "shadowRule": null, "shadowSku": null, "sellers": null, "cohort": null, "outcome": "served", "status": 200, "latencyMs": 2210, "inputTokens": 812, "outputTokens": 140, "error": null } ``` | Field | Meaning | |---|---| | `model` | The name the provider was asked for, which differs by route | | `units` | One served request is one unit. A refused or failed one is zero, and so is one the [response cache](cache.md) answered, since no provider served it. | | `lane`, `sku`, `decision`, `policy` | What makes the request attributable | | `via` | `direct`, or the substrate | | `routed` | False for a model name passed through untouched | | `serve` | `primary`, `fallback:<n>` for the [chain](../routing/fallbacks.md) entry that answered, or `rule:<n>` for a routed [rule](../routing/rules.md) | | `shadowRule`, `shadowSku` | Where a shadow rule matched: which rule, and what it would have served instead | | `cohort` | The 0 to 99 bucket a [traffic share](../routing/rules.md) was decided by, and null where no share rule looked. A canary nobody can audit is a canary nobody should trust: this is why that rule matched this request | | `sellers` | The hosts OpenRouter was allowed to serve this attempt from, in the policy's order, and null where the lane does not rotate. What was permitted, not which host answered | | `outcome` | `served`, `refused`, `failed`, `timed_out`, `caller_left` or `drained` | | `status` | What the provider answered, where it answered: null for a provider that could not be reached or never answered | | `latencyMs` | How long the attempt took. For `timed_out` before anything reached the caller, the budget that ran out; for `caller_left`, the moment the caller hung up; for `drained`, the moment a restart's drain window closed | | `inputTokens`, `outputTokens` | What the provider reported, or null. `inputTokens` is every input token: Anthropic reports the part it read from or wrote to its prompt cache beside `input_tokens` rather than inside it, and the runtime adds the three, as OpenAI's `prompt_tokens` already includes its cached part | | `cacheReadTokens`, `cacheWriteTokens` | The parts of `inputTokens` the provider read from its prompt cache and wrote to it, which it bills at their own rates, and null where it reported none. OpenAI reports the first; Anthropic reports both, and so does OpenRouter for the models it prices a cache write for | | `cacheWrite5mTokens`, `cacheWrite1hTokens` | The parts of `cacheWriteTokens` the provider kept for five minutes and for an hour, which it bills at different rates, as it reported them in `cache_creation`. Null where it did not report the split, and never derived from the total: a write with no split says nothing about its retention. Anthropic reports it; on a stream, each part is the last count it sent. OpenRouter reports a write as one count, so a row it served carries null in both | | `cache_hit` | `true` on a row the [response cache](cache.md) answered, with `serve` and `via` both `cache`, no units and every token count null. Null otherwise | | `secretsLifted`, `secretsRestored` | How many secrets [redaction](redaction.md) took out of the request, and how many surrogates it put back into the answer. Null where redaction did not apply to the request. Counts only: no secret is ever on a row | | `requestId` | The id the caller sent for the request in `x-vitna-request-id`, on every row that request wrote, fallback attempts included, and null where it sent none. See [the request](serving.md#the-request) | | `agentId`, `runId` | The agent or routine the caller says the request was made for, from `x-vitna-agent`, and which run of it, from `x-vitna-run`, on every row that request wrote, fallback attempts included, and null where it sent none. What the caller says, recorded as sent and checked against nothing. See [the request](serving.md#the-request) | | `listPriceUsd`, `chargedUsd` | What the request cost at the list price the record showed, and what the provider or router reported charging. This runtime writes null for both today; a writer that knows them fills them, and neither is ever estimated | | `hardware_cost_usd`, `cloud_market_usd`, `net_saved_usd`, `airgap`, `precision_tier` | What serving the request cost in hardware, a cloud price for the same request, the saving between the two, whether anything left the machine, and the quantization the model ran at. A writer fills one only with what it measured. This runtime measures none of them, so it writes null in all five, on every route | ## The three rules it inherits **Append only.** One write per row. There is no code in the runtime that can edit a line it has already written, and no compaction. **Nothing is estimated.** A provider that reported no token counts gets nulls, never a plausible figure. That is the synthetic fallback the record refuses in every other data path, at request scale. It covers cost and egress as well. Until 2026-09-27 `main` filled in the five cost and egress columns in the table above on any row a lane whose provider is `local` served: a hardware cost and a cloud price worked out of the token counts at fixed rates, the saving between them, `airgap: true` and a precision tier, whatever had actually answered. None of it was measured, so all five are null now. No published version of the runtime wrote them. **Three outcomes are the runtime stopping a request, not a provider failing one.** `timed_out` is a provider that did not answer within its [budget](serving.md#when-a-provider-hangs); before anything reached the caller the lane's chain is walked after it, and after, the response is ended. `caller_left` is a caller who hung up, after which the upstream request is aborted and nothing else is tried. `drained` is a request still under way when a [restart](serving.md#restarting) ran out of time to wait for it: the runtime cut it, told a caller still waiting 503, and wrote this row before it exited. **All three are requests the provider still bills** for whatever it generated before the abort reached it, so all carry zero units and none is free. The runtime writes none of them before 0.2.0. **A failure is a row, and it is worth nothing.** A refused or failed request is recorded with its outcome, because a ledger holding only successes makes "ran little" and "failed often" the same shape. It carries zero units, because a request nobody served is not a unit of work and counting it would put it in an import's denominator. A window of nothing but refusals therefore totals zero, and the import says the file says nothing about what ran. ## One request can be more than one row A request a fallback answered is two rows: the primary's failure, worth no units, and the fallback's serve, worth one. Totals stay right, and the ledger still says the primary was down. That is also what tells the two stories of a split window apart. A window where a minority share sits on `fallback:1` rows is a lane with a healthy failover. The same split sitting on `primary` rows is a deployment that has drifted onto a model nobody decided, which is a different problem with a different fix. ## Reading it back The file is the shape [`aperture:usage:import`](../routing/usage-import.md) already reads, so it is imported by the command that already existed: ```bash npm run aperture:usage:import -- \ --workload convexity/deep-research --file /var/lib/vitna/ledger.jsonl \ --window 2026-08-01..2026-08-31 --source-ref "runtime ledger, box 1" ``` `.jsonl` is recognised by extension. The reader takes the same two fields it takes from a CSV, `model` and `units`, and ignores the rest of the row. ## The run becomes evidence The import above says what ran. A second reader takes the same file and writes what the run measured, as a row on the workload in the register: ```bash npm run aperture:evidence:from-run -- --workload convexity/deep-research --file /var/lib/vitna/ledger.jsonl --label "Harness run, 10 Sep" ``` One measurement per sku the run served, carrying `latencyP90Ms` out of the `latencyMs` column and `n` out of the units, both naming the run they came from. It is the same append-only path [`aperture:evidence:add`](../cli/aperture/evidence.md) writes through and it passes the same refusals. What it does not write is a success rate. A `failed` row is a request nobody served, and the share of requests that came back is not the quality of the answers that did: Vitna measures what this file observed and grades nothing. ## Why this is better than an export The [first real import](../routing/usage-import.md), on 2026-08-24, taught three things, and all three are properties of reading somebody's export after the fact: - **A surface is not an intent.** That export could only be scoped by where a call came from, so a lane's window carried 56 percent of a model that lane never ran. A ledger is lane-exact because the caller named the lane. - **The register could not see volume.** A dormant lane and one carrying thirty thousand calls read identically. A row count is a volume. - **Most of a window resolved by the weaker of two routes.** The ledger records the name the policy said to send, so there is nothing to map back. Nothing pushes this anywhere on its own. [`vitna-runtime-push`](pushing.md) sends closed days to the record when an operator runs or schedules it, opt-in and off by default, and the file path stays the default path. ## What it can tell you without leaving the machine `inputTokens` and `outputTokens` per request per lane are a measurement nobody hands you: what one call on this lane actually consumes. [`runtime:profile`](profile.md) totals them into a [unit profile](../cli/aperture/workloads.md) per lane, with the window and the sample size beside each figure, reading this file and reaching no network. --- <!-- https://vitna.ai/docs/runtime/local.md --> # A lane on your own machine An operator note: one command from a workload in the register to a runtime answering on the same machine, run from a checkout of Vitna's repository, which is private. It is for Vitna's own operators. Outside Vitna, the runtime is installed from npm with a policy the console signs, which is [Installing the runtime](installing.md). ## Outside Vitna, start here instead Every `npm run` command further down this page needs a checkout of the repository and a register you can read straight from its database, and neither is published. The same runtime from npm takes a signed policy and its key from `/console/deployment`: ```bash npm install -g vitna-runtime ``` [Installing the runtime](installing.md) goes from there to a served lane, and [Serving a lane](serving.md) covers every variable. To try a runtime with no policy and nothing to sign, `vitna-relay` is in the same package: [Relay](relay.md). ## Operator note: one command from the register So an operator can send a request through a lane and read the row it wrote, from a checkout: ```bash npm run runtime:local ``` It reads your register, emits the same signed policy [`aperture:policy`](../routing/policy.md) emits, and starts the runtime on it. Nothing about the runtime is relaxed to make that work, and the section at the bottom is exact about what is different. ## Before you start Two things, and you probably have both. **A register you can read.** `DATABASE_URL` points at it. This command reads the record and writes nothing to it, and the runtime it starts never touches the database at all. **A key for one provider.** `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY`, whichever reaches the model your lane is decided on. It is yours, it is read out of your environment, and it goes to that provider and nowhere else. ```bash export DATABASE_URL=postgresql://localhost/vitna export OPENAI_API_KEY=sk-... ``` ```powershell $env:DATABASE_URL = "postgresql://localhost/vitna" $env:OPENAI_API_KEY = "sk-..." ``` ## The sequence **1. Give a workload a lane name.** A lane is a workload carrying an alias a caller can put in a request. A workload without one is decided and not routable, which is the ordinary state of most of them. ```bash npm run aperture:workloads npm run aperture:workload:edit -- <workload-id> --policy-alias deep-research ``` The workload also needs an accepted decision behind it, which is what a lane resolves to. If it has none, the command below says so and names the workload. **2. Start it.** ```bash npm run runtime:local ``` It prints the lanes it emitted, the decision behind each one, and where the policy and the ledger are: ``` policy /home/you/vitna/.vitna/policy.json, 2 lanes, digest 9f2c... deep-research claude-opus-4-8 anthropic decision 154 classification-and-lookups gpt-5.6-luna openai decision 151 ledger /home/you/vitna/.vitna/ledger.jsonl, one line per request key generated for this run, held in memory, gone when this exits Point an OpenAI client at http://127.0.0.1:8787/v1 and send a lane as its model: vitna/deep-research vitna/classification-and-lookups vitna-runtime 0.2.0 vitna-runtime: policy 9f2c... for convexity, generated 2026-09-10T09:00:00.000Z vitna-runtime: 2 lanes servable: deep-research -> claude-opus-4-8 (direct, completion budget 540000ms), classification-and-lookups -> gpt-5.6-luna (direct, completion budget 540000ms) vitna-runtime: OPENAI_API_KEY is set, and bills your openai account directly for the first attempt of classification-and-lookups, and every model name sent without a lane vitna-runtime: ANTHROPIC_API_KEY is set, and bills your anthropic account directly for the first attempt of deep-research vitna-runtime: upstream budgets: 540000ms for a completion (VITNA_UPSTREAM_TIMEOUT_MS), 300000ms to a stream's first byte (VITNA_UPSTREAM_FIRST_BYTE_MS), 300000ms of silence inside a stream (VITNA_UPSTREAM_IDLE_MS) vitna-runtime: listening on 127.0.0.1:8787, loopback only, so no key is asked for. POST /v1/chat/completions, POST /v1/messages, GET /status, GET /healthz. ``` With more than one organisation in your register, `--org` says which: `npm run runtime:local -- --org convexity`. With one, it takes it and says so. `--port` and `--dir` move the port and the directory. **3. Send a request**, from another shell. ```bash curl 127.0.0.1:8787/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model":"vitna/deep-research","messages":[{"role":"user","content":"hello"}]}' ``` ```powershell $body = '{"model":"vitna/deep-research","messages":[{"role":"user","content":"hello"}]}' Invoke-RestMethod -Method Post -ContentType application/json -Body $body ` -Uri http://127.0.0.1:8787/v1/chat/completions ``` Any OpenAI-compatible SDK works the same way: point its base URL at `http://127.0.0.1:8787/v1` and send `vitna/<lane>` where it wants a model name. The address rather than `localhost` on purpose: the runtime listens on IPv4 loopback only, and a client that resolves `localhost` to `::1` first and does not fall back would find nothing there. A name without the prefix is passed through untouched, so you can point a service that is already running at this and move one lane at a time. **4. Read what it recorded.** For a cloud lane, the proof command sends one short, fixed prompt and checks that the runtime appended exactly one served row under its loaded policy and the decision ID carried by that policy. Run it from a second shell on the same machine: ```powershell npm run runtime:prove -- --lane deep-research ``` The package exposes the same check as `vitna-runtime-prove`. It prints the provider, sku, decision, policy digest and reported token counts, without printing the prompt or answer. It refuses a local Anchor lane, a cache hit, a fallback, overlapping traffic or a missing ledger. This check confirms the serving path; the decision's separate price evidence is in its decision pack. A pack is signed only when exported with a signing key. ```bash cat .vitna/ledger.jsonl ``` ```powershell Get-Content .vitna\ledger.jsonl ``` One line per request: ```json {"at":"2026-09-10T09:01:04.221Z","model":"claude-opus-4-8","units":1, "lane":"deep-research","sku":"anthropic/claude-opus-4-8","provider":"anthropic", "via":"direct","decision":"154","policy":"9f2c...","routed":true, "outcome":"served","status":200,"latencyMs":1840, "inputTokens":11,"outputTokens":3} ``` `decision` is the point of the whole exercise. It is the accepted decision that put this model on this lane, and it is on the row because a lane cannot enter a policy without one. [The ledger](ledger.md) is what each field means and how the file reads back into the register. ## What just happened The command did four things, and the third is the one worth understanding. 1. It read your register and assembled a policy, refusing to write a partial one for the reason [`aperture:policy`](../routing/policy.md) refuses: a policy missing a lane leaves a runtime resolving that alias against whatever it loaded last. 2. It generated an Ed25519 keypair, used it to sign the policy, and wrote the file. 3. It started the runtime in a second process, handing it the path and the public half of that key. 4. It held the key nowhere. The private half was unreachable the moment the policy was written, and neither half is on your disk, in your environment or in the output. **The refusal you did not have to satisfy is still there.** The runtime loaded a signed document, checked the signature against a key it was pinned by something other than the file, and would have refused to start if it did not verify. That is the same check it does on a box, running the same code, and [Serving a lane](serving.md) is why it is worth anything. What changed is who holds the key for the ten minutes you are trying this: nobody, for one run. That is the arrangement the licence already describes. The policy parser and its canonical form are published under Apache-2.0 precisely so that a policy signed with your own key verifies against whatever key your runtime pinned, with no Vitna in the picture at all. This command is that sentence with nobody to type it. ## What is different from an install Four things, and none of them is a weakened check. **The signature is yours, not Vitna's.** The record never saw this key, so it publishes nothing to compare it against. Verifying the file offline works and says so: ```bash npm run aperture:policy -- --verify .vitna/policy.json ``` It reports the lanes, recomputes the digest and checks the signature, then says the key was not checked, because a signature proves only that whoever holds a key signed with it. Run it with `DATABASE_URL` set and it also compares that key against the ones the record has registered, so a record that has registered one says this file did not come from Vitna. It did not. You signed it. **Restart it to pick up a change.** A second run signs with a second key, so a policy it writes is refused by the first run's reload. That is [fail static](serving.md) doing its job rather than a fault. Stop the command and start it again. **Everything lands in `.vitna/`**, which is gitignored. The policy is signed by a key that no longer exists, and the ledger is a record of requests made from your machine. **Nothing is pushed anywhere.** That is not a setting here either. The serving process holds no code path that reaches vitna.ai, and [sending closed days to the record](pushing.md) is a separate command you schedule if you decide to. ## Operator note: benchmarking local throughput From a checkout too, you can run `scripts/bench-cortex.mjs`, a synthetic memory-hierarchy and hardware-amortization model (not a live GPU or NVMe timing benchmark): ```bash npm run aperture:anchor:bench ``` For real local inference timing and NVMe direct-I/O measurements, run the `vitna-anchor` binary directly (`generate --timing` and `read-experts` in `convexityos/vitna-anchor`). ## When you want it for real [Installing the runtime](installing.md) is the same runtime with the key held the way a production key is held: generated once, its public half published by the record and pinned by you out of band, its private half on the signer and nowhere else. Nothing you learned above stops being true. You stop being the one who signs. --- <!-- https://vitna.ai/docs/runtime/profile.md --> # Measuring your unit profile A unit profile is how many tokens one call on a lane actually consumes. It is the bridge between a published price and a cost per unit: input tokens per unit, multiplied by a price per million input tokens, is what one unit of that work costs at list. Your provider does not give you this number. The runtime already has it, because it wrote one line per request while it was serving. ```bash VITNA_LEDGER=/var/lib/vitna/ledger.jsonl vitna-runtime-profile ``` `vitna-runtime-profile` is on the path of any [install](installing.md) from 0.2.0; inside Vitna, from a checkout of its private repository, the same file runs as `npm run runtime:profile`. From the container it is the same binary behind a different entrypoint, reading the ledger volume you already mount: ```bash docker run --rm --entrypoint vitna-runtime-profile \ -e VITNA_LEDGER=/var/lib/vitna/ledger.jsonl \ -v "$PWD/ledger.jsonl:/var/lib/vitna/ledger.jsonl:ro" \ ghcr.io/convexityos/vitna-runtime:latest ``` ## What it prints ``` runtime-profile: /var/lib/vitna/ledger.jsonl 4812 of 4880 rows counted. This read one file and reached no network. lane deep-research window 2026-09-01T00:12:44.108Z .. 2026-09-09T23:41:02.550Z served 3104 units, and 11 failed, worth no units input 8214.62 tokens per unit, from 3104 served units (25498183 tokens) output 1120.4 tokens per unit, from 3104 served units (3477721 tokens) as flags --profile inputTokensPerUnit=8214.62 --profile outputTokensPerUnit=1120.4 ``` One block per lane. The window is the first and last served request the figure rests on, read off the rows themselves, so a lane that was quiet for a fortnight says so in its own dates. The token totals travel beside the rounded figure, so anybody holding this output can recompute it exactly. ## Nothing leaves the machine No account, no key, no database, no network. It reads the file your runtime writes and prints to your terminal, and it is worth running if you never speak to Vitna again. [Pushing the ledger](pushing.md) is a different command, opt-in and off by default, and this one neither calls it nor needs it to have been run. ## Nothing is estimated The rule the [ledger](ledger.md) keeps at row scale is kept here at aggregate scale. A provider that reported no token counts yields nulls: ``` lane classification-and-lookups window 2026-09-02T12:00:00.000Z .. 2026-09-09T18:22:11.000Z served 902 units input not measured: the provider reported no inputTokens on any served unit here output not measured: the provider reported no outputTokens on any served unit here as flags none, because nothing here was measured; a profile is not entered from an absence ``` Not zero, which would price the lane as free. Not a plausible figure borrowed from a sku that does report. **The denominator is the units that reported, not the units that ran.** A window where half the serves carried a count and half did not would otherwise measure at half its true rate, so the two numbers are kept apart and both are printed: ``` input 1000 tokens per unit, from 2 of 4 served units, the rest reporting no count (2000 tokens) ``` **A refusal and a failure are worth no units.** They are counted beside the profile, because a lane that failed often and a lane that ran little are the same shape otherwise, and they do not widen the window, because a profile measured over requests that produced nothing would describe a busier lane than you have. ## What it does not separate | Not separated | Why, and what to do about it | |---|---| | Cached input | Neither dialect reports the cached share of a prompt, so `inputTokensPerUnit` is every input token. A lane whose provider bills cache reads at their own rate is not fully described by it, and `cachedInputTokensPerUnit` is emitted not at all rather than as a zero | | Units that are not calls | One unit here is one served request. A workload whose unit is a document that takes three calls has a profile three times this one, and no ledger row knows that | | Two tokenizers | A lane that failed over, or that carries a promoted [rule](../routing/rules.md), spent its window across more than one sku, and the same prompt is a different number of tokens at each. The per-sku split prints under any lane that served more than one | ## Passthrough is measured and is not a lane A model name sent without the `vitna/` prefix is [passed through untouched](serving.md) and marked unrouted. It is still your traffic, so it is measured and grouped by the model that was asked for, under a heading that says it is not a lane. No `--profile` flags are offered for it: nothing in the record chose it, so there is no workload it could be a profile of. ## Narrowing it | Flag | What it does | |---|---| | `--file <path>` | The ledger to read, if you would rather not set `VITNA_LEDGER` | | `--window <a..b>` | Two calendar days, inclusive, spelled as [`aperture:usage:import`](../routing/usage-import.md) spells one | | `--lane <alias>` | One lane. Every other row is reported as another lane's rather than dropped | | `--json` | The same measurement as JSON on stdout, to keep rather than to read | ```bash vitna-runtime-profile --window 2026-09-01..2026-09-09 --lane deep-research ``` Whatever a run leaves out is printed per reason: rows outside the window, rows belonging to another lane, rows carrying no usable timestamp, and a truncated final line, which is the ordinary shape of an append-only file being read while it is written. ## Entering one is a separate act The `as flags` line is the fragment [`aperture:workload:edit`](../cli/aperture/workloads.md) takes. Typing it is somebody's decision, made at a prompt, against a register that is not on this machine. This command prints it and does nothing with it, which is the same division [pushing](pushing.md) keeps: the measurement is yours, and moving it is an act with a person behind it. Teams already running LiteLLM or OpenRouter can pair `vitna-runtime-profile` with the [LiteLLM cost map](../api/v1/prices/litellm-cost-map.md), [CI evaluation evidence](../api/v1/engine/org-evidence.md) (Promptfoo, Braintrust, Inspect AI), and [gateway adapters](../routing/adapters.md) without replacing their production proxy first: see the [three-minute gateway quickstart](../quickstart.md#three-minute-gateway-quickstart-litellm-and-openrouter). --- <!-- https://vitna.ai/docs/runtime/pushing.md --> # Pushing the ledger Sending a runtime's closed days to Vitna, deliberately, as a command of its own. ```bash VITNA_LEDGER=/var/lib/vitna/ledger.jsonl \ VITNA_POLICY=/etc/vitna/policy.json \ VITNA_PUBLIC_KEY=... \ VITNA_PUSH_KEY=vitna_push_... \ vitna-runtime-push ``` `--dry-run` shapes and prints the payloads with no key and no network. `--window 2026-08-24` pushes one day. `vitna-runtime-push` is on the path of any [install](installing.md) from 0.2.0. Inside Vitna, from a checkout of its private repository, the same file runs as a script, with `--` before its flags: ```bash npm run runtime:push -- --dry-run ``` From the container it is the same binary behind a different entrypoint, reading the ledger volume the runtime writes to and the policy it serves: ```bash docker run --rm --entrypoint vitna-runtime-push \ -e VITNA_LEDGER=/var/lib/vitna/ledger.jsonl \ -e VITNA_POLICY=/etc/vitna/policy.json \ -e VITNA_PUBLIC_KEY="$(cat policy-key.pub)" \ -e VITNA_PUSH_KEY="$VITNA_PUSH_KEY" \ -v "$PWD/ledger.jsonl:/var/lib/vitna/ledger.jsonl:ro" \ -v "$PWD/policy.json:/etc/vitna/policy.json:ro" \ ghcr.io/convexityos/vitna-runtime:latest ``` The key comes from Vitna, minted for your organisation and shown once: ask through the [contact form](https://vitna.ai/start#contact), and [/console/settings](https://vitna.ai/console/settings) lists the ones you hold. ## Why this is not part of the serving process The runtime's claim is that it never calls vitna.ai, and that claim survives push being enabled because push is not the runtime: it is this command, reading the same ledger file you could have handed to [`aperture:usage:import`](../routing/usage-import.md) by hand, run when you or a timer you installed runs it. The serving process contains no code path that reaches vitna.ai, with telemetry or without, and "off by default, forever" means precisely that nothing schedules this until a person does. The file path stays the default path. Pushing is the same reading arriving with less typing, not a different kind of reading. ## What a run does - Loads and verifies the policy against the pinned key first, because a push attributes readings to a policy digest, and attributing them to a document nobody signed would launder exactly the provenance the field carries. - Aggregates the ledger's **served** units per lane per **closed UTC day**, maps each lane to its workload through the policy, and posts one payload per day to `POST /api/v1/usage`. - Prints what it left out, per reason, because a silent drop is a share that is quietly wrong: today's rows (still open), unrouted passthroughs (no lane to be a reading of), refusals and failures (worth no units), and rows naming a lane the loaded policy no longer carries. ## What it holds, and does not No state. The server keeps one reading per workload per window, so a re-push of a day it already holds answers `alreadyStood` and costs one request; a pusher that tracked what it had sent would be a second copy of that answer that could drift from it. No adoption. What lands are readings; whether a workload's incumbent moves stays an operator's act, on the same file, through the import command. ## What a push carries Volume, and the operational figures beside it. Every one is counted exactly, on your machine, from rows the runtime already had, before anything leaves. | Figure | What it is | |---|---| | `units`, `windowUnits` | Served requests for this sku, and the workload's total that day. Serves only, which is what a share of a window means | | `refused`, `failed` | Attempts on this sku that produced no unit. A lane that fails often and a lane that runs little were the same shape in the record before these | | `primaryServes`, `fallbackServes`, `ruleServes` | How those units were reached. A lane serving 95 percent from its primary is a different lane from one serving all of it there | | `inputTokens`, `outputTokens` | Summed as the provider reported them. A provider that reported none contributes none, and the sum stays null | | `latencyP50Ms`, `latencyP95Ms` | The middle and the tail, exact from the day's own rows. Nothing is sampled: the runtime holds all of them, and an estimator would put a figure in a record that carries none | **No prompts and no responses, ever.** The ledger holds none, so a push cannot carry one. **Null is not zero.** A figure the runtime could not count is absent, and the record reads that as not reported. A zero would be a claim. **No money.** The runtime holds no prices, and a runtime that pushed spend would be freezing a price into telemetry. `aperture:reconcile` prices a window at the tape's own figures on each day instead. **No quality.** That is a measurement with a sample size and a source reference, and it arrives through the door evidence already has. A refusal that happened before any sku was chosen names none, so there is nothing in the register for it to be a reading of. Those are counted in the run's own summary and pushed nowhere. ## Scheduling it A daily timer after midnight UTC pushes yesterday once it has closed. On a systemd box, a oneshot service running `vitna-runtime-push` with the four variables, and a timer with `OnCalendar=*-*-* 00:20:00 UTC`, is the whole of it. A missed run costs nothing: the next one pushes every closed day the record does not already hold. **On Fly, the recipe as it stands installs no push**, because a Fly app has no timer and nothing in the recipe's [`fly.toml`](installing.md) schedules one. Two things have to be true first. The ledger must live on a volume, or it dies with the machine: the recipe's `[[mounts]]` block and `VITNA_LEDGER` line are commented out, and uncommenting both is the step. And the policy must be readable as a file by the pusher, which reads `VITNA_POLICY` and not the encoded secret. Then push by hand, on the machine that holds the volume: ```bash fly ssh console -C "vitna-runtime-push --dry-run" ``` with `VITNA_POLICY` and `VITNA_PUSH_KEY` set for that command. A scheduled Fly machine mounting the same volume is the other way to run it on a timer; this page does not document one yet. --- <!-- https://vitna.ai/docs/runtime/redaction.md --> # Redaction Known credential formats lifted out of a request before it leaves your machine, and put back into the answer, streams included. Off unless you turn it on, for the lanes you name or for traffic that names a model, except in [Relay](relay.md#redaction-is-on), which turns it on for you. ```bash VITNA_REDACT_LANES=deep-research,agent vitna-runtime # these lanes VITNA_REDACT_LANES='*' vitna-runtime # every lane VITNA_REDACT_PASSTHROUGH=1 vitna-runtime # every request naming a model vitna-relay # the same, on unless you pass --no-redact ``` It exists for the case a coding agent makes common: the agent reads a `.env` file, a config or a shell history into its context, and the next request carries your production keys to a model provider. With redaction on, the provider sees a stand-in for each, and the file the agent writes back gets the real value. ## What it lifts Formats with a shape of their own, and nothing found by guessing: | Kind | What is recognised | |---|---| | Provider keys | OpenAI (`sk-proj-`, `sk-svcacct-`, `sk-admin-` and the older `sk-`), Anthropic (`sk-ant-api03-` and the admin form), OpenRouter (`sk-or-v1-`) | | Cloud and service tokens | AWS access key ids (`AKIA`, `ASIA`), GitHub tokens (`ghp_` and its siblings, `github_pat_`), Slack (`xoxb-` and its siblings), Stripe (`sk_live_`, `sk_test_`, `rk_`), Google API keys (`AIza`) | | Signed tokens | JWTs, the three dot-separated parts each starting as base64 JSON does | | Private keys | A PEM `PRIVATE KEY` block, header to footer | | Connection strings | The password in `scheme://user:password@host`, and nothing else of the URI | | Assignments | The value of a secret-named assignment: `STRIPE_KEY=...` on a line of its own, as an env file writes it, or a quoted value given to a key named like `api_key`, `secret`, `token` or `password` in code, JSON or YAML | What it leaves alone, on purpose: - **Code that only mentions a secret.** `token = await getToken()`, `password: string`, `max_tokens: 1024`, `process.env.OPENAI_API_KEY` and `${GITHUB_TOKEN}` are not secrets, and a rule loose enough to catch them would rewrite code the model needs to read. An assignment's value is lifted only when it is quoted, or on an env-file line of its own, and at least eight characters long. - **Anything found by entropy alone, and personal data.** A hash, a UUID or a base64 blob looks like a secret to a scanner, and rewriting one hands the model a commit it cannot find. Personal data waits for a constraint vocabulary in the policy, which the runtime's plan keeps out of scope until then. - **Images and documents, tool definitions, and thinking.** A thinking block carries a signature over its text, so it goes as it was written, in both directions. A secret in a format not in the table leaves as it was. Redaction reduces what a provider sees; it is not a promise that nothing sensitive leaves. ## The stand-ins Each secret is replaced by a surrogate of the same kind: the same prefix, the same length, and a random letter or digit wherever the secret had one, with its punctuation kept. `sk-proj-` stays `sk-proj-`, so a model reads it as the key it is and writes it into code as it would the real one. A secret shorter than twelve letters and digits gets a longer surrogate, since a short one could turn up in an answer by chance. The same secret gets the same surrogate for as long as the process runs. So every turn of a conversation sends the same text, and a provider's prompt cache keeps hitting; after a restart the surrogates are new, and the first turn misses the cache once. A surrogate that comes back in a later request, because the client kept one somewhere, is left as it is. The pairs live in the serving process's memory and nowhere else: not on disk, not in the ledger, not on `/status`, not in a log. ## Putting them back Wherever an answer carries a surrogate, the runtime writes the secret in its place before the client sees it: in text, and inside a tool call's arguments or a tool's input, where it is escaped as JSON needs. A stream is handled the same way, per choice and per content block. When a piece ends with what could be the start of a surrogate, that tail is held back until the next piece says whether it is one, so a surrogate the provider sent across two chunks, or two events, is restored whole. Whatever is held is released when its block or choice ends, so nothing the model wrote is withheld. A model that alters a surrogate (shortens it, splits it with spaces, encodes it) writes something the runtime cannot recognise, and the client receives that as written. ## What the ledger says Two counts on every row of a request redaction applied to: `secretsLifted`, how many secrets its body had taken out, which is the same on every attempt since each sends the same body, and `secretsRestored`, how many surrogates the answer had put back. Both are null on a request redaction did not apply to, and `0` on one it applied to that carried none. `/status` totals them. Nothing else about a secret, not its kind and not its length, is written anywhere. ## When a request cannot be read A request redaction applies to and cannot read is refused with a 422 and a ledger row, and nothing is sent. It fails closed: sending it as it was would be the one thing the operator turned redaction on to prevent. ## What it costs It reads what the caller wrote, which the runtime otherwise does not: it reads to find credentials, routes on nothing it reads, and keeps what it finds in memory. The `VITNA_REDACT_LANES` names are checked against the policy at start, and one the policy does not carry refuses to start, because a typo there is a lane whose secrets leave while you believe otherwise. On Relay, which has no lanes, it is on unless `--no-redact` or `VITNA_REDACT_PASSTHROUGH=0` says otherwise. --- <!-- https://vitna.ai/docs/runtime/relay.md --> # Relay: the runtime with no policy Relay is `vitna-runtime` started with no policy at all, for a developer who wants a local gateway today and a register later, or never. Every request is served as exactly the model it names, on your own keys, and every one is a line in a ledger on your disk. ```bash OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... npx -p vitna-runtime vitna-relay ``` `vitna-relay` is on the path of any [install](installing.md) from 0.2.0, and 0.1.2 has no such command. After `npm install -g vitna-runtime` it is `vitna-relay`. ## What it is The same runtime, not a second gateway. The owner decided on 2026-09-29 that Relay keeps its name as a thin, zero-config preset over `vitna-runtime` rather than a product of its own, so there is one codebase: `vitna-relay` chooses a few defaults and starts the runtime in passthrough mode, and everything that serves a request is the code [serving a lane](serving.md) describes. Four properties it keeps, each held by a test: - **The model served is exactly the one you named.** There is no policy, so there is no lane, no fallback chain, no rule and no seller rotation, and nothing can answer with a model you did not ask for. A request naming `vitna/<lane>` is refused by name with a 404. - **It never calls vitna.ai.** Not at start, not per request, not to check for a newer version. Sending a ledger anywhere is [pushing](pushing.md), a separate command, and it needs a policy this has not got. - **It listens on loopback**, `127.0.0.1:8787`. Any other `VITNA_HOST` needs `VITNA_INBOUND_KEY` or it refuses to start, exactly as the runtime does: see [the front door](serving.md#the-front-door). - **Nothing is estimated.** A provider that reports no usage gets nulls on the row, never a plausible number. ## Pointing a client at it **OpenAI SDKs and OpenAI-compatible tools**: set the base URL to `http://127.0.0.1:8787/v1` and send any model name the provider behind it serves. An unrouted name goes through OpenRouter where `OPENROUTER_API_KEY` is set and to OpenAI directly where only `OPENAI_API_KEY` is, and the boot lines say which key carries what, so read them once. Use the name the provider uses: through OpenRouter that is `openai/gpt-5.6-luna`, directly it is `gpt-5.6-luna`. Nothing here translates one into the other. **Anthropic SDKs and Claude Code**: set `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`. With `ANTHROPIC_API_KEY` set, a request naming a model goes to Anthropic as it was written, with its `anthropic-version` and `anthropic-beta` headers, and comes back as Anthropic sent it, streams byte for byte. Thinking, prompt caching and every beta work as they do direct, and Anthropic's own errors reach the client in its own words, "prompt is too long" included. The one exception is [redaction](#redaction-is-on), which is on: a known credential format in the request is swapped for a stand-in before it leaves, and the real value is put back wherever the answer carries the stand-in. Your client's own credential is never forwarded. The runtime answers with the keys in its own environment, so what a request costs is billed to the `ANTHROPIC_API_KEY` Relay was started with, whatever the client is signed in as. A client still has to hold some credential before it will send anything. On loopback any value does, since nothing reads it; with `VITNA_INBOUND_KEY` set it must be that key, sent as a bearer. For Claude Code that credential is `ANTHROPIC_AUTH_TOKEN`, set in the shell Claude Code starts from, beside `ANTHROPIC_BASE_URL`. Unlike a lane, Relay needs no `MAX_THINKING_TOKENS=0`: a request naming a model is not translated, so thinking is carried. ## What it records One line per request in `~/.vitna/relay/ledger.jsonl`, made on first start. Your home directory rather than the one you ran it from, which is usually a repository somebody could commit a ledger to. Each row is marked unrouted, with no lane, no sku, no decision and no policy digest, because nothing in a record chose the model. What the provider reported is there: tokens in and out, the parts of the input a provider read from or wrote to its prompt cache, latency, status, and the provider's message when it refused. With redaction on, a row also counts the credentials lifted from the request and put back into the answer, and holds none of them. [The ledger](ledger.md) is what each field means. ## Flags | Flag | Variable it sets | What it does | |---|---|---| | `--port <n>` | `VITNA_PORT` | Listen on this port rather than 8787 | | `--ledger <path>` | `VITNA_LEDGER` | Append to this file rather than the one in your home directory | | `--no-ledger` | | Record nothing, which the runtime says out loud at start | | `--redact` | `VITNA_REDACT_PASSTHROUGH=1` | Lift known credential formats out of every request and put them back into its answer: [redaction](redaction.md). The default | | `--no-redact` | `VITNA_REDACT_PASSTHROUGH=0` | Send every request with its credentials as written | | `--cache <seconds>` | `VITNA_CACHE_TTL_SECONDS` | Answer a repeat of a deterministic request from memory for this long: [the response cache](cache.md) | A flag beats the variable, and the variable beats Relay's default. Every other variable of the runtime's (`VITNA_HOST`, `VITNA_INBOUND_KEY`, the upstream budgets, `VITNA_ALLOW_PROVIDERS`) works as it does for `vitna-runtime`. `VITNA_ALLOW_PROVIDERS` matters more here than anywhere, since every request Relay serves names a model rather than a lane, and such a request is held to the list by the route it takes: `openai` admits OpenAI directly, `openrouter` admits OpenRouter, and `anthropic` admits an Anthropic request sent as written. So `VITNA_ALLOW_PROVIDERS=anthropic` makes Relay an Anthropic-only gateway, refusing every chat completion by name. [The provider allowlist](serving.md#the-provider-allowlist) is the whole rule. ## Redaction is on The one thing Relay changes in a request, and the one place the runtime makes a change nobody asked for. A coding agent reads `.env` files, configuration and shell history into its context, and without this the next request carries any production key in them to a provider, where it cannot be called back. So Relay lifts each known credential format out of a request before it leaves, sends a stand-in of the same shape, and puts the real value back wherever the answer carries the stand-in: in text, in tool arguments, and in a stream when a stand-in arrives split across two chunks. A file the agent writes back gets the real key. What it does not do: - **It knows formats, not secrets.** Provider keys, cloud and service tokens, JWTs, private keys, the password in a connection string and secret-named assignments, as [what it lifts](redaction.md#what-it-lifts) lists them. A credential in any other format leaves as written. - **It does not read everything.** Images, documents, tool definitions and thinking go as written, since a thinking block's signature covers its text. The model only ever saw the stand-in, so thinking that mentions a key shows the stand-in. - **It does not survive a restart.** A secret keeps its stand-in for as long as Relay runs, so a conversation's prompt cache holds from turn to turn. A restart draws new stand-ins, and a cached prefix that held a secret misses once. `--no-redact`, or `VITNA_REDACT_PASSTHROUGH=0`, sends every request with its credentials as written, and Relay's lines at start say which it is. The owner decided on 2026-09-29 that Relay turns this on; every other way of starting the runtime keeps it off until an operator names it. ## What it refuses - **A policy.** With `VITNA_POLICY` or `VITNA_POLICY_B64` set it refuses to start: a machine holding a signed policy wants `vitna-runtime`, and serving that policy's traffic as passthrough would be the wrong runtime quietly. The opposite holds too: `vitna-runtime` still refuses to start without a verified policy, so a production runtime whose policy went missing can never become this. - **A network address without a key.** `VITNA_HOST=0.0.0.0` with no `VITNA_INBOUND_KEY` refuses to start, since anybody who found the address could spend your provider keys. - **`VITNA_REDACT_LANES`.** There are no lanes to name; redaction already covers every request. - **A `VITNA_REDACT_PASSTHROUGH` that is neither 1 nor 0.** `true` read as off would turn redaction off for somebody asking for it, so the value is refused at start, by name. - **A lane.** `vitna/<lane>` is a 404 that says this runtime serves no policy. ## When you want lanes Relay is the way in, not the destination. A lane is a model a person accepted a decision about, with a chain behind it and a ledger row that names the decision, and that needs a policy. The same npm package serves one: [installing the runtime](installing.md) is `npm install -g vitna-runtime` started as `vitna-runtime` with a signed policy and the key that signed it, on your own provider keys. Your Relay ledger is already in the shape it writes, so nothing you recorded stops being readable. --- <!-- https://vitna.ai/docs/runtime/route.md --> # Per-request auto-routing with vitna-route `vitna-route` selects the model and reasoning effort for each request deterministically from structural counts of the prompt, preserves prompt caches across multi-turn conversations, and requires an Aperture-signed policy pinned by `VITNA_PUBLIC_KEY` before routing live traffic. It ships inside the Apache-2.0 `vitna-runtime` package with zero external dependencies. Out of the box it starts in `shadow` mode (serving the configured incumbent while recording what rules would choose), and once validated over your shadow ledger it promotes to `route` mode under a signed `vitna-auto-route-policy-v1` artifact. ## Quick start Run `vitna-route` globally from `vitna-runtime` or directly from a checkout of this repository: ```bash # Shadow trial (loopback, unsigned shadow mode) OPENAI_API_KEY=... OPENROUTER_API_KEY=... vitna-route serve # From a repository checkout OPENAI_API_KEY=... OPENROUTER_API_KEY=... npm run runtime:route -- serve ``` Point any OpenAI-compatible SDK or coding harness at `http://127.0.0.1:8787/v1` and pass `"model": "auto"` (or `"vitna/auto"`). Requests naming a concrete model pass through with their parameters intact while still recording a comparison row in the local ledger. ```bash curl http://127.0.0.1:8787/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"auto","messages":[{"role":"user","content":"Summarize HTTP status codes in one sentence."}]}' ``` ## Coding harness quickstart Because `vitna-route` exposes `/v1/chat/completions`, `/v1/messages`, and `/v1/models` on one loopback port, coding agents and SDK clients work with only a base URL override: - **OpenAI SDK / OpenAI-compatible tools (`opencode`, `Crush`):** set `OPENAI_BASE_URL=http://127.0.0.1:8787/v1` and choose model `auto` (or `vitna/auto`). - **Claude Code / Anthropic SDKs:** set `ANTHROPIC_BASE_URL=http://127.0.0.1:8787` for byte-for-byte `/v1/messages` and `/v1/messages/batches` passthrough with per-turn decision attribution in the ledger. ## Aperture-signed promotion and offline verification Unsigned configurations are restricted to `shadow` mode: `vitna-route serve` refuses to boot an unsigned config with `"mode": "route"`. Moving into `route` mode requires an Ed25519 signature over the canonical JSON body (`auto-route-json-v1`), verified against a key you pin in `VITNA_PUBLIC_KEY`. ```bash # 1. Mint an Ed25519 keypair (or use your existing Aperture policy keypair) vitna-route keygen # 2. Sign a validated configuration into a route-mode policy tied to a decision id VITNA_ROUTE_SIGNING_KEY="$(cat route-key.priv)" \ vitna-route sign --org acme --decision 42 --mode route --out auto-route-policy.json # 3. Verify signature, digest, org, decision, and mode offline (exit non-zero on drift) VITNA_PUBLIC_KEY="$(cat route-key.pub)" \ vitna-route verify --policy auto-route-policy.json --org acme --require-mode route --json # 4. Serve under the verified signed policy (SIGHUP reloads fail-statically) VITNA_ROUTE_POLICY=./auto-route-policy.json \ VITNA_PUBLIC_KEY="$(cat route-key.pub)" \ vitna-route serve ``` Pass `--shadow` to `vitna-route serve` at any time as an instant rollback to shadow serving without re-signing or editing the policy file. ## Explain, report, and replay Every request leaves a single text-free JSONL row of feature counts, decision reasons, signed policy metadata, and upstream token usage under `VITNA_ROUTE_DATA_DIR/ledger/YYYY-MM-DD.jsonl` (default `~/.vitna/route/ledger`). No prompt or completion text is ever written. ```bash # Decide a single request locally with full scoring and effort reasons vitna-route explain request.json # Summarize shadow agreement, token costs, and TTFB from recorded traffic vitna-route report # Simulate a candidate policy or config over recorded feature counts vitna-route replay --config candidate.json --since 2026-11-01 ``` Setting `VITNA_LEDGER=/var/lib/vitna/ledger.jsonl` also mirrors each completed request onto the shared [runtime ledger](ledger.md) format so `vitna-runtime-profile` and `vitna-runtime-push` read auto-routed traffic directly. ## Configuration and front-door controls `vitna-route` shares the runtime's loopback and off-loopback security boundaries: - Binding a non-loopback address (`VITNA_HOST=0.0.0.0`) refuses to start unless `VITNA_INBOUND_KEY` or `VITNA_ROUTE_API_KEYS` is configured. - When running open on loopback without inbound keys, browser-forged non-loopback `Host` headers, `/v1/*` `POST` requests with an `Origin` header, and non-`application/json` `/v1/*` POST payloads are rejected with `403`. - `VITNA_ALLOW_PROVIDERS` restricts eligible models and passthrough routes per vendor (for example, `openai,anthropic` excludes OpenRouter-backed targets). - See [The runtime](overview.md) and [Emitting a routing policy](../routing/policy.md) for the sibling lane-based signed policy format (`vitna-routing-policy-v1`). --- <!-- https://vitna.ai/docs/runtime/serving.md --> # Serving a lane Starting a runtime against a signed policy, the two request dialects it answers, and how a lane reaches its model, fails over and stays inside the provider allowlist. ## Starting it ```bash VITNA_POLICY=/etc/vitna/policy.json \ VITNA_PUBLIC_KEY=$(cat /etc/vitna/policy-key.pub) \ VITNA_LEDGER=/var/lib/vitna/ledger.jsonl \ ANTHROPIC_API_KEY=... OPENAI_API_KEY=... \ vitna-runtime ``` `vitna-runtime` is on the path once `npm install -g vitna-runtime` has run, on Node 22.13 or later. [Installing the runtime](installing.md) covers npm, the container, and where the policy and its key come from. | Variable | Meaning | |---|---| | `VITNA_POLICY` | Path to a signed policy on this machine. Required, unless the document arrives encoded. | | `VITNA_POLICY_B64` | The same document, base64 encoded, for a platform where secrets are set and no file can be copied. Exactly one of the two, and both is refused at boot. | | `VITNA_PUBLIC_KEY` | The key to verify it against, pinned once out of band. Required. Several, comma separated, are accepted, which is how a signing key is replaced without an outage: a policy signed by any of them verifies. | | `VITNA_LEDGER` | Where to append one line per request. Optional, and its absence is said out loud. | | `VITNA_PORT` | Default 8787. | | `VITNA_HOST` | Where to listen. Default `127.0.0.1`, where no key is asked for. Anything else needs `VITNA_INBOUND_KEY`, or the runtime refuses to start. | | `VITNA_INBOUND_KEY` | The bearer key every `/v1` request and the full `/status` must send. See [the front door](#the-front-door). | | `VITNA_UPSTREAM_TIMEOUT_MS` | The longest a completion that is not streamed may take. Default 540000. See [when a provider hangs](#when-a-provider-hangs). | | `VITNA_UPSTREAM_FIRST_BYTE_MS` | The longest a stream may take to send its first chunk. Default 300000. | | `VITNA_UPSTREAM_IDLE_MS` | The longest a stream may go quiet between chunks. Default 300000. | | `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY` | Your own keys, for the providers your lanes name. | | `VITNA_ROUTE_<PROVIDER>` | `direct` or `openrouter`, to force a route. | | `VITNA_ALLOW_PROVIDERS` | Comma-separated. Unset is every provider. | ## Operator note: starting it from a checkout This section is for Vitna's own operators. The repository is private, so a reader outside Vitna has no checkout to run it from and starts the runtime from npm as above. In a checkout the same entry point runs as a script, with the same variables: ```bash VITNA_POLICY=/etc/vitna/policy.json \ VITNA_PUBLIC_KEY=$(cat /etc/vitna/policy-key.pub) \ VITNA_LEDGER=/var/lib/vitna/ledger.jsonl \ ANTHROPIC_API_KEY=... OPENAI_API_KEY=... \ npm run runtime:start ``` ## Why the key is pinned rather than read from the file A policy carries the public key that signed it, so a reader can see which key it was. Checking a signature against the key that came with it proves only that the file agrees with itself, which anyone who can write a file can arrange. So the runtime is given the key once, out of band, and refuses to start without one. The key to pin is the one `/console/deployment` prints beside the signed file, not the key over the daily roots on [/witness](https://vitna.ai/witness), which signs the record and not your policy; [Installing the runtime](installing.md) says where both come from. That is what "refuses to start unsigned" has to mean to be worth anything. ## The front door The runtime holds your provider keys, so anybody who can reach it can spend them. Since 0.2.0 it decides who can reach it rather than leaving that to whatever is in front of it: - **It listens on `127.0.0.1` by default**, where only the machine itself can reach it and no key is asked for. A runtime beside the service that calls it needs nothing new, and neither does the operators' `runtime:local`. - **Without a key, it trusts the machine's processes, not the machine's browser.** A web page open on the same machine can reach `127.0.0.1` too. So a runtime with no key answers 403, with no ledger row, to a request whose `Host` is not a loopback name (a DNS-rebinding page naming its own host), and to a `/v1` request that carries an `Origin` header or whose body is not sent as `content-type: application/json` (a page skipping the CORS preflight). An SDK sends none of those. To call the runtime from a browser, set a key. - **Anywhere else, it refuses to start without `VITNA_INBOUND_KEY`.** Set `VITNA_HOST=0.0.0.0` with no key and the boot names both variables and stops. The container image and the Fly recipe set that host, so both need the key. - **With a key set, every `/v1` request sends it** as `authorization: Bearer <key>`, compared in constant time. That is the header an OpenAI SDK sends its API key in, so the key goes where the SDK's key goes. An Anthropic SDK sends its API key as `x-api-key`, which is not read here: give it the key as an auth token instead (`ANTHROPIC_AUTH_TOKEN`, or `authToken` in code), which it sends as a bearer. A request without the key is answered 401 before its body is read, and writes no ledger row, because the ledger is your file and a stranger should not be able to append to it. - **A health check needs no key.** `GET /status` with no header answers `{ "ok": true, "version": "0.2.0" }` and nothing else, and so does `GET /healthz` whatever it sends. The full page, with the org, the lanes, the decision ids and which keys are set, answers only with the key. A request that sends the wrong key to `/status` is told 401 rather than handed the short body, because it meant to authenticate. **What a bearer key is.** It stops the anonymous caller and nothing more. It is one key for every client, so it cannot say which client sent a request, and rotating it means replacing it everywhere it is used. Per-client keys and mutual TLS wait for a customer who needs them. ## The request The runtime speaks the OpenAI chat completions dialect, because that is the one every SDK already speaks. Point a base URL at it and name a lane: ```bash POST /v1/chat/completions { "model": "vitna/deep-research", "messages": [{ "role": "user", "content": "..." }] } ``` `/chat/completions` answers the same, so a base URL that already ends in `/v1` works too. **A model name without the `vitna/` prefix is passed through untouched.** It is marked unrouted on the ledger and attributed to no decision, because nothing in the record chose it. That is what lets a team point one service at the runtime and move its lanes over one at a time rather than all at once. **Every answer says how it was routed**, in four response headers holding the same values as the [ledger](ledger.md) row of the attempt that answered: | Header | What it holds | |---|---| | `x-vitna-lane` | The lane the request named | | `x-vitna-sku` | The sku that answered, which is the fallback's where a fallback did. The one to price the answer at | | `x-vitna-decision` | The decision that authorized that sku | | `x-vitna-policy` | The digest of the signed policy that carried the decision | A header is left off where the row holds nothing: an unrouted name sends only `x-vitna-policy`, and a lane the policy does not carry sends its name and no sku or decision. A stream sends them with its first byte. A request refused at the [front door](#the-front-door) is told none of them. **A caller that keeps its own records can name each request.** Send `x-vitna-request-id` and every ledger row the request writes carries it as `requestId`, each fallback attempt included, so the two records join on it rather than on time and lane. It is recorded as sent, trimmed: 1 to 128 printable ASCII characters with no spaces. Anything else is refused with a 400 and a row, because a caller that sent an id is relying on the join, and a row it cannot find is one it would count as missing. An empty header is no id. **It can also say who the request was for.** An agent or a routine that calls models through the runtime sends its own id in `x-vitna-agent`, and the run it is in, in `x-vitna-run`. Every row the request writes carries them as `agentId` and `runId`, so what one agent's runs cost totals from the ledger rather than being worked out from times and lanes. Both follow the request id's rule, and a run with no agent is refused, since a run is always some agent's. Neither is checked against anything: they are what the caller says, as the request id is. None of the three is sent to a provider. ## The other dialect Some clients speak only Anthropic's messages API, coding harnesses among them. They reach the same lanes at the same base URL: ```bash POST /v1/messages { "model": "vitna/deep-research", "max_tokens": 1024, "messages": [{ "role": "user", "content": "..." }] } ``` `/messages` answers the same, for a base URL that already ends in `/v1`. **A request that names a model rather than a lane is not translated.** Where `ANTHROPIC_API_KEY` is set, and Anthropic is neither outside `VITNA_ALLOW_PROVIDERS` nor forced through OpenRouter, it goes to Anthropic as it was written, with the caller's `anthropic-version` and `anthropic-beta` headers and nothing else of the caller's, its credential least of all. The answer comes back as Anthropic sent it, a stream byte for byte, so thinking, prompt caching and Anthropic's own errors reach the client as they would direct, and the row is marked unrouted like any other model name. Nothing routed that request, so there was nothing to translate it for, and the translation below would only lose the things such a client leans on. Otherwise it is translated and sent where any unrouted name goes, as before. For a lane, it is a translation and not a second runtime. The request is read as a chat completion, routed by the one path that routes, and the answer is written back in the dialect it arrived in, so a lane, a chain, a rule, a shadow match and a [ledger](ledger.md) row are identical whichever endpoint asked. Nothing on the row says which dialect did: a lane's units are its units, and totalling harness traffic separately would be reporting on the client rather than on the lane. **Tools are translated, on every route.** Tool definitions, tool choice, a tool call coming back and a tool result going out all carry, including through a stream, where a call's arguments arrive as the fragments each dialect streams them in. That holds whether the lane is served through OpenRouter or by Anthropic directly, so a harness works without anybody changing which of their accounts serves their traffic. A `system` message inside the messages array is carried too, and stays where it was sent rather than joining the leading system prompt, because a system message the model reads in a different place is a different prompt. This dialect's own documentation does not describe one; Claude Code sends one, which is the sort of thing only running a harness finds. Two fields are carried by callers and deliberately not forwarded, and they are the only two. `cache_control` marks a prompt prefix for Anthropic's own prompt cache; it cannot change a token of the answer, only what the answer costs, and the ledger records the tokens the provider actually reported either way. `is_error` marks a tool result as a failure; its content is forwarded verbatim, so the model still reads the error the tool produced, and the other dialect has nowhere to put the flag. Everything else a caller sends and this runtime does not translate is refused by name, with what to do instead. The ones worth knowing before you start: | Sent | Answer | |---|---| | `thinking` | Refused. A thinking budget has no portable spelling in the dialect the runtime forwards, so turn extended thinking off for the lane. Claude Code sends this unless `MAX_THINKING_TOKENS` is `0`, which is why the launcher sets it. | | `context_management` | Refused. It edits the turn inside Anthropic's API and a routed lane has no equivalent step. Claude Code sends it only to manage thinking blocks, so it goes when thinking does. | | `output_config` | **Carried.** Its `effort` is a level, and the dialect this runtime forwards has the same three under the name `reasoning_effort`. A level outside `low`, `medium` and `high` is refused rather than rounded, and any other key of that object is refused by name. | | `top_k` | Refused. No counterpart, and rescaling it onto `top_p` would be an invented mapping. | | A server tool (`web_search`, `code_execution`, a container) | Refused by name. Anthropic's own API serves those rather than the model, so a lane cannot reach one whatever it is routed through. | | `mcp_servers` | Refused. Connect the server to your harness instead, where its tools arrive as ordinary tools. | | An image inside a `tool_result` | Refused. A tool message in the other dialect carries text and nothing else, and moving the image into a following user turn would put it where the model reads the user saying it. | | `POST /v1/messages/count_tokens` | 404, named. Counting needs the tokenizer of whichever model the lane serves, and an estimate would be a figure nobody measured. | Those refusals are the translation's, so they meet a request naming a lane. A request naming a model, with Anthropic reachable directly, goes as it was written, as above, thinking and all. A harness is pointed at a lane with its own environment: the base URL of this runtime, a key, and `vitna/<alias>` where it expects a model name. The key is never a provider key: on a runtime with no `VITNA_INBOUND_KEY` any non-empty value does, and on one with a key set it is that key, sent as a bearer. [Putting a coding harness on a lane](harness.md) is the command that does it, and covers the two things that go wrong when it is done by hand. ## How a lane reaches its model Vitna decides *what model*. Which of your accounts serves it is your own operational fact, so it comes from the environment rather than from the record: - A provider with a direct key and a direct adapter goes direct. The runtime speaks `openai` and `anthropic` directly. - Anything else rides OpenRouter, which reaches the rest through one dialect. - `VITNA_ROUTE_ANTHROPIC=openrouter` forces the substrate; `VITNA_ROUTE_ANTHROPIC=direct` forces the direct route. A forced route that cannot be taken is unreachable rather than quietly becoming the other one. An operator who wrote down `direct` has said something about committed spend or data handling, and a silent reroute would break it. The name sent differs by route and is never derived: Vitna registers `claude-opus-4-8` and OpenRouter lists `anthropic/claude-opus-4.8`. Both are signed into the lane, so this is a lookup rather than a rule. ## Failing over A lane with a [chain](../routing/fallbacks.md) behind it walks that chain when the primary cannot serve the request now: a transport failure, a 429, or the provider's own 5xx. Everything else stops at the first attempt. A 4xx is the caller's request being wrong, and asking a second model the same wrong question spends money to be told the same thing twice. **A 401 is deliberately on that side of the line**: a bad key is a configuration error, and failing over would serve from the chain indefinitely while the real fault stayed invisible. **One 4xx fails over, by name: Anthropic's empty account.** OpenAI reports an account with no credit as a 429, which fails over by its status. Anthropic reports one as a 400 ("Your credit balance is too low to access the Anthropic API"), which would otherwise stop the chain at exactly the moment a fallback is for. So a 400 from Anthropic, reached directly, whose message says "credit balance" or "billing" walks the chain, and its [ledger](ledger.md) row is a `failed` one whose `error` starts by naming it (`credit_exhausted`) before the provider's own words. Every other 400 still stops at the first attempt, including a malformed request's, which says neither. A budget that runs out before anything reached the caller fails over too, as [a hang](#when-a-provider-hangs) should. A caller who has hung up never does. There is no retry of the same model. A second attempt is a second serve and gets its own ledger row, so a chain of one is a chain of one. An exhausted chain returns the last failure. Nothing is invented in its place. ## The provider allowlist `VITNA_ALLOW_PROVIDERS=anthropic,openai` is a boundary, and it binds fallbacks exactly as it binds a primary: a provider outside it is unreachable by every route, including the substrate, including as the last entry of a chain. An allowlist a chain could reach past is not an allowlist. Unset means every provider, which is the right default for a gateway nobody has stated a boundary for. Inventing a restrictive one would break lanes for a compliance posture the operator never described. **A model name sent without a lane is held to it too, by its route.** A lane is checked against the provider whose model its signed policy names. A name with no `vitna/` prefix carries no such provider, and the runtime does not guess one from the name's shape, so the list is read against where the request would go: | Route an unrouted name takes | Needs listed | Because | |---|---|---| | OpenRouter, where `OPENROUTER_API_KEY` is set | `openrouter` | Behind a name sent there can be any author's model on any host, so listing a model's own provider cannot admit it | | OpenAI directly, where `OPENAI_API_KEY` is set | `openai` | OpenAI is who sees the request | | Anthropic directly, for a `/v1/messages` request [sent as written](#the-other-dialect) | `anthropic` | Anthropic is who sees it | A route the list does not admit is passed over, so with both keys set and `VITNA_ALLOW_PROVIDERS=anthropic,openai`, a name goes to OpenAI rather than through OpenRouter. With no admitted route the request is refused with a 422 naming the variable and what to list, and a ledger row, as every configuration refusal is. Listing `openrouter` admits no lane that was not admitted before. Until 2026-09-29 the list was not read for these names at all, so a name with no prefix reached OpenAI or OpenRouter whatever it said. The boot lines say which key an unrouted name will spend, and a key the list leaves nothing to carry says so, rather than that a fallback might reach it. An OpenAI or Anthropic key carries nothing once its provider is off the list. The OpenRouter key is different, since a lane is held to its model's provider and not its route: leaving `openrouter` off shuts out only names sent without a lane, so the key carries nothing only when every provider the list admits is reached some other way, directly for instance. ## What each lane has done `/status` carries a per-lane count of what has been served, refused, failed and failed over since the process started, with the tokens each lane spent. **It is read, never enforced.** Nothing is refused for exceeding anything, because a gateway that starts dropping production traffic at month end is a page at 2am that Vitna wrote. Enforcement that drops traffic is autopilot-shaped and waits for an explicit opt-in a customer decides. Nothing is counted in money either. A cost needs a price, and a price frozen into a signed policy at emit time is exactly the stale figure presented as current that the record exists to prevent. The units and the tokens are here; the tape prices them later, at the right dates. ## Rules A lane may carry [rules](../routing/rules.md): declared predicates on what the runtime can count about a request, evaluated in order, first match deciding. A SHADOW rule routes nothing and stamps the ledger row of what actually ran with what it would have served. A ROUTED rule's target leads the attempts, attributed to the decision that promoted it, with the lane's own order standing behind it, because serving the decided model is always authorized. What a rule may see is counts of what will be forwarded, never its meaning: `context_chars_over` compares the summed length of the request's string content against a threshold, exactly, and no token estimate is ever made. ## Streaming `"stream": true` works on both endpoints. A same-dialect route is forwarded byte for byte. An Anthropic-direct lane is translated into chat completion chunks, and the usage, which arrives split across two events in that dialect, is held and recorded together. A tool call streams through that translation as well, which takes one thing that cannot be derived either way: the two dialects number things differently. Every content block counts in one, and only the calls are numbered in the other, so a call that follows a sentence is block one and call zero. A caller on `/v1/messages` gets that dialect's events back: a message that opens, opens a content block, fills it, closes it, and closes the message. The one figure that cannot be where this dialect puts it is the input token count, which the other dialect reports at the end of a stream if at all. So it is null on the opening event and real on the closing one, rather than a zero nobody measured. `[DONE]` is a convention rather than a guarantee, so a provider that simply closes its socket still leaves the caller a finished message, and a stream that breaks after its headers have gone out ends with this dialect's error event rather than in silence. ## When a provider hangs Version 0.1.2 had no deadline of any kind. A provider that accepted a request and never answered held the caller and the lane for as long as the caller would wait, and a chain behind that lane was never walked, because nothing ever failed. Since 0.2.0 there are three budgets, in the class the provider SDKs already use: | Budget | Default | What it bounds | |---|---|---| | `VITNA_UPSTREAM_TIMEOUT_MS` | 540000 | A completion that is not streamed, from dispatch to its last byte | | `VITNA_UPSTREAM_FIRST_BYTE_MS` | 300000 | A stream, from dispatch to its first chunk | | `VITNA_UPSTREAM_IDLE_MS` | 300000 | A stream, between one chunk and the next | Each is at most 2147483647, the longest a timer can hold. A longer value is refused at boot rather than accepted, because Node fires a longer timer after one millisecond. **Why nine minutes, and not thirty seconds.** For a request that is not streamed, the first byte of the response is the whole body, so a short deadline there is a short deadline on the whole completion. Reasoning and long-context lanes take minutes when not streamed, which is why the OpenAI and Anthropic SDKs both default to ten. A thirty-second budget would abort a primary the provider was still generating, and billing, and then bill again for the fallback. It is a minute under the SDKs' ten rather than equal to it for the reason in the next section: a budget as long as the caller's never runs out while the caller is still there. **Why five minutes for a stream.** The runtime waits for a stream's first chunk, not its headers, and a reasoning model streamed direct can send no chunk until its hidden reasoning ends. The OpenRouter latency figures the record mirrors (read 2026-09-13) put time to first token above 30 seconds at the 90th percentile on 15 busy endpoints, OpenAI's own among them, and the slowest near five minutes. A budget shorter than that aborts a lane after the provider has started billing for its reasoning, then bills a fallback. Idle is the same figure, because a provider that opens with an empty chunk moves that silence inside the stream. A lane that never reasons can be given shorter budgets. **Your SDK's timeout must be longer than the runtime's budget, or a hang is never failed over.** The SDK starts its clock before the request leaves it, and the runtime starts its own only once the request has arrived and been routed. When the SDK gives up first, the runtime sees a caller who left, walks no chain, and the SDK then retries (twice, by default) against the same hung primary. With the defaults, a default SDK's ten minutes leaves a fallback one minute to answer. A lane that needs more, or a lane with a signed budget near ten minutes, needs the SDK's timeout raised to cover the budget and the fallback's answer. **What a timeout does depends on whether the caller has had anything.** - **Before any byte reached the caller**, it is a provider that cannot serve the request now. It writes a `timed_out` [ledger](ledger.md) row with `latencyMs` at the budget, and the chain is walked exactly as it is for an unreachable provider. A lane with no chain answers 504. The runtime reads a stream's first chunk before it sends the caller anything, so a provider that opens a stream and then goes quiet is still in this case. - **After bytes reached the caller**, the response is ended and the row says `timed_out`. It does not fail over, because a second provider's answer spliced after the first one's opening is not an answer. An Anthropic-dialect caller is sent that dialect's error event first, so it does not wait forever. **A caller who hangs up is followed out.** When the connection closes before the response has finished, the upstream request is aborted, a `caller_left` row is written with `latencyMs` at the moment they left, and the chain is never walked for them. Nobody is listening, so a fallback would be a second bill for a prompt whose answer nobody will read. **Both are requests the provider still bills.** An aborted HTTP request does not refund what the provider generated before the abort reached it. The runtime's budgets stop the second bill and the endless wait; they cannot refund the first. So set your SDK's own timeout longer than these budgets, knowing that when it gives up the runtime aborts and the provider keeps what it already charged, and set these budgets knowing that a budget shorter than a lane's real completion time pays for a failure. **A lane may carry its own completion budget.** A signed policy may put `upstreamTimeoutMs` on a lane, and a 0.2.0 runtime serves that lane under it in place of `VITNA_UPSTREAM_TIMEOUT_MS`. It governs the completion budget only; the stream budgets stay the environment's. The boot output and `/status` print the budget each lane runs under, so a lane signed shorter can be seen to be. It is signed into the policy's digest, the same way a seller set is. So **a 0.1.2 runtime refuses a policy that carries one**: it recomputes a digest without the field, the digest does not match, the refusal is logged, and the policy it was already serving stands. A policy with no budget on any lane is byte-identical to before and verifies everywhere. 0.1.2 enforces no deadline at all, from the environment or from a policy, so upgrade before signing a budget into a lane. ## What it refuses | Refusal | Status | |---|---| | A `/v1` request without the inbound key, where one is set | 401 | | Where no key is set: a non-loopback `Host`, or a `/v1` request with an `Origin` or a body not sent as `application/json` | 403 | | A model field naming nothing | 400 | | A lane the policy does not carry, with the ones it does | 404 | | A lane whose provider cannot be reached with the keys held | 422 | | A lane whose every attempt is outside `VITNA_ALLOW_PROVIDERS` | 422 | | A lane routed through OpenRouter that OpenRouter does not list | 422 | | A model name with no lane prefix, and no key that could serve it | 422 | | A field the Anthropic translation does not carry, such as `response_format` | 400 | | A field this runtime does not read out of a messages request, such as `thinking` | 400 | | A provider that did not answer within its budget, on a lane with no chain left | 504 | | A request still under way when a restart's drain window ran out | 503 | | A provider's own error | Whatever the provider said | The configuration refusals are 422, and were 502 before 0.2.0. Nothing upstream failed: the runtime was started without a way to reach that lane, which is as true on the third attempt as on the first, and the SDKs retry a 5xx. A 422 is answered once. A provider's own status is never rewritten, 5xx included. A field the translation does not carry is refused by name rather than dropped: a caller who sent `response_format`, had it dropped, and got an answer that ignored it has been given a wrong answer that looks like a right one. Route that lane through OpenRouter, which speaks the dialect natively. `tools` used to be on that list, and moving it off is worth stating plainly because the advice attached to it was wrong. A provider with a direct key takes the direct route by default, so that refusal was the default for anybody holding an Anthropic key, and its workaround was to force the route to OpenRouter. A forced route says something about committed spend or data handling. Asking somebody to say it in order to work around an incomplete translator was this runtime reaching into a decision that is theirs. Every one of these except the 401 and the 403 is a [ledger](ledger.md) row. A ledger holding only successes overstates what a lane did. ## What it changes, and only where you ask By default the runtime forwards a request as the caller wrote it, apart from the model name a lane decides, and returns the provider's answer as it came. Four switches change that, and each is off until you set it: | Switch | What it changes | |---|---| | `VITNA_COMPACT_CONTEXT=1` | Compacts prompts before dispatch: trailing whitespace and blank runs outside code fences, identical consecutive turns, and consecutive system turns merged into one | | `VITNA_STRUCTURAL_STOP=1` | Cuts whatever follows a JSON answer's close, for a request asking for `response_format: {"type": "json_object"}` or sending `structural_stop: true`, and ends a stream there. Until 2026-09-29 the request alone switched it on | | `VITNA_CACHE_TTL_SECONDS` | Answers a repeat of a deterministic request from memory: [the response cache](cache.md) | | `VITNA_REDACT_LANES`, `VITNA_REDACT_PASSTHROUGH=1` | Lifts known credential formats out of a request and puts them back into its answer: [redaction](redaction.md) | The first two write a figure of their own on the row when they run, `tokens_evaporated` and `tokens_early_stopped`, counted at four characters a token, which is an estimate rather than anything a provider reported. With neither switched on, no row carries one. ## Reloading `SIGHUP` re-reads the policy file. A reload that cannot be verified changes nothing: the running policy stands, the refusal is logged, and `/status` counts it. Fail static, because adopting an unverified document routes traffic nobody signed for, and refusing to serve turns a bad file into an outage. A stale policy is reported and never enforced. A runtime that stopped serving because its policy was old would convert a control-plane outage into a data-plane one, which is the coupling this design exists to refuse. ## Restarting `SIGTERM`, which systemd, `docker stop` and every `fly deploy` send, drains rather than cuts. The runtime stops accepting and lets the responses already under way finish for up to 30 seconds. Whatever is still running then is cut: a caller still waiting for a whole answer is told 503, a stream is ended with its dialect's error where it has one, and each writes a `drained` [ledger](ledger.md) row before the process exits, which takes up to 2 seconds more. A second `SIGTERM` exits at once. Before 0.2.0 a restart ended every stream in flight mid-body. **Whatever sends the `SIGTERM` must wait longer than that**, or it sends `SIGKILL` mid-drain and the rows go with the process. Neither common default does: `docker stop` waits 10 seconds and Fly waits 5. Give it 35: `kill_timeout = 35` in `fly.toml` (the shipped recipe sets it), `docker run --stop-timeout 35` or `docker stop -t 35`, `stop_grace_period: 35s` in Compose, and `terminationGracePeriodSeconds: 35` in Kubernetes. systemd's default of 90 seconds already covers it. ## Status ```bash GET /status ``` The runtime's version, whether it serves a policy (`mode` is `policy`, or `passthrough` for [Relay](relay.md), which serves none), the policy digest, its age, which lanes are servable, by which route and under which completion budget, which lanes are not and why, whether each key is set, the budgets in force, what each lane has done (including `timedOut`, `callerLeft` and `cacheHits`), what the ledger has written, and what the [response cache](cache.md) and [redaction](redaction.md) have done, as counts. With `VITNA_INBOUND_KEY` set, that page needs the key. Without the header it is `{ "ok": true, "version": "0.2.0" }`, which is also what `GET /healthz` always answers, so a health check needs no credentials and learns nothing else. Keys appear as `set` or `unset`. Never a value, never a prefix of one. A lane that cannot be served does not stop the others starting. A missing key for one provider must not take down the lanes of another, so the runtime starts, names what it cannot serve, and refuses those requests individually. ---