> Source: https://vitna.ai/docs/web/overview
> Raw Markdown: https://vitna.ai/docs/web/overview.md
> Corpus index: https://vitna.ai/llms.txt
# Vitna Web APIs
Hosted pilot at `https://web.vitna.ai` (OpenAPI at `/openapi.json`, MCP manifest at `/.well-known/mcp.json`, Agent Card at `/.well-known/agent-card.json`), with keys by invitation while public
signup is closed. Turn public web pages into Markdown, structured JSON, crawl
datasets, change alerts (`diffMode`), structural redlines (`/v1/diff`), parsed
PDF/Office documents (`/v1/parse`), company dossiers (`/v1/entity`, `/v1/ir`),
cited enrichment matrices (`/v1/grid`), claim verifications (`/v1/verify`),
structured registry firehose events (`/v1/firehose/events`), screenshots, and
printable PDFs (`pageRanges`, `scale`, `printBackground`), alongside 16 official
data sources (`courtlistener`, SEC EDGAR, Federal Register, CISA KEV, FRED, FINRA,
Treasury, NY Fed, BLS, and more) and 25 MCP tools (`/v1/mcp`). Native retrieval,
lexical reranking, observed field extraction, and bounded research analysis run in
the Vitna process by default. The product page and request playground are at
`/apis`. The native customer account console is at `/apis/account`.
To get a key, request an invitation at
[vitna.ai/start](https://vitna.ai/start#web); a person reads the request.
Activate the invitation on the
[account page](https://vitna.ai/apis/account), which shows an API key and an
owner token once each. Nothing is billed during the pilot: usage is measured
on a monthly statement, no rates are set, and payment collection is off. An
org's platform key (`vitna_key_`) works too; see
[One key for the hosted APIs](../one-key.md). Material for the people who
operate the service is collected at the end, under
[Operators: running the service](#operators-running-the-service).
## Authentication and the website relay
The hosted API answers at `https://web.vitna.ai`. Since 2026-10-07 a browser
that opens it, or its `/docs`, is sent to `/apis` and this reference on
vitna.ai, where the product's pages live: the operator sets
`VITNA_WEB_PAGE_URL` and `VITNA_WEB_DOCS_URL`, and a self-hosted instance with
neither set serves its own reference page.
Web operations use a `vitna_web_` bearer key:
```bash
curl http://127.0.0.1:8790/v1/read \
-H "Authorization: Bearer $VITNA_WEB_KEY" \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/","format":"markdown","render":true}'
```
The website can forward requests to the standalone service. Set
`VITNA_WEB_SERVICE_URL` in the website environment, for example to
`http://127.0.0.1:8790`. The relay forwards the caller key and maps
`/api/web/read` to `/v1/read`. It also supports the job and monitor routes
listed below. Every route carries a credential, so the relay sends nothing
over plain HTTP except to a literal loopback address (`127.0.0.1` or
`[::1]`); any other service URL must be HTTPS, and the relay answers `503`
otherwise.
The website relay is mounted as a catch-all route:
```text
GET /api/web/[...path]
POST /api/web/[...path]
DELETE /api/web/[...path]
```
Replace `[...path]` with the supported operation or job path when making a
request, such as `read` or `jobs/JOB_ID`.
With no service configured, the relay returns an explicit configuration
error. The `/apis` playground starts with illustrative output; **Execute
live request** sends a real request through the relay. Its password field
keeps the key in component memory, with no browser storage.
## API accounts and keys
An operator issues each invitation (see
[issuing invitations](#issuing-invitations)). It carries a one-time
activation token, which lasts 24 hours by default. Redeem it once, on the
[account page](https://vitna.ai/apis/account) or directly:
```bash
curl http://127.0.0.1:8790/v1/accounts/activate \
-H 'Content-Type: application/json' \
-d '{"activationToken":"REPLACE_WITH_PRIVATE_ACTIVATION_TOKEN","label":"Application"}'
```
The HTTP 201 response contains `apiKey`, `ownerToken`, account metadata,
and initial key metadata. Save the returned tokens privately; subsequent
reads do not return them. API keys use `vitna_web_`; owner tokens use
`vitna_web_owner_`. The owner token manages the account and keys, while an
API key executes the web operations.
| Endpoint | Owner-token operation |
|---|---|
| `GET /v1/account` | Account, key metadata, and accounting status |
| `POST /v1/account/rotate` | Replace the owner token; send an empty JSON object |
| `GET /v1/keys` | List active, expired, and revoked key metadata |
| `POST /v1/keys` | Issue a key with optional `label` and `expiresAt` |
| `POST /v1/keys/:id/rotate` | Replace that key, invalidating its previous token |
| `DELETE /v1/keys/:id` | Revoke that key |
| `GET /v1/invoices/YYYY-MM` | Metered monthly usage statement |
Use `Authorization: Bearer $VITNA_WEB_OWNER_TOKEN` for those routes.
Creation and rotation bodies accept optional `label` and an ISO UTC
`expiresAt`, such as `2026-12-01T00:00:00.000Z`. Labels accept up to 80
characters. API keys expire within 366 days; up to 20 keys per account are
retained. New requests reject revoked and expired tokens immediately.
Owner rotation invalidates the old owner token. Activation requests are
limited to 10 per minute per process; owner operations to 60 per minute per
account. These management routes use JSON bodies up to 4,096 bytes and do
not accept query parameters.
The SQLite store retains account metadata and credential hashes. Replaying
the startup invitation configuration cannot redeem a used invitation again
or restore a rotated credential. Customer payments are not collected.
Public signup is closed on the hosted pilot, and keys are issued by
invitation. `GET /v1/accounts/signup` reports whether it is open; the
operator's switch is described under
[optional public signup](#optional-public-signup).
A lost owner token cannot be shown again. Ask through
[the contact form](https://vitna.ai/start#contact) and an operator issues a
new one, as described under
[offline operator recovery](#offline-operator-recovery).
## Indexed search
```text
POST /v1/search
```
Search the local index with the same Web bearer key:
```json
{
"query": "product pricing",
"maxResults": 5,
"domains": ["example.com"],
"maxAgeDays": 30
}
```
`query` accepts 1 to 1,000 characters; `maxResults` defaults to 5 and accepts
1 to 20. Optional `index` targets `web` or a configured vertical index, and
optional `effort` (`fast`, `balanced`, `deep`) tunes native retrieval vs.
inline excerpt depth. Optional `domains` and `excludeDomains` filter public hostnames;
`maxAgeDays` accepts 1 to 3,650 and filters observed crawl age. Optional `asOf`
(or `contents.snapshotAsOf`, accepting `YYYY-MM-DD` or ISO 8601 with a UTC
offset) queries historical snapshots captured on or before that instant with
zero lookahead; it requires lexical ranking and cannot accompany live `scrape`.
`timeoutMs` accepts 1,000 to 60,000 milliseconds. Optional `contents` attaches stored
indexed text, `highlights`, or `excerpts` (with optional `maxTotalCharacters` /
`maxTotalTokens` and `injection` guard) in one native call without a live fetch.
The response contains `query`, ranked `results`, engine information, and
request `meta`. Titles, snippets, and crawl timestamps come from the indexed pages. By default,
search makes no live page fetch, and no matching indexed pages returns an empty
result list. See [Vitna Library](/library) on vitna.ai for the full catalog of
vertical indexes and native data sources.
To search and fetch each matching page in one call, set `scrape: true` or pass
a `scrape` object with `format`, `maxCharacters` (default 20,000, up to
100,000), `onlyMainContent`, `includeTags`, `excludeTags`, `render`, `waitFor`,
`timeoutMs`, `retries`, browser controls, `highlights`, `question`, `extract`,
`injection`, or `redact`. Each result then
carries a `scrape` object with `status: "ok"` and the page document, or
`status: "failed"` and the error code when that single URL could not be read.
When `scrape.highlights` is enabled without its own `query`, it defaults to the
search `query`. Each scraped hit counts against the client's page-attempt quota.
### Experimental local semantic ranking
Lexical retrieval remains the default. Operators can enable
`VITNA_WEB_SEARCH_SEMANTIC=true` on the native backend, then request
`ranking:"semantic"` on `POST /v1/search`. This fits a local TF-IDF/LSA
model over at most 128 currently enabled indexed documents, bounded to
300 title characters and 3,500 plaintext characters per document. It uses
eight retained dimensions by default and a cosine cutoff of 0.2. Training
and ranking use no external provider or model download.
The model learns associations within that bounded corpus. It abstains on
unobserved query terms and detected negation/exclusion. It cannot establish
every logical constraint or factual implication. Strict lexical admission
requires `ranking:"lexical"`; combining semantic ranking with strict mode
is rejected. Scores are latent cosine similarities, not probabilities.
Operator host scope restricts the training corpus. Domain, site and age
filters restrict returned IDs; the model still trains over the broader
allowed-host corpus. Source policy and current plaintext are rechecked
after asynchronous fitting, and a changed corpus makes the query abstain
instead of returning a stale passage. Content changes refresh the cached
fingerprint; timestamp-only refreshes need not retrain unchanged text.
Run the frozen authored evaluation locally:
```bash
npm run search:semantic
```
On its 32-document, 26-query synthetic fixture, default eight-dimensional
ranking raises held-out nDCG@5 from 0.566860 to 0.837790 and recall from
0.625000 to 1.000000. MRR falls from 0.916667 to 0.861111; negative accuracy
falls from 5/8 to 4/8. Exact-detail queries include regressions, and observed
cross-topic combinations still produce false positives. The frozen report
at `.search-data/reports/local-semantic.json` records all predeclared
dimensions and regressions. These measurements describe this authored
fixture, not broad web relevance or a general semantic model.
## Reader and scrape
```text
POST /v1/read
POST /v1/scrape
```
These routes are aliases. Read a page as Markdown, readable text, or cleaned
HTML. The reader prefers main/article content, removes page furniture and
scripts, and includes page metadata and links.
```json
{
"url": "https://example.com/docs",
"format": "markdown",
"render": true,
"waitFor": "main article",
"timeoutMs": 30000,
"maxCharacters": 50000
}
```
| Field | Default | Meaning |
|---|---|---|
| `url` | Required | Public HTTPS page, up to 4,096 characters |
| `format` | `markdown` | `markdown`, `text`, or `html` |
| `render` | `true` | Execute page JavaScript |
| `waitFor` | Omitted | CSS selector for content readiness; requires rendering |
| `timeoutMs` | 30,000 | Request deadline, from 1,000 to 60,000 milliseconds |
| `maxCharacters` | 100,000 | Output limit, from 1 to 1,000,000 characters |
| `onlyMainContent` | `true` | Prefer `<main>`/`<article>` and omit `<nav>`/`<aside>` chrome; `false` reads the full `<body>` (still omitting scripts and styles) |
| `includeTags` | Omitted | 1 to 20 CSS selectors (up to 100 chars each); render only matching subtrees in document order |
| `excludeTags` | Omitted | 1 to 20 CSS selectors (up to 100 chars each); omit matching subtrees everywhere, including inside `includeTags` |
| `extract` | Omitted | `schema`, `selectors` and `prompt`, as for Structured extraction; adds `data` from the same render |
| `highlights` | Omitted | `true` or `{ query?, maxCharacters? }` (up to 4,000, default 1,000); returns verbatim `highlights` with `offsetUnit: "unicode_code_points"` |
| `question` | Omitted | A question (up to 1,000 characters) answered from the rendered page; returns `answer` and `answerEngine`. Runs locally on the native extractive model or on a configured language model |
| `injection` | Omitted | `scan` reports what could steer a model reading the page; `strip` also removes invisible characters from `content` (see below) |
| `redact` | Omitted | `true`, or a list of `email`, `phone`, `card`, `iban`, `ssn` and `ip`; takes personal data out of the response |
The HTML output is cleaned content, not the complete serialized browser DOM.
Use [Unblocker](../unblocker/overview.md) when the complete DOM is needed.
Truncated reader output is marked in the response. The configured service
deadline can be shorter than the requested deadline.
**Personal data.** `redact: true` takes email addresses, phone numbers,
payment card numbers, IBANs, US Social Security numbers and IPv4 addresses
out of the response, each replaced by a marker such as `[redacted:email]`. A
list such as `["email", "phone"]` takes only those kinds. The response's
`redacted` counts what was replaced, by kind, for example
`{ "email": 2, "phone": 1 }`. Redaction covers `content`, `title`,
`metadata.description`, `links` and extracted `data`, and `/v1/extract` and
`/v1/crawl` accept the same field. The content is redacted before it is cut to
`maxCharacters`, so a cut cannot leave half an address behind. A language model
reading the page for extraction is given the redacted text, and its answer is
redacted too, since a model can repeat what it has seen elsewhere. A page's own
address (`url`, and the `url` of a failed page in a crawl) is how a result is
matched to its page, so it stays as read. A crawl follows a page's links as the
page wrote them and returns them redacted. An address a page splits with
invisible characters, such as a zero-width space, is still found, and the
replacement takes the invisible characters with it, so `injection: "strip"`
cannot rejoin one. With `injection`, the findings' excerpts are redacted too:
they quote the page's hidden parts, which the content leaves out.
Each kind is found by its shape and a check, never by guessing:
| Kind | Found when |
|---|---|
| `email` | An address with a domain. Image names such as `[email protected]` and remotes such as `[email protected]:org/repo` are not addresses |
| `phone` | A number with `+` and a country code, 8 to 15 digits; a `tel:` link; or a North American number with its punctuation, such as `(212) 555-0147`, `212-555-0147` or `212.555.0147` |
| `card` | 13 to 19 digits, run together or in the usual groups, with the prefix and length of Visa, Mastercard, American Express, Diners Club, JCB, Discover or UnionPay, passing the Luhn check |
| `iban` | A country code, two check digits and up to 30 letters and digits, passing the mod-97 check |
| `ssn` | Three, two and four digits with dashes, never starting 000, 666 or 900 to 999, with no all-zero group |
| `ip` | Four numbers of at most 255. Loopback, multicast, reserved and documentation addresses are left, as are network blocks such as `10.0.0.0/8` and numbers after "Section", "§" or "version" |
**What redaction does not find.** Names, street addresses, dates of birth and
other personal data with no shape of its own; IPv6 addresses; Social Security
numbers without dashes; and phone numbers outside North America written
without their `+` country code. A page about one of these formats, such as an
article on IP addressing, has its examples redacted. A reference number that
passes a card's checks reads as a card, and its marker shows where. An
asynchronous extraction keeps the pages it has read in its restart checkpoint
until the job ends, with or without redaction.
### Documents
A URL that serves a PDF, Word (`.docx`), Excel (`.xlsx`) or PowerPoint
(`.pptx`) file is read as text, with no extra field. A document is recognised
by its content type, or, when the server sends a generic type such as
`application/octet-stream`, by its file name, and its first bytes must agree.
The same reading applies wherever a page is read: Scrape, Extract, Crawl,
Monitors and Research. A page that sends the browser on to a document, as a
cleared challenge page or a script redirect can, is read as that document.
| Kind | Output |
|---|---|
| PDF | A section per page, headed `Page N`, with each page's text in the order the page draws it |
| Word | Headings, paragraphs, bulleted lists, web links and tables |
| Excel | A table per sheet, headed `Sheet: name`; dates and times as their cell formats show them, in ISO 8601, and other numbers as stored |
| PowerPoint | A section per slide in the deck's order, headed `Slide N` and the slide's title |
The response's `metadata.document` describes the file: `kind`,
`contentType`, `filename`, `bytes`, and `pages`, `sheets` or `slides`, with
`truncated: true` when the document passed the reading limit of 1,000,000
characters. Pictures, charts, formulas, speaker notes and other formatting are
not read. Page controls (`waitFor`, `actions`) do not apply to a document, so
a crawl that sets them still reads the documents it links to. `/v1/screenshot`
and `/v1/pdf` capture what the browser shows and do not read documents.
[Linked documents](#linked-documents) downloads the files themselves.
| Code | HTTP | Meaning |
|---|---|---|
| `document_too_large` | 422 | The file exceeds 16 MiB |
| `document_invalid` | 422 | The bytes are not a readable file of the kind they claim |
| `document_encrypted` | 422 | The PDF is encrypted |
| `document_no_text` | 422 | The file has no text; a scanned PDF needs OCR, which is not available |
| `document_unreadable` | 422 | The file's text could not be read, for example a PDF whose fonts carry no map to letters |
| `document_timeout` | 504 | Reading did not finish within 20 seconds or the request deadline |
A document is requested twice: once by the browser, which is stopped as soon
as the response's headers show a document, and once to read it. A download
link that works only once fails on the second request.
**Prompt-injection scan.** Set `injection` to `scan` on `/v1/read`,
`/v1/scrape` or `/v1/crawl` to have each page checked for what could steer a
language model that reads it. The scan uses rules, not a model, so it costs
nothing beyond the read. It reports mechanisms and phrases, and does not judge
intent: a page that writes about prompt injection is reported, because it
holds the phrases.
| Finding | What it is |
|---|---|
| `unicode_tags` | Invisible tag characters spelling text a model can read; the finding shows what they spell. Flag emoji are not reported |
| `invisible_run`, `invisible_characters` | Four or more zero-width characters in a row, the shape of a hidden encoding, or sixteen or more in all. The joiners emoji, Indic and Persian text need are not reported |
| `bidi_override` | A right-to-left override that makes text display other than it reads |
| `ignore_instructions`, `new_instructions`, `role_token`, `system_prompt_probe`, `persona_shift`, `conceal_from_user`, `addressed_to_ai` | Instruction-like text, such as "ignore previous instructions", or the role tokens of a chat template, such as `im_start` and `[INST]` |
| `exfiltration_link` | A Markdown image whose address carries a placeholder for data to send out |
Each finding says where it was found (`text`, `characters`, `hidden_text`,
`comment`, `attribute` or `meta`), whether that text reaches the returned
`content` (`inContent`), its severity, and an excerpt. Hidden places include
elements the reader leaves out, comments, `alt`, `title`, `aria-label` and
meta descriptions, and text hidden by inline styles the reader keeps, such as
zero opacity or font size and off-screen positioning. Text hidden by a
stylesheet class is not seen.
```json
"injection": {
"risk": "high",
"findings": [
{ "kind": "unicode_tags", "severity": "high", "where": "characters", "inContent": true, "excerpt": "28 invisible tag characters spelling: ignore previous instructions" }
]
}
```
`risk` is the most serious finding that reaches the content, or `low` when
every finding sits where the content does not reach. `strip` removes tag
characters, zero-width characters and bidi controls from `content`, keeping
flag emoji and joiners, and reports how many it removed as `stripped`. It
never rewrites visible text. Whatever a request asks, those characters are
removed from everything Vitna's own language model reads.
### Batch scrape
```text
POST /v1/batch
```
Scrape a known list of public HTTPS URLs with shared reader, browser, action,
extraction, highlight, question, injection, and redaction options.
```json
{
"urls": [
"https://example.com/pricing",
"https://example.com/security"
],
"format": "markdown",
"maxCharacters": 20000,
"highlights": { "query": "SOC 2", "maxCharacters": 800 },
"extract": { "selectors": { "heading": "h1" } }
}
```
Synchronous requests accept 1 to 20 URLs and answer `200` with `urls`, `pages`,
`errors`, `counts` (`total`, `completed`, `failed`), `truncated`, and `meta`.
Pass `async: true` to run up to 100 URLs as a background task (`202`), polled
with `GET /v1/jobs/:id`, cancelled with `DELETE /v1/jobs/:id`, and optionally
delivered to `webhookUrl`. `maxCharacters` defaults to 20,000 per page (up to
100,000), and `retries` defaults to `0`.
Each URL is isolated: a page that fails to render or extract is recorded in
`errors` (or `extractError` / `answerError` when reading succeeded) without
aborting the rest of the batch. Background batch jobs checkpoint after every
URL and recover across restarts, recording an in-flight ambiguous URL as
`outcome_unknown` rather than repeating work twice. Result payloads respect the
job byte budget and set `truncated: true` if remaining pages would exceed it.
## Browser controls
Page-reading requests also accept these options:
```json
{
"url": "https://example.com/",
"country": "US",
"session": "product-run-01",
"language": "en-US",
"viewport": { "width": 1440, "height": 960 },
"headers": { "dnt": "1" }
}
```
| Field | Meaning |
|---|---|
| `country` | Two uppercase letters selecting a configured proxy route that serves that country |
| `network` | `auto` (the default) starts on the cheapest configured route and steps up a tier after a block; `datacenter`, `isp`, or `residential` uses only routes of that kind |
| `session` | 1 to 64 letters, numbers, underscores, or hyphens; sticky proxy selection |
| `persistSession` | Explicit boolean; `true` requires `session` and retains tenant-isolated cookies/localStorage |
| `actions` | Up to 50 scroll, wait, click, write or press actions; requires rendering. Bounds, the GET-only guard and failed-step reporting (`error.action`) are under Browser actions in the [Unblocker request reference](../unblocker/overview.md#request). `/v1/read` and `/v1/scrape` also take screenshot and scrape steps, below |
| `language` | BCP 47 locale, such as `en-US`, up to 35 characters |
| `viewport` | Width from 320 to 4,096; height from 240 to 4,096 |
| `headers` | Only `accept-language` up to 200 ASCII characters and `dnt` equal to `0` or `1` |
A country request needs a matching route configured by the operator. An
unavailable country fails explicitly, as does a `network` with no route of
that kind (`network_unavailable`). When the operator configures residential
or ISP provider routes, a blocked attempt's retry moves up a tier, and the
bytes an attempt moves through those routes are metered for pricing; see
[proxy routes](../unblocker/overview.md#proxy-routes). When the operator runs
the [auto browser profile](../unblocker/overview.md#the-auto-profile), a
challenged attempt's retry also moves from Chromium to Camoufox, so a page
behind a browser check can still be read with the default `retries`. So can
a [document](#documents): Camoufox takes the file from the response to the
navigation that cleared the check, so `/v1/read` and each file
[`/v1/documents`](#linked-documents) downloads return it with the same 16 MiB
limit and first-bytes check, and without asking for it a second time. A
document Camoufox still cannot hand over, such as one whose bytes are not the
file they claim to be, fails as the `target_challenge` that sent the read to
Camoufox.
A session keeps route selection
deterministic while the route list stays unchanged. Every attempt still
uses a fresh browser/context. With explicit `persistSession: true`, the
renderer restores and saves that tenant's cookies and localStorage after
successful delivery and cleanup. State stays in RAM with a 15-minute sliding
TTL, 100 total sessions, 20 per tenant, and 256 KiB per session. It is lost on
renderer restart. Same-session concurrent persistence returns `session_busy`.
`DELETE /v1/sessions/SESSION_ID` clears the caller's session and cancels its
active attempt. The caller cannot submit proxy credentials, cookies,
authorization headers, or browser scripts.
**Screenshot and scrape steps.** On `/v1/read` and `/v1/scrape`, an action
sequence can capture the page partway through: `{ "type": "screenshot" }`
images what the page shows at that step, and `{ "type": "scrape" }` reads it.
A list paged by a button reads as one request:
```json
{
"url": "https://example.com/filings",
"actions": [
{ "type": "scrape" },
{ "type": "click", "selector": "button.next" },
{ "type": "scrape" },
{ "type": "screenshot", "fullPage": true }
]
}
```
The response's `content` is the page after the last step, as always, and
`actions` holds the captures, each with the `index` of the step that took it:
```json
{
"actions": {
"screenshots": [{ "index": 3, "contentType": "image/png", "base64": "iVBORw0KGgo...", "bytes": 48213 }],
"scrapes": [
{ "index": 0, "url": "https://example.com/filings", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": ["https://example.com/filings?page=2"], "truncated": false },
{ "index": 2, "url": "https://example.com/filings?page=2", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": [], "truncated": false }
]
}
}
```
A step waits for the page's text to settle before it captures, as the final
read does, so a scrape after a click takes the next page rather than the one
before it. Scraped pages use the request's `format` and `maxCharacters`. A
screenshot shows the viewport unless `fullPage` is `true`, and is a PNG unless
`format` is `jpeg`. A sequence may hold 10 screenshot and scrape steps, which
together may capture 8 MiB (images, and the HTML of scraped pages). A step
past that budget, or a full-page image larger than 16,384 pixels on a side or
32 million pixels in all, fails with `action_output_too_large` (HTTP 422) and
names itself in `error.action`. Other routes, crawls included, refuse these
steps, and `/v1/status` lists them under `capabilities.actionOutputs`.
A scraped page is treated as the final one is: with `injection` it carries its
own `injection` report, and with `redact` its personal data comes out and is
counted in the response's one `redacted`. A screenshot is an image, which
neither the scan nor redaction reads.
Without `waitFor`, the browser allows an initial 500-millisecond grace
period, then waits for 200 milliseconds of stable meaningful visible content,
within five seconds or the remaining request deadline. A persistent empty
or loading state fails with `target_incomplete`. Set a populated content
selector for delayed applications; page stability cannot establish that
every requested product field has finished loading. A `waitFor` selector
that never becomes visible fails with `selector_timeout` (HTTP 504) shortly
before the deadline rather than `request_timeout`, and a retry waits within
what is left of `timeoutMs`.
### Encrypted sessions across restarts
Session state remains in RAM by default. For local browsers, pair
`VITNA_WEB_SESSION_DATA` with `VITNA_WEB_SESSION_KEY_HEX`, an exact 32-byte
hexadecimal operator key. Use a separate private directory from
`VITNA_WEB_DATA`; with the supplied systemd unit it can be
`/var/lib/vitna-web-sessions`. Requests still need a named session and
`persistSession:true`. Browser-worker deployments configure persistence
on each worker instead of the API coordinator.
Cookies and local storage are encrypted using AES-256-GCM with fresh
nonces and authenticated tenant/session identity, expiry and byte bounds.
The same stable key restores unexpired sessions after restart. A wrong key
fails closed and preserves the prior database. A key that opens some rows
but not others is the right key with damaged rows: each damaged row is set
aside unchanged for inspection and counted, and the other sessions open
normally. After startup a damaged row affects only its own session, which
its owner can delete. TTL,
per-client/global capacity, successful-state commit and explicit deletion
continue to apply to durable rows. Deletion cannot be undone by a stale
active lease. Renderer shutdown retains ciphertext and closes storage only
after physical browser cleanup settles.
The helper rejects directory/data-file links and database hardlinks that
could bypass the writer lock. Existing POSIX storage must belong to the
service user and grant no group/other access. Windows deployments must
configure equivalent private ACLs. Session keys are operator secrets;
back up the key separately from a consistent stopped-service database
backup. Key rotation and replicated session storage are not implemented.
## Structured extraction
```text
POST /v1/extract
```
Provide CSS selectors for known fields. A string selector reads text; an
object with `attribute` reads an attribute from matching elements. This
path runs locally without a model. Supply up to 32 selector fields. Supported
syntax includes tags, IDs, classes, attribute matches, descendant and child
combinators, and comma-separated groups. A supplied selector scopes its field;
a missing match is not replaced with an unrelated structured-data value.
```json
{
"url": "https://example.com/products/desk",
"selectors": {
"name": "h1",
"price": ".price",
"image": { "selector": "img.product", "attribute": "src" }
}
}
```
To shape published structured data, provide a supported JSON Schema.
Extraction reads JSON-LD, microdata/RDFa, supported embedded application data,
and any supplied selectors. Canonical URLs, explicit primary-entity references,
product identifiers, and visible headings help select the page's company or
product. An unresolved choice between multiple entities returns
`extraction_ambiguous` (HTTP 422); explicit selectors can select the intended
fields. Schema fields can use object, array, string, number,
integer, and boolean types, including nullable unions with `null`.
Supported keywords include `properties`, `required`, `items`, `enum`,
`additionalProperties`, `description`, and `title`.
```json
{
"url": "https://example.com/products/desk",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": ["number", "null"] },
"currency": { "type": ["string", "null"] }
},
"required": ["name", "price", "currency"],
"additionalProperties": false
}
}
```
Missing fields use the same bounded observed-fact parser as native Research.
It recognizes explicit JSON objects, labelled literals, dotted/indexed fields,
two-column tables, definition lists, and a finite set of unambiguous prose
facts, including named company founding dates, exact employee counts,
headquarters, and product prices/weights. It preserves the source wording,
units, currencies, billing periods, and ranges internally. A measured value
such as `2 kg` needs a string field; it is not reduced to the number `2`.
The symbol `$` alone does not establish the currency code `USD`.
The parser rejects conflicting requested observations with
`extraction_ambiguous` (HTTP 422). Missing required fields return
`extraction_incomplete`; invalid extracted values return `extraction_invalid`,
also HTTP 422. If the configured native engine is called for remaining fields,
its unsupported or conflicting observations return
`native_extraction_unsupported` or `native_ambiguous_data`. An optional
self-hosted compatible model can handle language-based fields beyond these
rules. The final result must satisfy the schema, including nested objects,
arrays, required fields, and enums. Schema responses include configured engine
metadata in `extractionEngine`; extracted `data` follows the requested schema.
**Extract with a prompt.** Add `prompt`, up to 10,000 characters, to tell the
model what to extract in plain language. With a schema, the prompt guides how
the model fills it; without one, the model's JSON object is the data:
```json
{
"url": "https://example.com/investors/governance",
"prompt": "List every board member with their role and the committees they chair."
}
```
A prompt needs a language model (`VITNA_WEB_MODEL_BACKEND=self-hosted`); the
native engine returns `model_not_configured` (HTTP 503) rather than ignoring
it, and `/v1/status` reports `capabilities.promptExtraction`. With a prompt,
the model reads every field that no selector names, because published data
cannot follow an instruction such as "prices in euros". Without one, published
data, selectors and observed facts come first and the model fills only what
they miss. A prompt cannot accompany a recipe, and selectors with a prompt
need a schema that declares their fields.
**Long pages and repairs.** A language model reads the page in parts of 60,000
characters, cut at paragraph breaks, at most eight of them; each part is a
metered model request. The answers combine in page order: a list gathers
values from every part without repeats, and a single value is the first a part
established. An answer that does not match the schema is sent back once with
the problem named before the request fails with `extraction_invalid`. A
response that used a language model reports how it read the page:
```json
"extraction": { "parts": 3, "truncated": false, "repaired": 0 }
```
`truncated` means the text ran past eight parts. Data from a prompt without a
schema is limited to 1 MiB (`extraction_too_large`) and 32 levels of nesting.
The native engine reads a single pass, as before, since it refuses
observations that conflict and must see them together.
**Personal data.** With `redact`, as described under
[Reader and scrape](#reader-and-scrape), a language model reads the pages with
the named kinds already taken out, every string in `data` (and in `rows`) is
redacted as well, and the response's `redacted` counts what came out of them.
**Scrape and extract in one request.** `/v1/read` and `/v1/scrape` accept an
`extract` object with `schema`, `selectors` and `prompt`, under the same rules.
The response carries the page's `content` and the extracted `data` from one
browser render:
```json
{
"url": "https://example.com/products/desk",
"format": "markdown",
"extract": { "prompt": "The price in euros and whether it is in stock" }
}
```
### Extract across pages
Supply `urls` instead of `url` to combine up to 20 pages into one result:
```json
{
"urls": ["https://example.com/about", "https://example.com/contact"],
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"foundingDate": { "type": "string" },
"employees": { "type": "integer" }
},
"required": ["name"],
"additionalProperties": false
},
"async": true,
"jobTimeoutMs": 120000
}
```
For site discovery, provide `url` and `maxPages`, up to 20, with the map
discovery and path-filter options below. The default extraction page budget
is 1. Arrays are combined without duplicate values; scalar fields use the
first valid local value, and compatible nested objects can fill complementary
fields. Flat product extraction combines pages only when their selected
products have matching identity; one product's name cannot borrow another's
price. Use an object array such as `products` for multiple product pages.
Conflicting requested prose observations remain an error. Observed-fact
parsing and the optional model fallback use at most 100,000 combined content
characters. The parser also bounds source count, nesting, arrays, and fact
count; it does not read every page of a domain automatically.
The result includes `url`, successfully read `urls`, combined `data`, and
per-page `errors`. With `async: true`, the response is a job receipt instead.
Poll `/v1/jobs/:id` for progress and the final result. `jobTimeoutMs` accepts
1,000 to 600,000 milliseconds for background work; `timeoutMs` remains the
per-page browser deadline. An optional `webhookUrl` can receive a job
completion notification. Fetch the result from the job endpoint.
## Extraction recipes
```text
POST /v1/recipes/draft
POST /v1/recipes
GET /v1/recipes
GET /v1/recipes/:id
DELETE /v1/recipes/:id
```
A recipe names the fields to read from one kind of page, and how to read
each, once, so every page built from the same template can be extracted or
watched by its id. A draft builds one from examples: name a sample page and,
for each field, its value as the page shows it.
```json
{
"url": "https://shop.example.com/products/desk-lamp",
"samples": ["https://shop.example.com/products/office-chair"],
"fields": {
"name": "Desk Lamp",
"price": "$24.99",
"image": { "example": "https://shop.example.com/img/lamp.jpg", "attribute": "src" },
"reviews": { "examples": ["Great lamp", "Too bright"] },
"sku": { "describe": "the product's SKU" }
}
}
```
| Field form | Meaning |
|---|---|
| `"value"` or `{ "example": "value" }` | Text the page shows, matched regardless of case and spacing; an element whose text closely contains it also matches |
| `{ "examples": ["a", "b"] }` | Two to 20 values of one repeated field, which becomes a list |
| `{ "example": "value", "list": true }` | One value of a repeated field |
| `{ "example": "value", "attribute": "src" }` | An attribute's value rather than text. Without `attribute`, an example not found as text is looked for in `content`, `src`, `href`, `datetime`, `value`, `alt`, `title` and `data-src`, and a relative URL matches its absolute form |
| `{ "describe": "words" }` | No example: extraction finds the value from published data, observed facts and the operator's model when one is configured, and that value becomes the example |
The draft renders the pages with the request's browser controls, finds the
element holding each example in text a reader sees, never under a hidden
ancestor, and keeps the selector that reaches it most steadily. An id comes
first, then `itemprop`, `data-testid` and similar attributes, then classes,
then a class or tag scoped by a named ancestor. Generated class names such as
`css-1x2y3z` or `Button_label__a8Kq3` are never used. A single field's
selector must reach the example before anything else, because extraction
keeps the first match; a list field's must reach every example. Each
candidate is run on the page before it is chosen. The selector engine has no
positions, so an element told from its siblings only by position gets no
selector.
The response gives each field its `selector`, `attribute` and `list`, with
`basis` (`id`, `attribute`, `class`, `scoped` or `tag`), the number of
elements it `matches`, and whether the example matched as `text`, by
`contains` or as an `attribute`. A field without a selector gives its
`reason`: `example_not_found`, `examples_disagree`, `no_stable_selector` or,
for a description, `value_not_found`. A described field's value is returned as
`described`. `recipe.fields` is ready to save, and `preview` reads every page
with the recipe:
```json
{
"preview": [
{ "url": "https://shop.example.com/products/desk-lamp", "data": { "name": "Desk Lamp", "price": "$24.99", "reviews": ["Great lamp", "Too bright", "Works fine"] }, "missing": [] },
{ "url": "https://shop.example.com/products/office-chair", "data": { "name": "Office Chair", "price": "$89.00" }, "missing": ["reviews"] }
]
}
```
This preview is illustrative. A draft is one request, and it reads the
sample page and up to three `samples` through the same browser slots, page
quota and accounting as any read. A sample that cannot be read appears in the
preview with its error code; a first page that cannot be read fails the
draft.
Save a recipe with a `name`, its `fields`, and optionally `render` and
`waitFor`. Each field is a selector string, or an object with `selector`,
`attribute`, `list` and `describe`, of which a selector or a description is
required. A draft's `recipe.fields` can be sent as returned. A selector the
engine cannot parse is refused when the recipe is saved. A key holds up to 50
recipes of up to 32 fields each, and only that key can read, run or delete
them.
Run one by sending `recipe` in place of `schema` and `selectors`:
```json
{
"urls": ["https://shop.example.com/products/desk-lamp", "https://shop.example.com/products/office-chair"],
"recipe": "RECIPE_ID"
}
```
A recipe run adds `rows`, one `{ "url", "data" }` per page read, beside the
combined `data`; a page whose own extraction fails keeps its row with an
`error` code. A described field without a selector is found by extraction on
each page. The recipe's `render` and `waitFor` are defaults that the request's
own values override. `POST /v1/monitors` with `recipe` watches the recipe's
selected fields; a recipe with none is refused with `recipe_unmonitorable`.
An unknown or another key's recipe is `recipe_not_found` (404), and a key at
its limit is `recipe_limit` (429).
## Site map
```text
POST /v1/map
```
Discover links, sitemap entries, and indexed search results. Include and
exclude filters match case-sensitive pathname substrings, and excludes take
priority. Discovery defaults to the starting hostname. Additional domains
must be explicitly allowed.
```json
{
"url": "https://example.com/",
"maxPages": 100,
"maxBreadth": 100,
"sitemap": "include",
"maxSitemaps": 3,
"includePaths": ["/products/"],
"excludePaths": ["/account/"],
"timeoutMs": 30000
}
```
The default `maxPages` is 50, with a maximum of 1,000 for maps. Filters can
contain up to 20 paths per list. The seed page can be read to discover links
even when it does not match an include filter. The response reports
discovery counts and errors in addition to the URLs.
| Field | Default | Meaning |
|---|---|---|
| `maxDepth` | 20 for maps; 2 for crawls | Link depth from the seed, from 0 to 20 |
| `maxBreadth` | 100 | Maximum discovery breadth, from 1 to 1,000 |
| `sitemap` | `skip` | `skip`, `include`, or `only` |
| `sitemapUrls` | Omitted | Up to 5 explicit public HTTPS sitemap URLs |
| `maxSitemaps` | 3 | Sitemap document budget, from 1 to 10 |
| `searchQuery` | Omitted | Optional discovery query, up to 500 characters; uses the selected indexed Search backend |
| `searchLimit` | 20 | Search-result budget, from 1 to 100 |
| `includeSubdomains` | `false` | Allow public subdomains of the starting host |
| `allowedDomains` | `[]` | Up to 20 explicitly allowed public hostnames |
For example, add `searchQuery: "site:example.com products"` to discover
pages through the selected indexed Search backend. Search results and sitemap
entries pass the same destination checks, host scope, and path filters as
links found on a page. `sitemap: "only"` uses sitemap discovery without a
full link walk. A map request remains bounded by its total `timeoutMs`, and
by the service's request deadline when that is shorter. A map that runs out of
time, by either deadline, fails with `crawl_timeout` (HTTP 504); only a caller
that disconnects ends one with `request_cancelled`.
## Crawl jobs
```text
POST /v1/crawl
GET /v1/jobs/:id
DELETE /v1/jobs/:id
```
Collect cleaned pages in a background job:
```json
{
"url": "https://example.com/docs/",
"maxPages": 100,
"maxDepth": 4,
"maxBreadth": 100,
"format": "markdown",
"includePaths": ["/docs/"],
"excludePaths": [],
"timeoutMs": 30000,
"jobTimeoutMs": 120000
}
```
The create response includes a job `id`, `status`, `pages`, and `errors`.
Poll `GET /v1/jobs/:id` using the same client key to retrieve progress and
collected pages. `DELETE` cancels queued or running work.
The default page budget is 50 and the default depth is 2 links from the
seed. The supported depth is 0 to 20; the page budget is 1 to 500. A page
budget bounds attempted reads, while `maxCharacters` bounds each collected
document. `timeoutMs` bounds each browser read. `jobTimeoutMs` bounds the
whole background job, from 1,000 to 600,000 milliseconds, and defaults to
`timeoutMs` when omitted. Partial page failures appear in `errors`.
`redact` takes personal data out of every collected page before the job
stores it, with a `redacted` count on each page, as described under
[Reader and scrape](#reader-and-scrape). Crawls also accept `extract`,
`highlights`, and `question` to run per-page structured extraction, passage
selection, or page-grounded Q&A during the crawl; if extraction or Q&A fails on
one page, that page keeps its content and records `extractError` or
`answerError` while the crawl continues.
Crawls accept the same sitemap, search, domain, and path controls as maps.
Retrieve a bounded slice of crawl results with query parameters:
```text
GET /v1/jobs/JOB_ID?offset=0&limit=25&includeContent=false
```
`offset` accepts 0 to 500, `limit` accepts 1 to 100, and `includeContent`
chooses whether page content accompanies the page metadata. The response
includes `pagination` with `offset`, `limit`, `total`, and `nextOffset`.
Each job also caps returned content at two million characters and five MiB
of serialized pages/errors. When that budget is reached, completed pages
remain available and the job reports `truncated: true` with
`truncationReason: "result_limit"`. Up to 20 jobs per client are retained;
older finished jobs are removed to admit new work. Queued jobs recover after
restart. Crawl resumes its committed frontier and extraction resumes committed
page snapshots; unresolved page attempts are recorded as `outcome_unknown`
and skipped. An ambiguous final model invocation, running Research jobs, and
running catalog and backfill jobs remain failed rather than repeating metered
work. Recovery preserves the
original absolute deadline, backs off, and permits at most three restarts.
Extraction checkpoints share the five-MiB result bound; oversized checkpoints
fail explicitly. Recovery metadata is public, while its snapshots and intents
remain private. These leases are single-writer restart bookkeeping.
Jobs and monitors belong to the client identity associated with the key.
A different client cannot inspect or cancel them.
## Change monitors
```text
POST /v1/monitors
GET /v1/monitors
GET /v1/monitors/:id
DELETE /v1/monitors/:id
POST /v1/monitors/:id/check
```
Watch readable page text, or selected fields, on an interval. Supply a
public HTTPS webhook to receive changes:
```json
{
"url": "https://example.com/products/desk",
"intervalSeconds": 3600,
"selectors": { "price": ".price", "availability": ".availability" },
"webhookUrl": "https://your-app.example/webhooks/vitna"
}
```
The interval defaults to 3,600 seconds and accepts 60 seconds to 30 days.
The first successful check establishes a baseline. Subsequent checks compare
normalized text or selector values. A missing field can be a change; it is
not treated as a price of zero.
To score changes structurally instead of flat text diffs, pass `diffMode`
(`prose`, `pricing`, `policy`, or `risk_factors`) and optional
`minMateriality` (`0` to `1`). When the computed structural materiality score
falls below `minMateriality`, the monitor records a `below_materiality`
suppression entry, skips webhook delivery, and preserves its baseline; when
it meets or exceeds the threshold, the event carries typed hunks, numeric
deltas, and a SHA-256 `diff.attestation` digest.
To suppress minor noise on a page monitor, set `meaningfulChange: true`
(default 5% changed-token ratio), `{ "minChangedRatio": 0.1 }`, or
`{ "prompt": "..." }` (requires a configured language model). When a change is
judged not meaningful, the monitor records a `suppressed` reason in history,
sends no webhook, and keeps the previous baseline intact so gradual drift still
crosses the threshold.
To watch search results for a query instead of a single URL, create a web
monitor with `kind: "web"` (or omit `url` and pass `query`), along with
optional `maxResults` (1 to 20, default 10), `domains`, `excludeDomains`,
`ranking`, `goal`, `dedup` (`"story"` default or `"url"`), `maxAgeDays` (1 to
3,650), `minSnippetCharacters` (1 to 1,000), `summarize` (`true` for cited
bullet summaries), and `schema` (for cited structured event payloads):
```json
{
"kind": "web",
"query": "vitna release notes",
"domains": ["example.com"],
"goal": "pricing change or new enterprise plan",
"dedup": "story",
"maxAgeDays": 14,
"summarize": true,
"intervalSeconds": 3600,
"webhookUrl": "https://your-app.example/webhooks/vitna"
}
```
Web monitors report `added` and `removed` result URLs against the previous
snapshot. With `dedup: "story"`, syndicated reprints with matching canonical URLs,
normalized headlines, or 3-word shingle Jaccard similarity at or above 0.6 are
suppressed across up to 200 seen stories without firing webhooks (`suppressedCount`
is recorded in check history). When `goal` is set, alerts fire only when a newly
added result matches the goal (judged by the configured language model when available,
or deterministically by query-term match otherwise). Setting `summarize: true` or
`schema` attaches cited `summary`, `data`, and `citations` fields to `monitor.changed`.
List or inspect monitors to see their current snapshot, recent changes,
check times, and webhook delivery state. The manual `/check` route takes
an empty JSON object and runs a check immediately. `DELETE` stops the
monitor. Scheduling requires the service process to remain running.
Readiness selectors and retry settings are saved with each monitor. History
is bounded to 20 checks. Webhook failures retain the same event ID across
up to five delivery attempts. An event that fails all five leaves the
queue, so a later successful delivery reports `delivered` again; the loss
stays visible as `webhook.failedEvents`, a running count, and
`webhook.lastFailure`, which names the event, its attempts and the last
error.
## Screenshots and PDFs
```text
POST /v1/screenshot
POST /v1/pdf
```
Capture a browser view:
```json
{
"url": "https://example.com/",
"format": "png",
"fullPage": true,
"width": 1440,
"height": 960
}
```
Screenshots support `png` and `jpeg`. The default viewport is 1,440 by 960
pixels; width accepts 320 to 4,096 and height accepts 240 to 4,096. PDFs
use Chromium print output and need only a URL for the default layout. On
`/v1/pdf`, you can also bound long pages with `pageRanges` (page numbers or
ranges between 1 and 1,000, such as `"1-3"` or `"1-2,5"`), `scale` (`0.1`
to `2`), and `printBackground` (boolean, default `true`) to keep artifacts
under the 10 MiB capture limit:
```json
{
"url": "https://example.com/",
"pageRanges": "1-2",
"scale": 0.75,
"printBackground": false
}
```
Both routes return an artifact in a JSON envelope:
```json
{
"artifact": {
"contentType": "image/png",
"base64": "BASE64_ENCODED_BYTES",
"bytes": 12345
}
}
```
This envelope is illustrative. Decode `base64` into bytes and save them with
the returned content type. The playground provides a download button for
live artifacts.
## Linked documents
```text
POST /v1/documents
```
Download the PDFs a page links to, as the files they are:
```json
{ "url": "https://investors.example.com/financials", "maxFiles": 10 }
```
The page is read in the browser, as for Read, and every document link on it
is listed. A link counts when its path ends in `.pdf` or its `type` attribute
is `application/pdf`; a link whose text, title, label or class says "PDF"
("Annual report (PDF, 2 MB)") counts too and is listed after those, since it
can lead to a page rather than a file. Links in embedded viewers (`iframe`,
`embed`, `object`) count, fragments are dropped, and a link back to the page
itself does not count. Each file is then fetched through the same browser and
network path as the page, with its `country`, `session`, `network` and
`language`, so a site that serves the page to the browser serves its files
too. A `url` that is itself a document is the one file.
| Field | Default | Meaning |
|---|---|---|
| `kinds` | `["pdf"]` | Which documents to find: any of `pdf`, `docx`, `xlsx`, `pptx` |
| `maxFiles` | 10 | How many of the listed links to download, 1 to 20, in the order listed |
| `download` | true | false lists the links without downloading any |
| `timeoutMs` | 30,000 | The whole request's deadline, the page and every file together |
The browser fields (`render`, `waitFor`, `actions`, `country`, `session` and
the rest of [Browser controls](#browser-controls)) apply to the page; files
take its network and identity but not its `waitFor` or `actions`.
```json
{
"url": "https://investors.example.com/financials",
"found": [
{ "url": "https://cdn.example.com/2025-annual-report.pdf", "text": "2025 Annual Report" },
{ "url": "https://investors.example.com/static-files/0f1c", "text": "Q3 results (PDF)" }
],
"documents": [
{
"url": "https://cdn.example.com/2025-annual-report.pdf",
"text": "2025 Annual Report",
"kind": "pdf",
"contentType": "application/pdf",
"filename": "2025-annual-report.pdf",
"bytes": 2481503,
"sha256": "9f2c...",
"base64": "JVBERi0xLjcK..."
}
],
"errors": [{ "url": "https://investors.example.com/static-files/0f1c", "code": "not_a_document" }],
"truncated": false
}
```
This response is illustrative. `found` lists up to 500 links in the order
they are tried. `documents` holds each file downloaded, with `finalUrl` when
a redirect served it from elsewhere, and `errors` each link that did not give
one: `not_a_document` when it served a page, `document_kind_excluded` when it
served a kind not asked for, `documents_too_large` when it would take the
answer past 32 MiB, or the code a Read of it would fail with, such as
`target_http_error` or `document_too_large` for a file over 16 MiB. A link in
`found` but in neither list was not reached, and `truncated` is then true:
`maxFiles`, the byte budget or the deadline stopped the downloads, or the
page had more than 500 document links. The downloads already made are always
returned; only a page that cannot be read fails the request.
The page and each file attempted count against the daily page quota, and each
is metered as a browser read. Files are read one at a time, so ten files take
about ten times as long as one; ask for fewer, or a longer `timeoutMs`, on a
slow site.
## Browser API
```text
GET /v1/browser (WebSocket upgrade)
```
Drive a real Chromium with Puppeteer, Playwright or any Chrome DevTools
Protocol (CDP) client. Each connection launches one browser for that session
and removes it, with its profile, when the connection closes. Scripts, clicks,
typing, forms and screenshots are the client's to send. The endpoint is off
unless the operator sets `VITNA_WEB_BROWSER_API=true`.
Connect with a `vitna_web_` key in the `Authorization` header. The website
relay does not carry WebSockets, so connect to the Web service's own origin:
```js
import { chromium } from "playwright";
const browser = await chromium.connectOverCDP("wss://web.example.com/v1/browser", {
headers: { Authorization: `Bearer ${process.env.VITNA_WEB_KEY}` },
});
const context = await browser.newContext();
const page = await context.newPage();
await page.goto("https://example.com/");
await browser.close();
```
Puppeteer's `connect` takes the same endpoint as `browserWSEndpoint` and the
header in `headers`. The relay is tested with Playwright's `connectOverCDP`
and with a raw CDP client. Add `?country=US` to select an operator proxy
route that serves that country, as page reads do, and `?network=residential`
(or `isp`, `datacenter`) to use only routes of that kind. A session has no
retries to step up through, so without `network` it uses the cheapest
configured route. Add `?session=NAME` (1 to 64 letters, numbers, underscores
or hyphens) to keep a route and its provider address from one session to the
next, as `session` does for page reads: the provider is asked for the same
address whenever that key opens a session of that name, and an unnamed
session is given a fresh one. Whether the address is held is the provider's
to decide. No other parameter is accepted.
The browser starts before the handshake completes, so every refusal is an
ordinary HTTP answer with the JSON error body:
| HTTP | Codes |
|---|---|
| 400 | `invalid_request`, `country_unavailable`, `network_unavailable` |
| 401 | `unauthorized` |
| 404 | `not_found` when the Browser API is not enabled |
| 426 | `upgrade_required` for a request that is not a version 13 WebSocket upgrade |
| 429 | `session_limit`, `service_busy`, `daily_limit`, `rate_limited`, with `Retry-After` |
| 503 | `browser_unavailable`, `service_unavailable` |
### What a session may ask of its browser
Every command passes an allowlist before it reaches the browser. A refused
command is answered with a CDP error, `-32601` for a method that is not
available and `-32602` for parameters that are not, and the browser never
sees it. The relay refuses what would reach past the session:
- **This machine's files**: `DOM.setFileInputFiles`, `Page.handleFileChooser`,
`Input.dispatchDragEvent` carrying files, `Network.loadNetworkResource` and
`DOM.getFileInfo`. A navigation by `Page.navigate`, `Target.createTarget` or
a rewritten `Fetch.continueRequest` must be to an `https:`, `http:` or
`data:` URL, or `about:blank`. File uploads are therefore unavailable.
- **The network guard**: `Target.createBrowserContext` with `proxyServer`,
`proxyBypassList` or `originsWithUniversalNetworkAccess`, and
`Target.setRemoteLocations`.
- **The machine itself**: `Tethering`, `SystemInfo`, `Extensions`, `PWA`,
`Cast`, `Memory` and `Tracing`, and every `Browser` method except
`getVersion`, `close`, the window and permission methods, and
`setDownloadBehavior`, which is rewritten as below.
- **The relay**: `Target.exposeDevToolsProtocol` and
`Target.sendMessageToTarget`. Attach with `flatten: true`, as Puppeteer and
Playwright do.
Downloads are rewritten rather than refused: `Browser.setDownloadBehavior`
and `Page.setDownloadBehavior` are forwarded with `behavior: "deny"`, because
clients ask for downloads into a directory on their own machine. A domain or
method not on the allowlist, including one added in a later Chromium, is
refused until it is reviewed.
The relay keeps one browser session for itself. It denies downloads in every
new browser context before the context's id reaches the client, and it closes
any page past the page limit. Its session id is refused in a client's
commands and its events are not forwarded.
Network access is the same as for page reads: public HTTPS destinations on
port 443 through the guarded proxy, each address checked and pinned, within
the operator's `VITNA_WEB_ALLOWED_HOSTS`. Secure WebSockets are carried; QUIC
and unproxied UDP are disabled. A plain HTTP page is answered with 403 by the
guard itself.
### Session limits and close codes
| Environment | Default | Meaning |
|---|---|---|
| `VITNA_WEB_BROWSER_API` | `false` | `true` enables `/v1/browser` |
| `VITNA_WEB_BROWSER_API_SESSIONS` | 2 | Sessions open at once in this process, 1 to 20 |
| `VITNA_WEB_BROWSER_API_SESSIONS_PER_CLIENT` | 1 | Sessions open at once for one client, at most the process limit |
| `VITNA_WEB_BROWSER_API_SESSION_MS` | 300,000 | Whole-session limit, 10,000 to 3,600,000 |
| `VITNA_WEB_BROWSER_API_IDLE_MS` | 120,000 | Time without a client command, 5,000 to 3,600,000 |
| `VITNA_WEB_BROWSER_API_MAX_BYTES` | 104,857,600 | Network transfer through the guard, both directions, 1 MiB to 10 GiB |
| `VITNA_WEB_BROWSER_API_MAX_PAGES` | 10 | Pages open at once; the relay closes the newest past it, 1 to 100 |
A session also allows 10 browser contexts, 2,000 commands awaiting the
browser, 4 MiB per client message and 64 MiB per browser message. Output a
client leaves unread is capped at 64 MiB. The connection closes with:
| Code | Reason |
|---|---|
| 1000 | The client closed the connection, or sent `Browser.close` |
| 1001 | `service_shutdown` |
| 1007 | A message that was not valid UTF-8 or JSON |
| 1008 | A message that was not a command, or `client_not_reading` |
| 1009 | A client message over 4 MiB |
| 1011 | The browser failed while the session ran |
| 4001 | `session_timeout` |
| 4002 | `idle_timeout` |
| 4003 | `transfer_limit` |
| 4004 | `browser_exited`, a browser that exited unasked |
| 4005 | `output_too_large`, a browser message over 64 MiB |
A session is one request against the client's per-minute and daily limits,
or one shared-quota admission. Its browser time, from launch through cleanup,
is recorded as browser milliseconds and priced at
`VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD`, like every other browser
operation. A session through a residential or ISP route also records its
transferred bytes for per-GB pricing. The session's entry in `/v1/usage` and
its receipt carry the operation `browser`, the close reason as its outcome,
`closeCode`, `network`, `browserProfile`, `commands`, `refusedCommands` and
`transferredBytes`. `GET /v1/status`
reports `capabilities.browserApi` and, when enabled, `limits.browserApi`,
whose `execution` says where sessions run: `{"mode":"local"}`, or the workers
and their combined capacity. Its `browserProfile` is the profile the calling
key's sessions launch with.
### Browser profiles
A session's browser is `native` unless the operator grants its account the
`cloakbrowser` profile, the Unblocker's profile of that name: an
operator-installed [CloakBrowser](../unblocker/overview.md#browser-profiles-and-live-checks)
executable, launched with the same fingerprint flags and a 1920 by 947
desktop window. A named session keeps its fingerprint as it keeps its address,
so it returns as the same device; an unnamed one draws a new fingerprint.
```dotenv
VITNA_WEB_BROWSER_API_CLOAKBROWSER_CLIENTS=["acct_ID","pilot"]
VITNA_WEB_BROWSER_API_CLOAKBROWSER_PATH=/opt/cloakbrowser/chrome
```
The grant lists client ids, a static key's name or an account's id, 1 to
1,000 of them. Like a proxy-access grant it is the operator's check on the
customer: confirm who the customer is and what the traffic is for before
adding one. Public signup never adds an account to it. Every other key's
sessions stay native, on `VITNA_WEB_BROWSER_PATH` or the bundled Chromium.
The path is set here when sessions run on this machine, and on each worker
otherwise (below); startup refuses a path without grants, or grants without a
path to launch them. The CloakBrowser
[binary license](https://github.com/CloakHQ/CloakBrowser/blob/main/BINARY-LICENSE.md)
is the operator's to review.
The relay does not solve challenges. A profile changes how the browser
presents itself, and the network guard and allowlist are the same for every
profile. Verification pages, CAPTCHAs and interstitials reach the client as
they would reach any browser, and what to do with them is the client's. This
differs from page reads, where the Unblocker waits on a recognised Cloudflare
challenge for its client.
By default, sessions launch on the Web process's own machine, including when
page reads use owned browser workers. The executable is
`VITNA_WEB_BROWSER_PATH`, or Playwright's bundled Chromium installed by
`npm run web:install` and `npm run web:browser`; a browser channel alone is
not accepted. The browser runs with its sandbox on, so the host must support
it, on Linux through unprivileged user namespaces or the setuid sandbox. A
reverse proxy must pass the upgrade through: `deploy/vitna-web.nginx.conf.example`
has a `location = /v1/browser` block that does.
### Sessions on owned workers
To run sessions on machines you operate, list them in
`VITNA_WEB_BROWSER_API_WORKERS`:
```dotenv
VITNA_WEB_BROWSER_API=true
VITNA_WEB_BROWSER_API_SESSIONS=8
VITNA_WEB_BROWSER_API_WORKERS=[{"id":"browser-a","origin":"https://browser-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","sessions":8}]
```
Each worker is the `npm run web:worker` process described under
[owned browser workers](#owned-browser-workers), with
`VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS` set to the sessions it may hold (1 to
20; unset or `0` leaves it off). It uses its own `VITNA_WEB_BROWSER_PATH`
and `VITNA_WEB_ALLOWED_HOSTS`, with the same rule that a channel alone is not
accepted. `sessions` defaults to 2, up to 20 per worker and 20 workers, and
`VITNA_WEB_BROWSER_API_SESSIONS` may not exceed their total. These workers
are separate from `VITNA_WEB_BROWSER_WORKERS`, so a machine can serve Browser
API sessions without page reads, or both.
The Web process keeps the client's connection, the command allowlist and
every limit above; the worker runs the browser and its network guard. They
share one connection per session, an HTTP upgrade to `/v1/cdp` on the
worker's server authenticated with its operator token, which carries the
DevTools messages and the guard's transfer count. The operator proxy route a
session selects is sent over that connection, credentials included, so give
workers the same trust as the Web host. A worker answers the upgrade only
once its browser is ready, so a worker that is full or cannot launch is
passed over for the next, most free sessions first; a session already
running is never moved. With every worker full, `/v1/browser` answers 429
`service_busy`. A worker closes a session's browser when its connection
drops, when the worker stops, or 30 seconds past `VITNA_WEB_BROWSER_API_SESSION_MS`
if the Web process never ends it. Its `/healthz` reports `browserApi` with its
sessions, the profiles it can launch and how many are active.
A worker offers the `cloakbrowser` profile only with its own
`VITNA_WEB_BROWSER_WORKER_CLOAKBROWSER_PATH`, an absolute path, and its native
sessions keep using `VITNA_WEB_BROWSER_PATH`. Grants stay on the Web process.
A worker asked for a profile it cannot launch answers 422 `profile_unavailable`
and is passed over like a full one; when no worker offers the account's
profile, `/v1/browser` answers 503 `browser_unavailable`.
## Proxy access
```text
GET /v1/proxy
GET /v1/proxy/keys
POST /v1/proxy/keys
DELETE /v1/proxy/keys/KEY_ID
```
Datacenter proxies from Vitna's own egress gateways, for your own HTTP
clients. The operator grants access account by account, with a monthly
allowance. The account owner then mints proxy keys with the owner token. The
key is shown once:
```bash
curl -X POST https://web.example.com/v1/proxy/keys \
-H "Authorization: Bearer $VITNA_WEB_OWNER_TOKEN" \
-H "Content-Type: application/json" -d '{"label":"crawler"}'
```
The response carries `username` (the account), `proxyKey`, and the key's `id`
and `label`. Connect through any gateway in `GET /v1/proxy` as an HTTPS
proxy:
```bash
curl -x "https://ACCOUNT:[email protected]:8791" https://example.com/
```
`GET /v1/proxy` returns `access` (`granted`, `suspended`, or `not_granted`),
`username`, `endpoints` (each gateway's host, port, and country), `limits`
(`monthlyBytes`, `concurrency`, `tunnelsPerMinute`), the month's
`usage.bytes`, and the keys without their secrets. Revoking a key takes
effect at every gateway within about a minute, including on tunnels already
open. Proxy routes take the owner token, never an API key, and are relayed
by the website like the other account routes.
Only HTTPS destinations are reachable: CONNECT to port 443 of a public
address. A refusal names its reason in an `X-Vitna-Error` header, listed in
the [gateway reference](../unblocker/overview.md#run-an-owned-egress-gateway).
Bytes in both directions are metered and settled hourly into the account's
statement as `proxyBytes`. They are priced per decimal GB at
`VITNA_WEB_BILLING_PROXY_GB_MICRO_USD`. Settled and still-open hours both
count against the month's allowance, which gateways enforce within about a
minute of each other.
The operator configures the grants and gateways:
```dotenv
VITNA_WEB_PROXY_ACCESS={"acct_ID":{"monthlyGb":50,"concurrency":10,"tunnelsPerMinute":600}}
VITNA_WEB_PROXY_GATEWAYS=[{"id":"egress-de","tokenHash":"GATEWAY_TOKEN_HASH","endpoint":"egress-de.example.com:8791","country":"DE"}]
VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000
```
| Grant field | Default | Meaning |
|---|---|---|
| `monthlyGb` | Required | Decimal GB per UTC month, 1 to 100,000 |
| `concurrency` | 10 | Tunnels open at once, 1 to 1,000 |
| `tunnelsPerMinute` | 600 | Tunnels started per minute, 1 to 100,000 |
| `suspended` | `false` | `true` stops the account's access and closes its tunnels |
A grant names an account this service holds, and public signup never
creates one. A grant is the operator's check on the customer: confirm who
the customer is and what the traffic is for before adding one. Changes take
effect on restart, and gateways see them within a minute. Each gateway's
`tokenHash` comes from `npm run egress:gateway-key` on that gateway, and its
`endpoint` is the public host and port customers connect to.
## Product and company data
```text
POST /v1/product
POST /v1/company
```
Both accept a URL and the common browser options. Extraction reads
published JSON-LD, microdata/RDFa, supported embedded application data, and
page metadata, with bounded fallbacks for explicit labels and supported
factual prose. These routes run locally without model configuration.
Product data contains `name`, `description`, `price`, `currency`,
`availability`, `sku`, `brand`, `images`, `offers`, and `variants`. Published
`url`, `gtin`, and `mpn` can also be returned. Records are matched using page
identity, canonical URLs, identifiers, and primary-entity references. Related
records for the same product can contribute offers and variants; unrelated
product cards do not contribute a price to the selected product.
When offers publish different prices, `price` is `null` and `priceRange` can
contain observed `low` and `high` bounds with `currency`. A published single
bound remains a single bound. Prices in different currencies are not combined
into a range. Offers preserve their own `price`, `currency`, `availability`,
and `url`, plus published `lowPrice`, `highPrice`, `offerCount`, or `seller`.
A variant price becomes the selected price only when the page identifies that
variant; otherwise variant prices remain in `variants` and `offers`.
On a Shopify storefront, recognized by Shopify's own page markers, a product
URL (`/products/<handle>`, under an optional locale or collection) also reads
the store's own record from `/products/<handle>.json`. That record sits beside
the page's as `platform`: `name: "shopify"`, `productId`, `handle`, `vendor`,
`productType`, `tags`, `options`, `priceRange`, and every variant with its
`sku`, `barcode`, `price`, `compareAtPrice`, `currency` and `options`. Its
values fill only the fields the page left empty: `brand` from the vendor,
`currency`, `images`, and the `sku` and `gtin` of a product with one variant.
The page's own price and availability stand. When the page's published data
cannot be resolved, for example a graph past the 25,000-node bound, the store's
record answers alone, with one offer per variant; when the store's record
cannot be read, the page's record stands as on any other site. The record is a
plain public HTTPS read with no browser and no further page quota.
Company data contains `name`, `description`, `url`, `industry`, `locations`,
`contactLinks`, `socialLinks`, and `products`. Published `legalName`,
`foundingDate`, `employees`, `email`, and `telephone` can also be returned.
Multiple distinct contacts produce `emails` or `telephones` arrays. A scalar
contact is returned when the page identifies a primary contact or publishes
only one value. Public contact links and `sameAs` social profiles are retained;
a directory's own footer contacts are not assigned to an external company it
profiles. An unresolved choice between companies returns
`extraction_ambiguous` (HTTP 422).
Unknown baseline scalar fields are `null` and baseline lists are empty.
Additional fields appear only when supported by published values. Prices,
dates, employee counts, locations, and contacts are not estimated from other
entities, approximate prose, or missing data.
```json
{ "url": "https://example.com/", "render": true }
```
Available fields depend on what the page publishes. These endpoints do not
fill missing company facts or product details with invented values.
## Store catalogs
```text
POST /v1/catalog
```
List a Shopify or WooCommerce store's products from the store's own catalog:
```json
{ "url": "https://shop.example.com/", "maxProducts": 250 }
```
`url` is any page of the store; its origin is the store. `maxProducts` accepts
1 to 1,000 and defaults to 100. The store's home page says which platform it
runs on, through Shopify's or WooCommerce's own page markers or a WordPress
REST link. When the home page names neither, as on a headless storefront, each
platform's catalog is tried in turn, Shopify first, and each attempt counts as
a page.
The response gives:
- the `store` origin, after following at most two redirects (such as a
`myshopify.com` address, or a bare domain, to the store's own address);
- `platform`, `shopify` or `woocommerce`;
- the store's `currency`;
- `products`;
- the catalog `pages` read;
- `truncated`, when more products remain;
- on WooCommerce, the store's `total` product count.
Each product carries `id`, `handle`, `url`, `title`, `description`, `vendor`,
`tags`, `options`, `images`, `priceRange`, `available` (whether any variant
is), and its `variants`. Each variant has `id`, `title`, `sku`, `price`,
`compareAtPrice`, `currency`, `available` and `options`.
**Shopify.** The catalog is `/products.json`, read a page of 100 at a time.
The `currency` comes from the storefront's own `Shopify.currency` script, and
is `null` on a headless storefront without one. Every product has at least one
variant. Products add `productType`, `publishedAt` and `updatedAt`, and
variants add `barcode`.
**WooCommerce.** The catalog is the Store API that WooCommerce publishes under
the WordPress REST root, read up to 100 products a page in product id order.
The root is `/wp-json/`, or the one the home page names, such as `/?rest_route=/`
or a subdirectory install. A root on another origin is not followed.
Each page's variable products have their variations read beside it, 100 at a
time and up to 250 per product, so each variant carries its own SKU, price and
stock. A variant whose attribute accepts any value names no option for it.
A simple product has no `variants`; its `sku`, `price` and `available` are its
own. Prices are converted from the Store API's minor units, and `vendor` is the
product's brand. Products add:
| Field | Value |
|---|---|
| `sku` | The product's own SKU |
| `type` | `simple`, `variable`, `grouped` or `external` |
| `categories` | Category names |
| `attributes` | The product's attributes that do not define variants, each with its `name` and `values` |
| `price`, `compareAtPrice` | The product's one price and its regular price when on sale; `null` when its variations are priced differently |
| `currency` | The store's currency code |
| `onSale`, `onBackorder` | As the store reports them |
| `lowStockRemaining` | Where the store shows a low-stock count |
| `rating` | `average` and `count`, where the product has reviews |
The catalog is read with plain public HTTPS requests through the same guard as
crawl discovery, with no browser. Every catalog read counts against the page
quota: one per 100 Shopify products, and on WooCommerce one per page of
products plus one per 100 variations. The request counts once against the
request limits. A WooCommerce page may decode to 32 MiB; its compressed
transfer stays within the 2 MiB bound of every public read.
A catalog request must finish within the service's request deadline, 30
seconds by default. WooCommerce stores often serve the Store API uncached, and
a page of 100 products can take several seconds. For a large catalog on a slow
store, set `async: true` to read it as a background job:
```json
{ "url": "https://shop.example.com/", "maxProducts": 1000, "async": true, "jobTimeoutMs": 600000 }
```
The response is a job receipt (HTTP 202) with `kind: "catalog"`. Poll
`GET /v1/jobs/:id`: the job's `progress` counts each catalog read as it
happens, and its `result` is the catalog answer described above.
`jobTimeoutMs` (300,000 by default, up to 600,000) and `webhookUrl` work as
on other [background jobs](#background-jobs), and both require `async: true`.
A job's result is stored whole, within five MiB. A catalog job therefore stops
before the product that would take its result past that bound, and reads no
more pages. It reports `truncated: true` with
`truncationReason: "result_limit"`, as a crawl job does. Products measured on
live stores average about 4 KB, so 1,000 products usually fit. A catalog cut
short by `maxProducts` names no `truncationReason`.
A service restart fails a running catalog job with `job_interrupted`; send it
again. A queued catalog job survives a restart. A catalog dataset reads the
same catalog on a schedule, with its 30-minute run deadline.
The store's `robots.txt` is read first, under the agent name `VitnaWeb`. A
store that disallows its catalog to that agent, or to every agent, is refused
with `catalog_disallowed` (403). The catalog is `/products.json` on Shopify and
the Store API path on WooCommerce. The store's `Crawl-delay`, up to two
seconds, is kept between reads.
These errors can also be returned:
- `catalog_unsupported` (422): the site publishes neither catalog.
- `catalog_unavailable` (502): a later WooCommerce page or variation read
fails. A partial catalog is not returned.
- `destination_blocked` (400): with `VITNA_WEB_ALLOWED_HOSTS` set, every read
must stay in scope, including the redirects it follows.
Requests identify themselves as
`VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview)`.
## Careers
```text
POST /v1/careers
```
A company's open roles from its own job board on Greenhouse, Lever, Ashby,
Workable, Recruitee or Personio, read from each platform's public job board
API:
```json
{ "url": "https://jobs.lever.co/acme", "maxJobs": 200, "descriptions": true }
```
`url` is the board (`job-boards.greenhouse.io/NAME` or `boards.greenhouse.io/NAME`,
`jobs.lever.co/NAME` or `jobs.eu.lever.co/NAME`, `jobs.ashbyhq.com/NAME`,
`apply.workable.com/NAME`, `NAME.recruitee.com`, `NAME.jobs.personio.de` or
`.com`), one of its job pages, or a careers page that links or embeds exactly one board.
`maxJobs` accepts 1 to 1,000 and defaults to 100. `descriptions`, true by
default, includes each posting's description as plain text that keeps its
lines; false leaves it out.
The response gives the `board` (`platform`, `name`, an EU Lever board's or a
Personio board's `region`, and its public `url`), `jobs`, the `pages` read, `truncated`
when more postings remain, and, for Greenhouse, the board's own `total`. Each
job carries the same fields on every platform, `null` or empty where the
board publishes nothing:
| Field | Meaning |
|---|---|
| `id` | The board's id for the posting |
| `title`, `url`, `applyUrl` | The posting, its page, and where to apply |
| `company` | The company name, where the board gives one (Greenhouse, Workable, Recruitee, Personio) |
| `department`, `departments`, `team` | The board's own grouping |
| `locations`, `country` | Every location the posting names |
| `workplaceType`, `remote` | `remote`, `hybrid` or `onsite` as the board states it, and whether the role is remote |
| `employmentType` | The board's own words, such as `FullTime` or `Permanent` |
| `publishedAt`, `updatedAt` | ISO timestamps, where published |
| `compensation` | Lever's or Recruitee's published salary range (`min`, `max`, `currency`, `interval`), or Ashby's `summary` where the company shows pay on its postings |
| `description` | The posting's text, when `descriptions` is true |
| `platform` | `greenhouse`, `lever`, `ashby`, `workable`, `recruitee` or `personio` |
Boards are read with plain public HTTPS requests through the same guard as
store catalogs, with no browser, and the API host's `robots.txt` is read
first under the agent name `VitnaWeb`. Every page read counts against the
page quota: one for a Greenhouse, Ashby, Workable or Recruitee board, one per
50 postings on Lever (with Lever's `Crawl-delay` kept between them), and one
for a careers page that has to be read to find its board. A Personio board is
one read, or up to three with descriptions, since Personio's feed carried its
description sections only when a language was named (measured 2026-10-07):
it is read as it stands, then in English, then in German, until one carries
them. With `VITNA_WEB_ALLOWED_HOSTS` set, the
board's API host must be in scope as well as the `url`. Postings an Ashby
board marks unlisted are left out, as are Recruitee offers not published.
Workable says only whether a role is remote, so its other roles have no
`workplaceType`. A renamed Recruitee company's board redirects to its new
name, which is followed. A whole board's response may decode to
32 MiB; its compressed transfer stays within the 2 MiB bound of every public
read.
| HTTP | Codes |
|---|---|
| 403 | `job_board_disallowed` when `robots.txt` disallows the board's API, which Personio's does for a company it does not know |
| 404 | `job_board_not_found` for a board that does not exist |
| 422 | `job_board_unsupported` when no board is found at `url`; `job_board_ambiguous` when a careers page links more than one, naming each |
| 502 | `job_board_unavailable` when the board answers with anything but its postings |
The platforms document these APIs as public (retrieved 2026-10-07): the
[Greenhouse Job Board API](https://docs.greenhouse.io/job-board.html), the
[Lever Postings API](https://github.com/lever/postings-api), Ashby's
[public job posting API](https://developers.ashbyhq.com/docs/public-job-posting-api),
Recruitee's offers API (`NAME.recruitee.com/api/offers/`), and Personio's
[XML job feed](https://developer.personio.de/docs/retrieving-open-job-positions);
Workable's board widget API (`apply.workable.com/api/v1/widget/accounts/NAME`)
is the one its own job widget reads. SmartRecruiters publishes a posting API
too, but its API host's `robots.txt` disallows every agent but LinkedIn's,
so it is not read.
A careers page rendered only by script names no board in its HTML; send the
board's own URL instead.
## Investor Relations (`POST /v1/ir`)
```text
POST /v1/ir
```
Discover and classify Investor Relations presentations, press releases, financial
reports, earnings transcripts, and webcasts from a company URL or IR portal via
Q4 feeds and rendered IR pages:
```json
{ "url": "https://investor.nvidia.com/", "kinds": ["presentation", "press_release", "financial_report"], "maxItems": 25 }
```
## Document parsing (`POST /v1/parse`)
```text
POST /v1/parse
```
Parse a public HTTPS URL or base64-encoded PDF, DOCX, XLSX, or PPTX document
into structured pages, typed blocks, extracted tables, Markdown, and character
citation offsets:
```json
{ "url": "https://example.com/report.pdf", "format": "both", "maxPages": 100 }
```
## Claim verification (`POST /v1/verify`)
```text
POST /v1/verify
```
Deterministically verify a factual claim against supplied URLs or native search
results, returning per-segment support status, verbatim quotes with character
offsets, content hashes, and a SHA-256 attestation digest:
```json
{ "claim": "Example Domain is maintained for illustrative examples.", "urls": ["https://example.com/"], "maxSources": 3 }
```
## Structural change detection (`POST /v1/diff`)
```text
POST /v1/diff
```
Compare two public URLs (`beforeUrl`, `afterUrl`) or two inline text snapshots
(`beforeText`, `afterText`) across `prose`, `pricing`, `policy`, or `risk_factors`
modes with materiality scores and a SHA-256 attestation digest:
```json
{ "beforeText": "Starter is $29/mo.", "afterText": "Starter is $39/mo.", "mode": "pricing" }
```
## Entity alt-data dossiers (`POST /v1/entity`)
```text
POST /v1/entity
```
Assemble a company dossier in parallel across requested modules (`profile`,
`careers`, `ir`, `infra`, `filings`, `news`):
```json
{ "domain": "nvidia.com", "ticker": "NVDA", "modules": ["profile", "careers", "ir", "infra", "filings", "news"] }
```
## Enrichment matrices (`POST /v1/grid`)
```text
POST /v1/grid
```
Enrich an `entities` by `columns` table synchronously (up to 10 entities, or
asynchronously with `async: true` for up to 100 entities) using `entity`,
`search`, `verify`, or `source` column strategies with cited source URLs, quotes,
and attestation digests. Pass `discover` (with `query`, optional `entityType`,
`count`, `criteria`, `domains`, `excludeDomains`, `queries`, `index`, and `asOf`)
and optional hard `budget` controls (`maxSearches`, `maxVerifications`,
`maxDurationMs`) to discover and verify matching entities in one call, with or
without explicit `entities` or `columns`:
```json
{
"discover": {
"query": "AI inference hardware startups",
"entityType": "company",
"count": 10,
"criteria": ["founded after 2022", "builds custom silicon"]
},
"budget": { "maxSearches": 6, "maxVerifications": 20 },
"columns": [{ "id": "org", "strategy": "entity", "field": "profile.name" }]
}
```
Pass the same `discover`, `budget`, `entities`, and `columns` definition inside
`source.grid` on `POST /v1/datasets` to maintain it as a recurring living
webset with incremental diffs, webhooks, and CSV/JSONL export.
## Datasets
```text
POST /v1/datasets
GET /v1/datasets
GET /v1/datasets/DATASET_ID
DELETE /v1/datasets/DATASET_ID
POST /v1/datasets/DATASET_ID/run
GET /v1/datasets/DATASET_ID/records?run=latest&format=jsonl
GET /v1/datasets/DATASET_ID/changes?after=RUN
```
A dataset is a collection you define once and Vitna keeps current. You give
it a source, the field that identifies a record, and how often to run. Each
run keeps a snapshot of the records and what changed since the run before.
```json
{
"name": "Competitor prices",
"source": { "recipe": "RECIPE_ID", "urls": ["https://shop.example.com/p/1", "https://shop.example.com/p/2"] },
"key": "url",
"ignoreFields": ["checkedAt"],
"intervalSeconds": 86400,
"webhookUrl": "https://hooks.example.com/vitna"
}
```
| Source | Fields | Records |
|---|---|---|
| A recipe over pages | `recipe`, `urls` (1 to 100) | One per page: its `url` and the recipe's fields |
| A recipe over a site | `recipe`, `url`, `maxPages` (1 to 100, default 20), `includePaths`, `excludePaths` | One per page the site map finds |
| A store catalog | `catalog` (a Shopify or WooCommerce store's URL), `maxProducts` (1 to 1,000, default 250) | One per product, as `/v1/catalog` returns it |
| A careers board | `careers` (a board or careers page URL), `maxJobs` (1 to 1,000, default 250), `descriptions` (default true) | One per posting, as `/v1/careers` returns it, keyed by `id` |
| Vitna's market record | `market`: `model`, `tooling`, `extraction`, `speech`, `retrieval`, or `compute` | One per figure on the class's tape, with an `id` naming its sku, seller, meter and any band |
| A grid (living webset) | `grid`: `discover`, `budget`, `entities`, `columns`, `concurrency` (same contract as `POST /v1/grid`, without task fields) | One flat record per matched or supplied entity (`id`, `name`, `domain`, `ticker`, `url`, optional `verification`, and each column id) |
`key` names the field that identifies a record from one run to the next. It
defaults to `url` (`id` for a market, a careers board, or a grid); records without it are skipped and
counted.
When a `grid` run hits a hard `budget` cap or a column cell error, the run is
marked `partial`, preserves previously discovered records, and removes none.
`ignoreFields` lists fields whose changes don't count, such as timestamps.
`intervalSeconds` is 900 to 2,592,000, a day by default. The first run starts
when the dataset is created. `POST .../run` runs it now; a dataset that is
already running answers 409 `dataset_running`.
`GET /v1/datasets/DATASET_ID` lists the runs, newest first. Each gives its
`status`, the counts of `records`, `added`, `changed`, and `removed`, the
`pages` read, and the first 100 page `errors` with their total. A run's
status is one of:
- `succeeded`: it read every page.
- `partial`: it missed a page or stopped at a limit. It keeps the records it
could not reach as they were and removes nothing. A record leaves the
dataset only on a run that read its whole source.
- `failed`: its source could not be read at all. The previous records stay.
`GET .../records` returns a run's records as JSONL (`format=jsonl`, the
default) or CSV (`format=csv`). CSV has one column per field, in the order
fields first appear. Nested values are written as JSON, and text a
spreadsheet would run as a formula gets a leading `'`. `run` is `latest`
(the default) or a run number.
`GET .../changes?after=RUN` returns what each run after `RUN` added, changed,
and removed, oldest first, whole runs at a time, up to 10,000 changes. Added
and changed entries carry the record as it now is; removed entries carry the
key. Read on with `after` set to the response's `next`; `more` says further
runs follow. `resync: true` means runs after yours have been pruned, so read
the records afresh. With `webhookUrl`, each run sends a
`dataset.run.completed` event with its counts, retried with backoff up to
five times.
Runs use the same browser slots, page quota, and accounting as API calls,
one run at a time per service. A run that a restart interrupts starts again
at the next tick. The limits:
| Limit | Value |
|---|---|
| Datasets per key | 5 |
| Records per run | 5,000, each up to 16 KiB |
| Size per run | 8 MiB |
| Runs kept | The last `retainRuns` (1 to 30, default 10), within 32 MiB |
The latest run is always kept, and so is the run the next one is compared
against.
### Published datasets
```text
GET /v1/published
GET /v1/published/SLUG
GET /v1/published/SLUG/records?run=latest&format=jsonl
GET /v1/published/SLUG/changes?after=RUN
```
The operator can publish datasets that every key can read. Vitna's market
record is the main example: each class's prices, kept current, with a feed
of what moved. Published datasets take the same `records` and `changes`
parameters. No key can run or remove them, and reading one counts as a
request like any other.
```dotenv
VITNA_WEB_PUBLISHED_DATASETS=[{"slug":"market-model","name":"AI model prices","source":{"market":"model"},"ignoreFields":["confirmedAt","fetchedAt","evidenceSha256","observationId"],"intervalSeconds":3600}]
```
Each entry is a dataset definition with a `slug` (up to 48 lowercase
letters, numbers, and hyphens) and no webhook. A published dataset keeps its
runs across restarts while its source and key stay the same. It starts
afresh when either changes, and is removed when the configuration drops it.
A market source reads the record's public tape at
`https://vitna.ai/api/v1/markets/{class}/tape`; `VITNA_WEB_RECORD_ORIGIN`
names another origin. Ignoring the confirmation and evidence fields, as in
the example, means a price that is re-confirmed unchanged doesn't count as
a change.
## Firehose
```text
GET /v1/firehose
```
A continuous stream of freshly crawled web pages: every news page in Common
Crawl's news crawl (CC-NEWS), about 300,000 a day in many languages, read by
cursor. Common Crawl publishes a news file every one to two hours. The service
reads each file whole and keeps every HTTP 200 HTML page's title and text, in
the order it read them, for a retention window of 72 hours by default.
```text
GET /v1/firehose?after=0&limit=100&domain=reuters.com,de&language=en,de
```
| Parameter | Default | Meaning |
|---|---|---|
| `after` | 0 | The cursor: records after this one. Send the previous answer's `next` |
| `limit` | 100 | Records per answer, 1 to 1,000 |
| `domain` | Omitted | Up to 20 domains, separated by commas. A domain covers its subdomains, and a top-level domain such as `de` or a suffix such as `co.uk` covers every site under it |
| `language` | Omitted | Up to 10 language codes, such as `en` or `de`, from the page's own `lang` attribute or `Content-Language` header |
| `text` | `true` | `false` leaves each page's text out |
```json
{
"records": [
{
"id": 4102,
"url": "https://www.example.com/news/rates",
"host": "www.example.com",
"fetchedAt": "2026-10-07T14:58:25.000Z",
"language": "en",
"title": "Rates rise",
"text": "The bank raised rates by a quarter point.",
"htmlBytes": 280039,
"digest": "sha1:NQ5RS4U4IVNJT3QEUFJYXWG45ZAZ2XDJ",
"source": {
"crawl": "CC-NEWS",
"file": "crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz",
"offset": 951,
"length": 58004,
"url": "https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz"
}
}
],
"next": 4102,
"more": true,
"missed": false
}
```
Read on by sending `next` as `after`.
- `more` is true while further records already match.
- When `more` is false, the cursor has caught up and `next` holds its place, so
polling with it returns only the pages read since.
- `missed: true` means records past the cursor were pruned before they were
read.
Each record's `fetchedAt` is when Common Crawl fetched the page. `language` is
`null` when the page names none, and `digest` is the payload's SHA-1 as Common
Crawl records it. `source` names the news file and the `offset` and `length`
of the page's WARC record in it, so the page's HTML is one ranged read away:
```bash
curl -r 951-58954 https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz | gunzip
```
The firehose is off until the operator sets `VITNA_WEB_FIREHOSE_PATH` to its
store's SQLite file. Until then it answers `firehose_not_configured` (503), and
`firehose_empty` (503) while the store holds nothing yet. `GET /v1/status`
reports `capabilities.firehose` and `limits.firehoseRecords`. While the
firehose is on, it also reports `firehose`: the store's `files`, `records`,
its newest `latestFile`, and the `from` and `to` of its pages' fetch times.
Each record returned adds one `firehoseRecords` to usage and to an account's
statement. The ledger rate is `VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD`,
and the operator estimate rate for static clients is
`VITNA_WEB_FIREHOSE_RECORD_USD`. A read counts once against the request
limits and uses no page quota.
Common Crawl publishes the crawl under its terms of use; the pages themselves
remain their publishers'.
### Monthly crawl backfill
```text
POST /v1/firehose/backfill
```
A backfill reads one domain's pages from a Common Crawl monthly crawl
(CC-MAIN), in the firehose's record shape, so a feed can start with a site's
recent history. It is always a background job: the request answers HTTP 202
with a job of `kind: "backfill"`. Poll `GET /v1/jobs/:id`; its `result` holds
the pages.
```json
{ "domain": "example.org", "path": "/news/", "language": "de", "maxPages": 200 }
```
| Field | Default | Meaning |
|---|---|---|
| `domain` | Required | A registered domain, such as `example.org`, with its subdomains |
| `path` | Omitted | A path prefix, such as `/news/`, read on the domain itself and its `www` host |
| `crawl` | The newest | A monthly crawl, such as `CC-MAIN-2026-39` |
| `language` | Omitted | One language code, such as `de`, as Common Crawl's own detection reports a page's languages |
| `maxPages` | 100 | Pages read, 1 to 500 |
| `jobTimeoutMs` | 600,000 | The job's deadline, up to 600,000 milliseconds |
| `webhookUrl` | Omitted | A public HTTPS destination told when the job finishes |
Common Crawl's URL index lists the domain's captures in URL order. A URL
captured more than once in the crawl is read once, from its latest capture.
Up to `maxPages` pages are read, each with one ranged request, and each
record is checked against the index before it is kept: its URL, its length
and its payload digest. Aim a backfill at a section with `path`, since URL
order otherwise starts wherever the site's addresses sort first. A record's
`language` is still the one the page declares, which can differ from the
index's detection.
The result gives:
- `crawl` and `domain`, and `path` or `language` when they were asked for;
- `records`, each with `source.crawl` naming the monthly crawl, and no `id`,
since a backfill's pages are not in the stream;
- `captures`, the index rows read (up to five for each page asked for);
- `pages`, the records read, and `skipped`, those that failed a check;
- `truncated`, when more pages exist than were read.
A backfill can also stop early, keeping the pages it already read, with a
`truncationReason`: `result_limit` when the job's five-MiB result is full,
or `commoncrawl_blocked` or `commoncrawl_unavailable` when Common Crawl
stops answering. Many publishers block Common Crawl's crawler, and a backfill
of such a domain finds nothing (`captures: 0`).
Every Common Crawl request from every backfill takes its turn at one a second
by default, so 100 pages take about two minutes. The host shares Common
Crawl's allowance with Vitna Search's archive imports.
- `VITNA_WEB_FIREHOSE_BACKFILL_RATE` raises the pace, up to 4 a second.
- A busy index (a 429 or a 5xx) is asked again twice, 10 and 20 seconds later.
- A 403 stops every backfill for 20 minutes. New ones answer
`commoncrawl_blocked` (503) until then, and `GET /v1/status` reports
`firehoseBackfill.blockedUntil`.
Each record returned is one `firehoseRecords` unit. Backfills are off until
the operator sets `VITNA_WEB_FIREHOSE_BACKFILL=true`, and answer
`firehose_backfill_not_configured` (503) until then. They do not need the
stream's store. `GET /v1/status` reports `capabilities.firehoseBackfill`,
`limits.backfillPages` and, while backfills are on, `firehoseBackfill` with
its `ratePerSecond`. A service restart fails a running backfill with
`job_interrupted`.
### Structured topic events
```text
GET /v1/firehose/events
```
Aggregates normalized public-registry and SEC filing events across up to eight
structured topics (`cisa-kev`, `federal-register`, `fda-drug-recalls`,
`fda-device-recalls`, `treasury-debt`, `sec-8k`, `sec-13d`, `sec-form4`) with a
5-minute in-memory TTL cache per topic:
```text
GET /v1/firehose/events?topic=cisa-kev,sec-8k&since=2026-10-01&limit=20
```
| Parameter | Default | Meaning |
|---|---|---|
| `topic` | All 8 topics | Comma-separated or repeated topic names |
| `since` | Omitted | Earliest publication date formatted `YYYY-MM-DD` |
| `limit` | 20 | Maximum events returned across selected topics, 1 to 100 |
### Ingestion
The Web service only reads the store. `platform/web/firehose-run.mjs` fills
it, run on a timer (`deploy/web-api/firehose.timer`) as the Web service's own
user, so both can open the file. Each run:
1. Lists the current month's news files on `data.commoncrawl.org`, and the
previous month's while the retention window reaches into it.
2. Reads up to `VITNA_FIREHOSE_FILES_PER_RUN` files it does not yet hold,
oldest first.
3. Prunes pages past the window or the text budget, oldest file first.
A file is one request: about 1 GB and 26,000 pages, measured on 2026-10-07,
read in about two minutes. That keeps far inside Common Crawl's request
limits.
- **A 403 stops the run** at once, with no further request, and the run exits
with status 2 so the block shows in the journal.
- **A 429, a 5xx or a dropped connection** stops the run until the next one.
- **A run that stops part-way keeps the pages it read** and resumes at the next
record with a ranged read, so no page is stored twice.
- **One run at a time:** a lease in the store keeps a second run from starting
while one is still going.
| Setting | Default | Meaning |
|---|---|---|
| `VITNA_FIREHOSE_PATH` | Required | Absolute path of the store's SQLite file; the Web service's `VITNA_WEB_FIREHOSE_PATH` names the same file |
| `VITNA_FIREHOSE_RETAIN_HOURS` | 72 | Hours of pages kept, by when their file was crawled, from 1 to 720 |
| `VITNA_FIREHOSE_MAX_TEXT_GB` | 6 | Text kept at most, in GiB; past it the oldest files' pages go first |
| `VITNA_FIREHOSE_MIN_FREE_GB` | 10 | A run reads no file while the disk has less free space than this |
| `VITNA_FIREHOSE_FILES_PER_RUN` | 2 | Files read in one run, from 1 to 24 |
Pages average 4.2 KB of text, so 72 hours of news is about 4.5 GB.
The raw CC-NEWS crawl stream does not yet carry:
- the monthly crawl as a stream, only as a backfill of one domain at a time;
- each page's HTML in the answer;
- delivery by webhook or to a storage bucket.
Structured public-registry event topics are served separately at
`GET /v1/firehose/events`.
## Research
```text
POST /v1/research
```
Search and read bounded page sets, then optionally return extractive findings:
```json
{
"query": "What does this company offer?",
"maxSources": 3,
"answer": true
}
```
Discovery uses the local Vitna index by default; populate that index as
described below, or supply your own `urls` array. `maxSources` defaults to 3
and accepts 1 to 20. With `answer: true`, the native engine selects observed
source passages using lexical matches and returns numbered citations.
Set `answer` to `false` to retrieve page content without an answer:
```json
{
"query": "What does this company offer?",
"urls": ["https://example.com/"],
"maxSources": 3,
"answer": false
}
```
The response includes the query, source page content, an answer when
requested, and page errors. Native passage answers begin with "Extractive
findings from supplied sources". Supported comparisons and arithmetic begin
with "Grounded comparison" or "Grounded calculation" and show observed operands,
units, formulas, and numbered citations. Supported arithmetic includes sum,
average, difference, ratio, percentage change, minimum, and maximum over named
entities and explicit compatible attributes. Conflicting facts, units, billing
periods, missing facets, and unsupported qualifiers refuse instead of guessing.
`generation` carries `engine: "vitna-native"`, `mode: "extractive"`,
`planning: "intent_decomposition"`, and `reasoning: "bounded_rules_v1"`.
Optional `analysis` contains validated coverage and finite calculations with
existing source IDs. This is a finite rule engine; general reasoning still
requires a separately operated self-hosted model. Unsupported requests return
HTTP 422 `native_research_unsupported`.
Supplied URLs bypass indexed discovery. Disabling Search or the answer
engine can return `search_not_configured` or `model_not_configured` when
the request needs that engine.
### Multiple searches and structured results
Use search rounds to refine discovery after earlier pages have been read:
```json
{
"query": "Compare available products and pricing for this company.",
"maxSources": 10,
"maxQueries": 6,
"maxRounds": 3,
"maxPages": 20,
"planQueries": true,
"answer": true,
"async": true,
"jobTimeoutMs": 300000
}
```
| Field | Default | Meaning |
|---|---|---|
| `query` | Required | Question or research brief, up to 1,000 characters |
| `maxSources` | 3 | Readable source budget, from 1 to 20 |
| `maxQueries` | `maxRounds` | Search-query budget, from 1 to 10 |
| `maxRounds` | 1 (`2` with `followLinks`) | Search/read rounds, from 1 to 3 |
| `maxPages` | `maxSources` | Page-attempt budget, from 1 to 40 |
| `followLinks` | `false` | Follow query-relevant links discovered on read pages across rounds (same host when `urls` are supplied) |
| `planQueries` | `false` | Bounded query planning; deterministic source-based variants in native mode |
| `schema` | Omitted | Supported JSON Schema for `data`; requires `answer: true` |
| `stream` | `false` | Server-Sent Events progress and final result |
| `async` | `false` | Submit a durable background job |
| `jobTimeoutMs` | 300,000 | Background deadline, up to 600,000 milliseconds; requires `async: true` |
| `webhookUrl` | Omitted | Public HTTPS completion destination; requires `async: true` |
The input remains one `query`. The returned `queries` array records the
searches actually executed. Up to 20 supplied `urls` bypass search
discovery; they cannot be combined with `planQueries: true`. With
`followLinks: true`, research scores unseen links found on each read page
against the query terms and follows top-ranked candidates in subsequent rounds
(scoped to the seed hosts when `urls` are supplied), bounded by `maxRounds`,
`maxPages`, and `maxSources`. The response `budget` reports actual rounds,
searches, page attempts, sources, model calls, context characters, limits, and
the stopping reason.
Provide a schema to receive validated JSON in `data` alongside the answer
and collected pages. Native mode extracts supported literal values; an
optional self-hosted model can generate schema-shaped data:
```json
{
"query": "List the products this company sells.",
"urls": ["https://example.com/"],
"answer": true,
"schema": {
"type": "object",
"properties": {
"products": { "type": "array", "items": { "type": "string" } }
},
"required": ["products"],
"additionalProperties": false
}
}
```
### Timelines and multi-attribute comparisons
Native Research can order observed historical events, select explicit
earliest/latest events and calculate elapsed days between two cited dates.
It accepts full ISO calendar dates, timezone-qualified ISO instants and
unambiguous month-name/day/year dates. Year-only dates, numeric slashes,
implicit timezones, mixed day/instant precision and crawl metadata are
refused. Conflicting dates for the same named event remain ambiguous;
requesting a year cannot resolve that contradiction.
Structured responses expose `analysis.timeline` with at most 24 ordered
events, normalized dates, precision and source IDs. Elapsed calculations
use cited epoch-millisecond operands and an explicit day result.
Multi-attribute comparisons expose at most 24 `analysis.comparisons` cells
with entity, attribute, observed display value and source ID. Explicit date
filters apply before selecting historical numeric records; absent or
conflicting cells are refused rather than silently filled. All emitted
analysis must pass the finite public response contract.
The preserved native evaluation remains 44/44 cases. A separate authored
timeline/comparison group passes 40/40, including 22 refusal diagnostics,
in `outputs/web-native-evaluation/report-v3.json`. These fixtures exercise
the bounded deterministic grammar, not open-ended general reasoning.
### Stream progress
Set `stream: true` on a synchronous research request. The response content
type is `text/event-stream`. Progress event names include `started`,
`round`, `planning`, `search`, `source`, `source_error`, `generating`, and
`completed`. Events include a monotonic sequence number and measured
progress. The final `result` event contains the complete JSON response:
```text
event: source
data: {"type":"source","sequence":3,"url":"https://example.com/"}
event: result
data: {"query":"What does this company offer?","answer":null,"mode":"sources","sources":[],"meta":{}}
```
This wire example is illustrative. A stream failure uses an `error` event
with `error.code` and `error.message`. Inspect that event even if the HTTP
status was 200. The synchronous stream deadline is at most 60 seconds and
may be shorter under the deployment settings. Disconnecting cancels the
request. The playground displays progress, the final result, and stream
errors; cURL users should include `-N` to see events as they arrive.
### Background jobs
Set `async: true` for research, extraction or a [store catalog](#store-catalogs)
that needs a longer deadline. A [firehose backfill](#monthly-crawl-backfill) is
always a job.
The create response uses HTTP 202 and includes `id`, `kind`, `status`,
timestamps, `jobTimeoutMs`, `progress`, `result`, and `error`. Poll
`GET /v1/jobs/:id`. Completed research, extraction, catalog or backfill
output is in `result`;
failed jobs carry `error`. `DELETE /v1/jobs/:id` cancels active work.
`async` and `stream` cannot both be true. Async jobs can use an optional
completion webhook. Store the job ID even when using a webhook, so progress
and delivery outcomes remain inspectable. Job scheduling and callbacks
require the service process to keep running.
## Sources
```text
GET /v1/sources
GET /v1/sources/:provider
POST /v1/sources/find
POST /v1/sources/run
```
Sources are official data providers, federal agencies and registries with
public APIs, that an application or agent can discover, read the contract of,
and call with the same key as every other Web API. Each provider offers
capabilities, and each capability publishes a contract: its `input` and
`output` as JSON Schema, its `price`, and the `terms` its data comes under.
Reading the list, a provider's contracts, or a `find` is free. A call through
`/v1/sources/run` is metered once it has returned data.
Sources are off until the operator sets `VITNA_WEB_SOURCES=true`.
`GET /v1/status` lists the enabled providers in `capabilities.sources` (empty
while they are off), and every sources route answers `sources_not_configured`
(503) until then.
| Provider | Capabilities | Data |
|---|---|---|
| `federal-register` | `documents/search`, `documents/get` | Rules, proposed rules, notices and presidential documents, with agencies, dockets, comment deadlines and effective dates |
| `ecfr` | `sections/search`, `sections/text` | The Code of Federal Regulations as in force today or on a past date, and a section's full text |
| `usaspending` | `awards/search`, `agencies/list` | Federal contracts, grants and other awards, and each agency's budget authority and obligations |
| `sec-edgar` | `companies/lookup`, `filings/recent`, `filings/search`, `facts/concept` | Companies' CIKs, their recent EDGAR filings, full-text filing search, and XBRL financial facts |
| `openfda` | `drugs/labels`, `drugs/reactions`, `recalls/search`, `devices/events`, `drugs/recalls`, `devices/recalls` | Drug labeling, adverse event counts, food/drug/device enforcement recalls, and MAUDE medical device events and CDRH recalls |
| `clinicaltrials` | `studies/search`, `studies/get` | Registered clinical studies, their status, phase, eligibility and outcomes |
| `cisa-kev` | `catalog/search`, `catalog/summary` | CISA Known Exploited Vulnerabilities catalog search, ransomware campaign filter, and vendor summary |
| `fred` | `series/observations` | St. Louis Fed FRED macroeconomic time series observations with period changes |
| `finra` | `short-volume/ticker`, `short-volume/top` | Daily Reg SHO short sale volume by ticker and top shorted symbols |
| `treasury` | `debt/daily`, `cash/tga`, `rates/average` | Treasury Debt to the Penny, TGA operating cash balance, and average interest rates |
| `nyfed` | `rates/sofr`, `rates/effr`, `operations/rrp` | New York Fed daily SOFR, EFFR reference rates, and overnight Reverse Repo operations |
| `bls` | `series/observations`, `series/latest` | Bureau of Labor Statistics CPI, unemployment, payroll, and price time series |
| `courtlistener` | `courts/list`, `opinions/search`, `dockets/search` | Free Law Project U.S. court directory, judicial opinions, and RECAP federal court dockets |
| `fcc` | `filings/search`, `proceedings/search` | FCC ECFS telecommunications docket filings and rulemaking proceedings |
| `eia` | `energy/inventory`, `electricity/demand` | Weekly U.S. petroleum/natural gas inventories and hourly RTO electricity grid demand |
| `dol-h1b` | `employers/approvals`, `lca/search` | USCIS H-1B Employer Data Hub approval/denial counts and DOL OFLC LCA wage filings |
### Find a capability
Describe what you need, and optionally narrow it to a `category`
(`government`, `finance`, `health` or `security`) or a `provider`:
```json
{ "query": "food recalls for listeria", "limit": 3 }
```
`limit` accepts 1 to 20 and defaults to 5. Each result is a whole contract,
ranked by how well its title, keywords and description match the query, with
its `score`:
```json
{
"query": "food recalls for listeria",
"results": [
{
"score": 9,
"provider": "openfda",
"capability": "recalls/search",
"title": "Search FDA recalls",
"description": "Recall enforcement reports for drugs, foods or medical devices: ...",
"input": { "type": "object", "properties": { "product": { "type": "string", "enum": ["drug", "food", "device"], "default": "food" } }, "required": [], "additionalProperties": false },
"output": { "type": "object", "properties": { "total": { "type": "integer" } }, "required": ["total", "recalls"], "additionalProperties": false },
"example": { "provider": "openfda", "capability": "recalls/search", "options": { "product": "food", "query": "listeria", "classification": "Class I", "limit": 5 } },
"price": { "resource": "sourceRequests", "unit": "call", "microUsd": null, "discovery": "free" },
"available": true,
"terms": { "license": "CC0 1.0 Universal (public domain dedication)", "url": "https://open.fda.gov/license/", "attribution": "Source: openFDA, U.S. Food and Drug Administration.", "notices": ["Do not present this data as endorsed by the FDA."], "retrieved": "2026-10-07" }
}
]
}
```
The schemas are shortened here. Without a `query`, `find` lists every
capability in catalog order. `GET /v1/sources` lists the providers, and
`GET /v1/sources/:provider` returns one provider with all of its contracts.
Contracts are closed: an option the input schema does not name is refused
with `invalid_options` (400), and the message names the option and the rule
it broke, such as `options.title must be at most 50.` Every object in an
output schema is closed too, and a call whose mapped result does not match its
output schema fails with `source_contract` (502) rather than reaching the
caller in an undeclared shape.
### Call a capability
Send the provider, the capability and its options. `example` in every
contract is a body that works:
```json
{ "provider": "ecfr", "capability": "sections/search", "options": { "query": "emergency exits", "title": 14, "limit": 5 } }
```
```json
{
"provider": "ecfr",
"capability": "sections/search",
"data": {
"total": 101,
"page": 1,
"pages": 21,
"sections": [
{
"citation": "14 CFR 25.807",
"title": "14",
"part": "25",
"section": "25.807",
"type": "Section",
"heading": "Emergency exits.",
"partHeading": "Airworthiness Standards: Transport Category Airplanes",
"excerpt": "types of exits are defined as follows: (1) Type I. This type is a floor-level exit with a ...",
"inForceFrom": "2016-12-30",
"inForceUntil": null,
"url": "https://www.ecfr.gov/current/title-14/section-25.807"
}
]
},
"source": {
"name": "Electronic Code of Federal Regulations",
"publisher": "Office of the Federal Register (National Archives) and the U.S. Government Publishing Office",
"url": "https://www.ecfr.gov/api/search/v1/results?query=emergency+exits&per_page=5&page=1&date=current&hierarchy%5Btitle%5D=14",
"retrievedAt": "2026-10-07T05:00:00.000Z",
"license": "U.S. Government work, not subject to copyright in the United States (17 U.S.C. 105)",
"licenseUrl": "https://www.copyright.gov/title17/92chap1.html#105",
"attribution": "Source: the Electronic Code of Federal Regulations (eCFR), Office of the Federal Register.",
"notices": ["The eCFR is authoritative but unofficial. Legal research should be verified against the official annual CFR, the daily Federal Register and the List of CFR Sections Affected."],
"modifications": "Vitna returns the fields this contract names, under its own field names, with markup removed. A text field is cut at a word boundary at the length its description states (4,000 characters unless it says otherwise)."
}
}
```
Like every response it also carries `meta`. `source.url` is the upstream request the data came from, with any key
removed. `dataAsOf` appears when the provider states how current its data
is: openFDA's last update date, and ClinicalTrials.gov's processing
timestamp exactly as its API writes it, with no time zone. Show `attribution`
and `notices` wherever you display the data. ClinicalTrials.gov's terms ask
redistributors to name it as the source, show the processing date, keep the
data current and state modifications, which is what `dataAsOf` and
`modifications` are for. Contact names, emails and phone numbers from study
records are never returned.
### Errors
| Code | Status | Meaning |
|---|---|---|
| `invalid_options` | 400 | The options break the contract; the message names the option |
| `source_not_found` | 404 | No such provider or capability, or the provider has no such record |
| `source_rejected` | 400 | The provider refused the query, in its own words where it gives them |
| `source_rate_limited` | 429 | The provider is limiting requests; `Retry-After` says when to retry |
| `source_capacity` | 429 | Vitna has used the provider's daily allowance; it resets at 00:00 UTC |
| `source_busy` | 429 | Too many calls to that provider are already waiting |
| `source_unavailable` | 502 or 503 | The provider could not be reached or answered with an error (502), or it needs operator configuration (503) |
| `source_response_too_large` | 502 | The answer was over 2 MiB; narrow the request |
| `source_contract` | 502 | The provider answered in a shape that does not match the contract |
### Pace and allowances
Each provider has its own lane: its own concurrency, a minimum spacing
between requests, and a daily ceiling where the provider publishes one. Vitna
shares a provider's per-address allowance among all of its customers, so
these are set below the providers' published limits (all read 2026-10-07):
- SEC EDGAR allows 10 requests a second and requires every client to declare
a user agent naming its operator and a contact address. Vitna sends 5 a
second, and the provider is unavailable (`source_unavailable`, 503, with an
`unavailableReason` in its contracts) until the operator sets
`VITNA_WEB_SOURCES_CONTACT`.
- openFDA allows 240 requests a minute and, without a key, 1,000 a day per
address. Vitna keeps to 900 a day without a key, and 110,000 a day once the
operator sets `VITNA_WEB_SOURCES_OPENFDA_KEY`, against openFDA's 120,000.
- CourtListener allows keyless access to `/courts/` (`courts/list`), while
`opinions/search` and `dockets/search` use an API token supplied via
`VITNA_WEB_SOURCES_COURTLISTENER_KEY`.
- FCC ECFS and EIA v2 work out of the box with `DEMO_KEY` (900/day ceiling) and
scale when `VITNA_WEB_SOURCES_FCC_KEY` or `VITNA_WEB_SOURCES_EIA_KEY` is set.
- `dol-h1b` reads USCIS H-1B Data Hub employer CSVs directly for
`employers/approvals`, and uses `VITNA_WEB_SOURCES_DOL_KEY` for DOL OFLC
`lca/search`.
- FederalRegister.gov and eCFR.gov allow automated access only through their
APIs and publish no rate limit. Vitna sends 2 requests a second to each.
- ClinicalTrials.gov publishes no rate limit. Vitna spaces its requests 1.2
seconds apart.
- USAspending.gov publishes no rate limit for its search API. Vitna sends 4
requests a second, 2 at a time.
Requests identify themselves as
`VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview)`, followed by the
operator's contact address for SEC EDGAR. Reference lists (the SEC's ticker
map, the eCFR title index, USAspending's agency list, ClinicalTrials.gov's
processing date) are cached for an hour or a day once they load; a failed load
is never cached.
### Metering
`GET /v1/sources`, `GET /v1/sources/:provider` and `POST /v1/sources/find`
are free. They count once against the request limits like any request. A
`POST /v1/sources/run` that returns data adds one `sourceRequests` to usage
and to an account's statement; a call that fails is not metered. The ledger
rate is `VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD`, and every contract's
`price.microUsd` shows it (`null` while it is unset). The operator estimate
rate for static clients is `VITNA_WEB_SOURCE_REQUEST_USD`.
One call is one capability. Batches of calls, feedback that credits a call,
and a terms acceptance step for providers whose terms need one are not yet
available. None of the current providers asks for a separate acceptance step.
## MCP
```text
POST /v1/mcp
```
The Web API is also an MCP server, so an agent can find and call data sources,
read pages and, where the operator offers it, search Google through the tools
its client already understands. It uses the same `vitna_web_` key or org key as every other
route, sent as a bearer token:
```bash
claude mcp add --transport http vitna https://web.vitna.ai/v1/mcp \
--header "Authorization: Bearer $VITNA_WEB_KEY"
```
A client configured with JSON takes the same URL and header:
```json
{ "mcpServers": { "vitna": { "type": "http", "url": "https://web.vitna.ai/v1/mcp", "headers": { "Authorization": "Bearer vitna_web_..." } } } }
```
Connect to the API host itself. The website relay at `/api/web` does not
forward `/v1/mcp` or MCP's headers. `GET /v1/status` reports the endpoint and
the protocol versions it speaks in `capabilities.mcp`, and unauthenticated
`GET /.well-known/mcp.json`, `GET /.well-known/agent-card.json`,
`GET /openapi.json`, and `GET /llms.txt` expose the machine-readable schema
and full tool catalog before signup.
| Tool | What it does | Route it calls |
|---|---|---|
| `find_sources` | Ranks source capabilities for a task, each with its input schema and a working example | `POST /v1/sources/find` |
| `describe_source` | A provider's contracts, or one capability's, with output schemas and terms | `GET /v1/sources/:provider` |
| `run_source` | Calls one capability and returns its data with its source | `POST /v1/sources/run` |
| `read_page` | Reads one public page as markdown or text, cut at 20,000 characters unless `maxCharacters` says otherwise; also accepts `onlyMainContent`, `includeTags`, `excludeTags`, `highlights`, `question`, `extract`, `injection` and `redact` | `POST /v1/read` |
| `batch_scrape` | Reads up to 20 public pages in one call with shared reader, DOM filter, `highlights`, `question`, `extract`, `injection` and `redact` options; per-URL errors are isolated in `errors` | `POST /v1/batch` |
| `extract_data` | Pulls fields from one page, up to 20 pages, or pages discovered from a starting URL, by selectors, a JSON Schema, a `prompt` (where a language model is configured), or a saved recipe; waits for the result | `POST /v1/extract` |
| `map_site` | A site's URLs from its links and optionally its sitemaps, without reading them | `POST /v1/map` |
| `crawl_site` | Starts a background crawl and returns its job id: 25 pages of 20,000 characters each, for up to 5 minutes, unless asked otherwise; accepts per-page DOM filters, `highlights`, `question`, `extract`, `injection` and `redact` | `POST /v1/crawl` |
| `get_crawl` | A crawl's status and a slice of its pages, 25 at a time and without their text unless `includeContent` asks | `GET /v1/jobs/:id` |
| `cancel_crawl` | Stops a queued or running crawl; pages already read stay readable | `DELETE /v1/jobs/:id` |
| `research_web` | Runs bounded multi-page research across search hits or supplied `urls`, with optional `followLinks` and structured `schema` output; waits for the result | `POST /v1/research` |
| `list_open_roles` | A company's open roles from its Greenhouse, Lever, Ashby, Workable, Recruitee or Personio board: the first 25 postings without their text unless `maxJobs` or `descriptions` says otherwise | `POST /v1/careers` |
| `diff_pages` | Deterministic structural change detection and redlines across two URLs or inline text snapshots with materiality scoring and SHA-256 attestation | `POST /v1/diff` |
| `parse_document` | Parses PDF, DOCX, XLSX or PPTX into structured pages, typed blocks, tables, markdown and character citation offsets | `POST /v1/parse` |
| `verify_claim` | Deterministically verifies a factual claim against supplied URLs or search discovery with verbatim evidence offsets | `POST /v1/verify` |
| `entity_dossier` | Builds a live company alt-data dossier across web profile, ATS roles, IR releases, DNS/SaaS infra, SEC filings and news | `POST /v1/entity` |
| `enrich_grid` | Evaluates a synchronous entities-by-columns enrichment matrix across entity, search, verify and official source strategies | `POST /v1/grid` |
| `discover_grid` | Discovers and verifies up to 10 entities against explicit `criteria` with verbatim quotes and `sha256:` attestations under hard `budget` controls, with optional `columns` enrichment | `POST /v1/grid` |
| `investor_relations` | Discovers Investor Relations presentations, earnings releases, financial reports, transcripts and webcasts | `POST /v1/ir` |
| `list_firehose_events` | Reads normalized CISA KEV, Federal Register, FDA, Treasury, SEC 8-K, SC 13D and Form 4 event feeds | `GET /v1/firehose/events` |
| `capture_screenshot` | Renders a page in Chromium and returns a PNG or JPEG screenshot | `POST /v1/screenshot` |
| `capture_pdf` | Renders a page in Chromium to PDF with `pageRanges`, `scale` and `printBackground` controls | `POST /v1/pdf` |
| `list_linked_documents` | Discovers or downloads PDF, DOCX, XLSX and PPTX files linked from a page | `POST /v1/documents` |
| `search_index` | Ranked pages from Vitna's own index, with optional `domains`, `excludeDomains`, `maxAgeDays`, `asOf` snapshot cutoff, `ranking`, `effort`, inline `contents`, and one-call live `scrape` | `POST /v1/search` |
| `google_search` | Google's results page for a query, in SerpApi's field names unless `shape` asks for Vitna's own | `POST /v1/serp` on the search service |
The source tools are listed only while sources are enabled,
`search_index` and `discover_grid` only where a search engine is configured, and
`google_search` only where the operator names a search service in
`VITNA_WEB_SERP_URL`. `read_page`, `batch_scrape`,
`list_open_roles` and the crawl tools ask for less than their routes do by
default, because a whole page, job board or crawl can be more than a model
should take in one result. A crawl tool's job deadline is the exception: it
defaults to five minutes, where the route's defaults to one page's 30
seconds. `extract_data` does not offer the route's background mode; it waits
for the result within the request deadline. Every tool is
read-only except `crawl_site` and `cancel_crawl`, which start and stop a
background crawl. A tool call is the REST request it stands for, sent with the
caller's key, so it counts against the same request and page limits, is
metered the same way (`find_sources` and `describe_source` are free, and a
`run_source` call is one `sourceRequests` once it returns data), and appears
in `/v1/usage` under the route's own operation.
`google_search` is the exception, because its route is not this service's.
The call goes to the search service's `POST /v1/serp` with the caller's own
org key, `vitna_key_...` (see [one key](../one-key.md)), so the search
service admits it, counts it against the org's search limits, applies its
deadline, cache and `fresh` captures, and logs it, exactly as a direct
request. It does not count against this service's request limit or appear in
`/v1/usage`. A `vitna_web_` key covers the Web APIs only and is not sent on:
the tool answers `platform_key_required`.
Its arguments are SerpApi's, mapped onto the route's fields:
| Argument | Route field | Allowed values |
|---|---|---|
| `q` | `query` | Required; 1 to 1,000 characters |
| `gl` | `country` | Two-letter country code, such as `us` |
| `hl` | `language` | Language tag, such as `en` or `pt-BR` |
| `start` | `offset` | 0 to 190; 0 unless given |
| `num` | `maxResults` | 1 to 10; 10 unless given |
| `extras` | `extras` | `true` adds People Also Ask questions, related searches and ads |
| `no_cache` | `fresh` | `true` captures a new page rather than reuse the key's recent answer |
| `device` | None | `desktop` only. Vitna captures Google's desktop results page and has no mobile or tablet capture, so the tool refuses either as `invalid_arguments` |
| `shape` | `shape` | `serpapi` unless given, or `vitna` for the route's default fields |
The result is the route's answer as `structuredContent`, in the shape
described under [SerpApi-shaped response](../search/serp.md#serpapi-shaped-response).
When Google does not answer and the search service hands the search to Bing,
the result says so: `meta.engine` is `bing`, and in SerpApi's shape
`search_metadata` gives `bing_url` instead of `google_url`. A search can take
up to the search service's deadline, about a minute on the hosted service, so
give the client's tool-call timeout at least that.
Results carry the route's response as `structuredContent` and as JSON text.
When a route refuses a call, the tool result has `isError: true` and its text
gives the route's code and message, such as
`invalid_options: options.title must be at most 50.` or, for a rate limit,
the number of seconds to wait. Arguments that break a tool's input schema are
refused the same way, as `invalid_arguments`. An unknown tool or a malformed
message is a JSON-RPC protocol error.
### Protocol versions
The endpoint speaks protocol `2026-07-28` and the earlier `2025-11-25`,
`2025-06-18` and `2025-03-26`, and keeps no session for any of them:
- A `2026-07-28` request carries `io.modelcontextprotocol/protocolVersion` and
`io.modelcontextprotocol/clientCapabilities` in `params._meta`, and mirrors
its version, method and tool name into the `MCP-Protocol-Version`,
`Mcp-Method` and `Mcp-Name` headers. A header that is missing or does not
match the body is refused with `-32020` (400), missing metadata with
`-32602` (400), and an unknown method with `-32601` (404).
`server/discover` reports the supported versions and the server's identity.
- An earlier client opens with `initialize` and is answered with the version
it asked for, or `2025-11-25` when it asked for one this server does not
speak. Later requests name the version in `MCP-Protocol-Version`; a request
without the header is read as `2025-03-26`. No `Mcp-Session-Id` is issued,
and one sent by the client is ignored.
- A version the server does not speak is refused with `-32022` (400), whose
`data.supported` lists the versions it does.
Each POST carries one JSON-RPC message; batches are refused. Responses are
always a single JSON object, never an SSE stream, so `GET` and `DELETE` answer
`405`. A request carrying an `Origin` header is refused with `403` unless the
operator lists that origin in `VITNA_WEB_MCP_ORIGINS` (comma-separated), so
only the browser origins the operator names can call it from a web page.
## Capabilities and usage
```text
GET /v1/status
GET /v1/usage
```
Use the same bearer key to inspect the configured capabilities and limits,
or your client usage in the current UTC day. The status response indicates
which engines and search discovery are configured, and `capabilities.documents`
lists the document kinds read (`limits.documentBytes` and
`limits.documentCharacters` bound them). Inspect `engines`,
`searchIndex`, and Research `generation` metadata to distinguish native
extractive behavior from a selected self-hosted model. The usage
response includes request and page-attempt counts, limits, and recent
request outcomes. Resource counters include browser milliseconds, render
attempts, Search calls, answer/extraction-engine calls, the bytes moved
through residential and ISP routes (`residentialBytes`, `ispBytes`), and the
bytes an account's own proxy keys moved (`proxyBytes`), and the firehose
records returned (`firehoseRecords`). `searchRequests`
and `modelRequests` also count native in-process invocations; these names
do not imply an external network request.
Static client keys have no usage statement. API accounts can retain a
metered usage ledger; payment collection is disabled.
### Operator cost estimates
Optional rates let an operator estimate measured resource usage:
```dotenv
VITNA_WEB_BROWSER_SECOND_USD=0.001
VITNA_WEB_SEARCH_REQUEST_USD=0.005
VITNA_WEB_MODEL_REQUEST_USD=0.01
VITNA_WEB_RESIDENTIAL_GB_USD=5
VITNA_WEB_ISP_GB_USD=2
VITNA_WEB_PROXY_GB_USD=3
VITNA_WEB_SOURCE_REQUEST_USD=0.002
VITNA_WEB_FIREHOSE_RECORD_USD=0.0002
```
These rates are illustrative configuration values, not Vitna prices. Each
accepts a nonnegative USD rate per measured browser second, dispatched
Search/model-engine invocation, source call that returned data, firehose
record returned, or decimal GB
(10^9 bytes) moved through a residential or ISP route or on an account's proxy
keys. `/v1/usage` returns `resources` and a `cost` object
with `estimatedUsd`, `rates`, `basis: "operator_rates"`, and `complete`.
When all rates are unconfigured, `estimatedUsd` is `null`. With only some
rates configured, the estimate covers those resources and `complete` is
false when an observed resource has no rate. It does not imply that the
unpriced resources cost zero.
The estimate covers only the configured rate model. It is not a payment
charge, a provider invoice, or a measurement of hosting and proxy bandwidth
costs. Use provider billing records to reconcile external spend.
### Account usage statements
Activated accounts retain idempotent resource observations and daily totals
in the Vitna SQLite ledger. Browser durations, render attempts, Search calls,
answer-engine calls, source calls that returned data, firehose records
returned, bytes through
residential and ISP routes, and bytes on the account's proxy keys are
attributed to the account. Ledger pricing uses
separate nonnegative integer microdollar rates, with one USD equal to
1,000,000 microdollars:
```dotenv
VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD=1000
VITNA_WEB_BILLING_SEARCH_REQUEST_MICRO_USD=5000
VITNA_WEB_BILLING_MODEL_REQUEST_MICRO_USD=10000
VITNA_WEB_BILLING_RESIDENTIAL_GB_MICRO_USD=5000000
VITNA_WEB_BILLING_ISP_GB_MICRO_USD=2000000
VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000
VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD=2000
VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD=200
```
These are illustrative operator values. Browser observations round up to a
whole millisecond, and each event charge rounds up to a whole microdollar.
Route bytes are priced per decimal GB for every attempt that moved them,
delivered or not, because the operator's provider bills them that way.
The monthly `/v1/invoices/YYYY-MM` owner route returns usage, observed event
counts, priced totals, rate completeness, and daily entries. Its
`kind` is `usage_statement` and `paymentCollection` is `false`.
`pricedMicroUsd` covers resources with a configured rate;
`totalMicroUsd` stays `null` when an observed resource has no rate or
accounting is incomplete. A current month remains `accruing`.
Check `complete` and `billingStatus` before treating a statement as settled.
Failed or pending accounting can mark the statement incomplete even when
an API operation returned successfully. The service retains pending
observations for recovery; it does not silently report a complete bill for
unrecorded usage. Ledger records have a bounded 400-day retention window;
statements outside a fully retained month return `invoice_unavailable`.
These statements do not charge a card, debit a prepaid balance, reconcile
hosting bills, or deliver customer payment collection.
## Native engines and optional endpoints
The default backends are in process:
```dotenv
VITNA_WEB_SEARCH_BACKEND=native
VITNA_WEB_MODEL_BACKEND=native
VITNA_WEB_SEARCH_RERANK=true
```
Native retrieval uses Vitna's SQLite FTS5 index with title-weighted BM25
ranking and an optional local lexical reranking pass. Its file defaults to
`search.sqlite` under `VITNA_WEB_DATA`; override it with
`VITNA_WEB_SEARCH_DB`. Starting the service opens that index but does not
crawl the web or add example documents. Empty or unrelated indexes can
return no matches.
Populate selected public HTTPS sources explicitly:
```bash
node --env-file=.env.web platform/web/native-search-crawl.mjs --seed https://example.com/ --max-pages 50 --max-requests 100
```
Repeat `--seed` for up to 100 seed URLs, or use `--catalog file.json` with
the existing Vitna Search source catalogue format. Seed URLs cannot contain
query strings. Crawls are bounded by `--max-requests`, `--duration-ms`, and
`--timeout-ms`, respect the source scope, and add or refresh selected sources
without removing other sites from the index. This crawler fetches public
pages; it does not execute JavaScript. Use browser APIs for rendered pages.
With configuration already in the environment, the equivalent script is
available as:
```bash
npm run web:index -- --seed https://example.com/
```
To refresh an explicit source list in the service, configure exactly one of
`VITNA_WEB_SEARCH_SEEDS` (a JSON array of public HTTPS seed URLs) or
`VITNA_WEB_SEARCH_CATALOG` (a local Search catalog file):
```dotenv
VITNA_WEB_SEARCH_SEEDS=["https://example.com/docs/"]
VITNA_WEB_SEARCH_REFRESH=true
VITNA_WEB_SEARCH_REFRESH_MS=300000
VITNA_WEB_SEARCH_REFRESH_REQUESTS=30
VITNA_WEB_SEARCH_REFRESH_DURATION_MS=120000
```
Configured seeds or a catalog enable maintenance unless refresh is explicitly
`false`. With neither supplied, startup makes no refresh requests. Due sources
are selected by their oldest live page and catalog `refreshHours`, with bounded
requests, duration, source batches, robots rules, conditional fetches, and the
existing crawl lease. The default batch has three sources and seed sources
have twenty pages each. `searchIndex.maintenance` exposes safe run summaries.
Query text cannot enlarge this operator-defined source scope.
Native Web search returns at most 20 candidates per query. It supports
`site:hostname` and `-site:hostname` filters. Retrieval and reranking remain
lexical; they do not establish semantic understanding or a global web index.
Native Search applies bounded evidence admission, with `relevanceMode` set
to `balanced` by default or explicitly to `strict`. Balanced mode can retain
useful partial wording but may miss unsupported constraints; strict mode
requires all normalized content terms and can lose paraphrases.
Native Research and schema extraction use no external model endpoint. They
support observed facts, finite comparison rules, and compatible numeric
calculations, and refuse missing or ambiguous evidence. They do not provide
general language-model reasoning.
For a separate Vitna Search process, select the service backend explicitly:
```dotenv
VITNA_WEB_SEARCH_BACKEND=service
VITNA_WEB_SEARCH_URL=http://127.0.0.1:8789
VITNA_WEB_SEARCH_KEY=YOUR_SEARCH_CLIENT_KEY
VITNA_WEB_SEARCH_MODE=index
VITNA_WEB_SEARCH_RERANK=true
```
Web discovery uses indexed search. The separate Search service retains its
own SERP contract, but the Web engine does not select a paid provider or
SERP fallback. The service adapter returns up to ten results per query;
larger source budgets use multiple queries or supplied URLs. The Search
service applies its own default retrieval: hybrid where its
[semantic retrieval](../search/overview.md#semantic-retrieval) is configured,
keyword otherwise.
The MCP tool `google_search` is offered only when the operator names a
Search service whose SERP route it calls, whichever backend serves index
search:
```dotenv
VITNA_WEB_SERP_URL=http://127.0.0.1:8789
```
Each call carries the caller's own org key there, never
`VITNA_WEB_SEARCH_KEY`, so the URL must be https, or http on a loopback
address. That Search service must have SERP retrieval enabled and accept org
keys, which takes the same `VITNA_PLATFORM_VERIFY_URL` and
`VITNA_PLATFORM_SERVICE_TOKEN` there as here.
For language-based extraction or synthesized research answers, select an
operator-owned compatible model runtime. `MODEL_URL` is its full chat
completions endpoint; model name and optional private key are separate:
```dotenv
VITNA_WEB_MODEL_BACKEND=self-hosted
VITNA_WEB_MODEL_URL=http://127.0.0.1:8000/v1/chat/completions
VITNA_WEB_MODEL_KEY=YOUR_RUNTIME_KEY
VITNA_WEB_MODEL=YOUR_MODEL_NAME
```
The server requests JSON output, validates extraction against the schema,
and checks that research answers reference the pages supplied to the model.
Runtime credentials stay on the server. Setting endpoint variables requires
the corresponding explicit backend; native mode does not silently call
them. Each backend also accepts `disabled` to turn off that engine.
## Deployment boundaries
The suite is an operator-configured service. Keep its client hash map,
invitation hashes, and optional model/Search credentials in the service environment.
Expose it through a TLS reverse proxy when remote access is needed. Website
relay configuration belongs in the website environment.
The data directory configured with `VITNA_WEB_DATA` stores crawl jobs,
monitors, quotas, and recent usage. Keep that directory across process
restarts and run one service writer per directory. State is stored in
`state.sqlite` using SQLite WAL transactions. A legacy `state.json` is
imported once into a new database and retained as a recovery backup. Stop
the service before copying the complete data directory for a filesystem
backup, so the database and its journal files are captured consistently.
The default request limits are 30 requests per minute, 1,000 requests per day, and 2,000 page
reads per day, with two concurrent operations. Configure them using
`VITNA_WEB_RPM`, `VITNA_WEB_DAILY_LIMIT`, `VITNA_WEB_DAILY_PAGES`, and
`VITNA_WEB_CONCURRENCY`. `VITNA_WEB_TIMEOUT_MS` sets the default browser
deadline, initially 30,000 milliseconds.
The browser service restricts requests to public HTTPS destinations and
checks redirect destinations and resolved addresses. Rendering does not
guarantee that a website will be reachable or that every delayed field will
be ready. Choose a content selector with `waitFor` where appropriate.
See [Unblocker setup](../unblocker/overview.md) for browser installation and
proxy configuration, and [Search setup](../search/overview.md) for the
separate search service.
The deployment package includes `deploy/vitna-web.env.example`,
`deploy/vitna-web.service`, `deploy/vitna-web.nginx.conf.example`, and
`deploy/vitna-web.Dockerfile`. These prepare a deployment; they do not publish
a hosted endpoint. Browser settings use `VITNA_WEB_BROWSER_PATH`,
`VITNA_WEB_BROWSER_CHANNEL`, and `VITNA_WEB_BROWSER_PROFILE`. Its `auto`
reads in Chromium and moves a challenged read, page or document, to Camoufox, and it and
`camoufox` take `VITNA_WEB_CAMOUFOX_PYTHON`, `VITNA_WEB_CAMOUFOX_FINGERPRINTS`
and `VITNA_WEB_CAMOUFOX_OS`, with the meanings the
[Unblocker's](../unblocker/overview.md#the-auto-profile) settings of those
names have; owned browser workers ignore all four. A service whose `HOME`
does not last, as a systemd runtime directory does not, also needs
`XDG_CACHE_HOME` set to the cache Camoufox's browser was fetched into; it
reaches the fingerprint helper and no browser. The hosted Web API reads the
hosted Unblocker's install: `deploy/web-api/install-camoufox.sh` gives
`/opt/vitna-camoufox` the Web API user's group, starts Camoufox as that user
under its unit's confinement, and prints the lines its env file needs
(`deploy/web-api/OVH-RUNBOOK.md`, The Web API's auto browser profile).
Each request's line in the service's log names the host it read, the engine
that served the read (`chromium`, `camoufox` or `cloakbrowser`) and its
attempts, with `challenged` when a challenge sent the read on; a request that
read more than once, as `/v1/documents` does, also lists each read in `reads`
(up to 20, a failed one with its `code`) and counts them in `readCount`. A
job's reads are not on the line of the request that started it. vitna.ai keeps
the host, engine and attempt count from an org's receipt, as it does from the
Unblocker's. `/v1/usage` lists the requests alone.
Proxy routes use `VITNA_WEB_PROXIES` or labelled `VITNA_WEB_PROXY_ROUTES`,
and an optional
exact-host scope uses
`VITNA_WEB_ALLOWED_HOSTS`.
For labelled routes, configure private operator credentials:
```dotenv
VITNA_WEB_PROXY_ROUTES=[{"id":"us-primary","country":"US","proxy":"https://user:[email protected]:8443"}]
```
The operator supplies and labels these routes. Country selection does not
provision a proxy network or verify the provider location. Residential and
ISP accounts from a provider the operator holds take the same list, with
credential templates; see
[proxy routes](../unblocker/overview.md#proxy-routes).
Keep proxy and integration credentials in private service configuration.
The supplied systemd unit and Docker image run one process with two browser
slots by default. Configure queue, page, request, and job limits for your
machine, and verify representative workloads before increasing them.
SQLite persistence and larger budgets do not by themselves establish
coordinator high availability or a hosted service. Owned browser workers
can distribute rendering, as described under
[owned browser workers](#owned-browser-workers). The selected Vitna Search backend
reads its populated index. The web suite does not provision a global web
index. Owned CONNECT gateways can supply egress routes; see
[owned egress setup](../unblocker/overview.md#run-an-owned-egress-gateway).
## Operators: running the service
This section is for the operator of a Web API service and needs a checkout of
Vitna's private repository. Nothing in it is needed to use the hosted pilot
at `https://web.vitna.ai`, and it is not an offer of a self-hosted
deployment. Until 2026-10-11 it opened this page as "Run locally", and the
invitation, signup, recovery, quota, worker, metrics and benchmark material
joined it from the customer half of this page the same day.
The web suite uses the browser runtime installed for Vitna Unblocker. It has
its own client keys and service process.
```bash
npm run web:install
npm run web:browser
npm run web:demo
```
The demo uses a real browser against local illustrative pages. It does not
contact a target website or a paid model. To start your own service, mint a
client key:
```bash
npm run web:key -- pilot
```
Keep the printed token private. Put the printed client hash, rather than the
token, into an ignored `.env.web` file:
```dotenv
VITNA_WEB_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"}
VITNA_WEB_PORT=8790
VITNA_WEB_DATA=.web-data
```
Start the service with that configuration:
```bash
node --env-file=.env.web platform/web/start.mjs
```
When the settings are already in the environment, use:
```bash
npm run web:start
```
The default service address is `http://127.0.0.1:8790`. Browser operations
use the installed Chromium runtime. Native engines make no model or search
provider call at startup. Invite activation and owner key management are
available through the account console when the website relay is configured.
Customer payment collection and open self-service registration remain unavailable.
### Issuing invitations
Static keys from `web:key` remain supported. For controlled account
activation, create an operator invitation:
```bash
npm run web:invite -- --client customer-01 --display-name "Customer application"
```
The command prints a one-time `activationToken` and an `invites` array. It
does not contact the service or activate an account. Put only the returned
hashed `invites` array in `VITNA_WEB_INVITES`, then start the service. Keep
the activation token private. An account client identifier must differ from
every static client identifier in `VITNA_WEB_CLIENTS`.
The CLI defaults to a 24-hour invitation, a 90-day API-key expiry, and a
366-day owner-token expiry, measured from invitation creation. Options
`--expires-hours`, `--key-days`, and `--owner-days` can shorten those windows;
the maximum invitation lifetime is 720 hours.
### Optional public signup
Public signup is disabled by default. Enable it explicitly with
`VITNA_WEB_SIGNUP_ENABLED=true` and a private, stable 32-byte hexadecimal
`VITNA_WEB_SIGNUP_SECRET`. `GET /v1/accounts/signup` reports safe availability
metadata. The account console at `/apis/account` offers signup when enabled.
Request a short-lived challenge with `POST /v1/accounts/signup/challenge`
and an empty JSON object. The response specifies a SHA-256 proof over the
UTF-8 bytes of `challenge + ":" + nonce`, with 18 leading zero bits and a
bounded integer nonce. Submit the challenge, nonce and optional label to
`POST /v1/accounts/signup`. This proof adds computational friction; it does
not verify a human, email address or identity. No email or payment service
is involved.
Account creation, signup counters and a hash-only replay receipt commit in
one SQLite transaction. The successful response returns the API key and
owner token once. Repeating the same successful signup returns public
account metadata with `duplicate:true` and `credentialsUnavailable:true`;
it cannot recover the credentials. Keep the original response privately.
Default issuance limits are three challenges per minute and 20 per UTC day
per network bucket. Signup allows three accounts per network bucket per
UTC day, 100 per coordinator per day and at most 1,000 retained accounts. IPv4 /24
and IPv6 /64 buckets are HMAC-hashed before storage. Failed proof attempts,
retained network counters and replay receipts also have separate bounds.
Explicit positive integer envs can lower or adjust coordinator capacity:
`VITNA_WEB_SIGNUP_SIGNUPS_PER_DAY` (1 to 100,000) and
`VITNA_WEB_SIGNUP_MAX_ACCOUNTS` (1 to 1,000). The OVH pilot ran with
20 signups per UTC day and at most 100 accounts until the owner closed
signup on 2026-10-11. These limits apply across
network buckets and count existing native accounts; the account cap does not
reset at midnight. Omitted variables preserve the defaults. Configuring
capacities while signup is disabled, or using noncanonical/out-of-range
integers, rejects startup. Set them together with explicit enablement.
The server uses its transport peer address. Headers supplied by a caller
cannot change that address. Behind a reverse proxy, configure exact peer
IPs in `VITNA_WEB_TRUSTED_CLIENT_PROXIES` and have that proxy overwrite
`X-Vitna-Client-IP` with its observed client address. Missing or malformed
identity from a trusted proxy fails closed. For website relays, also configure
exact normalized egress IPs in `VITNA_WEB_TRUSTED_CLIENT_RELAYS` and the
dedicated token SHA-256 in `VITNA_WEB_CLIENT_RELAY_TOKEN_HASH`. The relay sends
`X-Vitna-Relay-Client-IP` plus `X-Vitna-Relay-Token`; the API checks its trusted
socket, nginx-observed relay address and token before using the visitor IP.
A configured relay without an authenticated assertion fails closed. The Next
relay additionally requires a distinct nginx ingress token authenticated by
`VITNA_WEB_RELAY_INGRESS_TOKEN_HASH`, and its private outbound
`VITNA_WEB_RELAY_CLIENT_TOKEN`. It requires HTTPS beyond literal loopback.
Nginx must overwrite ingress identity and establish trusted Cloudflare peers
before reading Cloudflare identity. Caller `X-Forwarded-For` or
`CF-Connecting-IP` headers alone are never trusted.
### Offline operator recovery
If the owner token is lost or expired, or the activation response was lost
after the invitation was redeemed, recover access through the operator's
local account CLI. Stop the Web service first. The CLI opens the same
single-writer SQLite data directory and refuses its live-writer lock;
do not bypass that lock. Rotate the owner token by client identity:
```bash
npm run web:account -- --data .web-data owner:rotate --client customer-01
```
The result contains a new `ownerToken` and `ownerExpiresAt`. Save the token
privately, then restart the service so its credential caches reflect the
change. The old token is invalidated. This recovery requires filesystem
access to the data directory; it does not require the lost owner token or
an HTTP admin credential. If `--data` is omitted, the CLI uses
`VITNA_WEB_DATA`, then `.web-data`.
Other offline commands inspect account/key metadata or replace and revoke
an API key:
```bash
npm run web:account -- --data .web-data accounts:list
npm run web:account -- --data .web-data keys:list --client customer-01
npm run web:account -- --data .web-data key:rotate --client customer-01 --key REPLACE_WITH_KEY_UUID
npm run web:account -- --data .web-data key:revoke --client customer-01 --key REPLACE_WITH_KEY_UUID
```
Lists contain public metadata and hide credential hashes. `key:rotate`
returns the replacement `apiKey` and key metadata once. Stop the service
before these commands and restart it after mutations, preserving the
single-writer boundary and refreshing cached credentials.
### Shared quota admission
The owned quota authority coordinates admission across Web and Unblocker
API processes over authenticated HTTP. Mint a separate operator credential
and start a single authority with explicit authoritative service limits:
```bash
npm run quota:key
npm run quota:start
```
The authority requires `VITNA_QUOTA_TOKEN_HASH` and `VITNA_QUOTA_SERVICES`;
its private durable directory defaults to `.quota-data`, port to 8793 and
host to loopback. `deploy/vitna-quota.env.example` documents service and
per-client limit configuration. Non-loopback listening requires configured
TLS certificate and key files. API processes explicitly select
`VITNA_WEB_QUOTA_BACKEND=shared`, `VITNA_WEB_QUOTA_URL` and
`VITNA_WEB_QUOTA_TOKEN`; standalone Unblocker uses the corresponding
`VITNA_UNBLOCKER_QUOTA_*` variables. Partial configurations are rejected.
Authority limits override local quota settings. Each admitted API request
and each browser attempt reserves its own atomic, idempotent operation.
Retries consume page-attempt quota. Status/usage reads are free, scoped by
service/client, and report the authority's limits and counters. Web resource
timings and recent-operation lists remain scoped to their coordinator.
Unreachable authorities fail closed before browser or search dispatch;
uncertain admissions are never automatically replayed or failed over.
Deduplication IDs contain bounded timestamps and are retained for 24 hours.
The same ID with a changed admission conflicts; an expired ID cannot become
a new charge after pruning. A denied operation stays denied on replay even
after quota rollover. New logical work needs a new ID. Backwards clocks
cannot reset consumed windows. Counters/events, queues and HTTP bodies are
bounded, and quota changes plus deduplication commit in one SQLite
transaction.
The authority requires a dedicated private directory and rejects linked
storage, reused account state and foreign database schemas before opening
them for writes. Existing POSIX ownership/modes must be private; Windows
operators must configure private ACLs. Run one writer per authority directory.
This adds shared admission, not full coordinator or authority high
availability. Account credentials,
usage statements, jobs and monitors still belong to one API coordinator;
separate API directories do not replicate that state. Shared quotas do not
make public signup accounts portable across coordinators. Session workers
also retain their own writer directories and stable routing.
### Owned browser workers
The default `VITNA_WEB_BROWSER_BACKEND=native` executes Chromium in the Web
process. To separate browser execution onto machines you operate, run
`npm run web:worker` on each worker and explicitly select the worker backend:
```dotenv
VITNA_WEB_BROWSER_BACKEND=workers
VITNA_WEB_BROWSER_WORKERS=[{"id":"worker-a","origin":"https://worker-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","concurrency":2}]
```
Workers require `VITNA_WEB_BROWSER_WORKER_TOKEN_HASH`. Generate its SHA-256
hash with `npm run web:worker:hash`, reading the token from private
`VITNA_WEB_BROWSER_WORKER_TOKEN` configuration. Use a distinct operator token,
not a customer API key. Nonlocal worker binds require the worker TLS key and
certificate files. Loopback HTTP is supported for local development.
```bash
npm run web:worker:hash
npm run web:worker
```
`deploy/vitna-browser-worker.env.example`, the systemd unit, and Dockerfile
provide the full configuration and deployment templates.
The pool supports up to twenty fixed workers, bounded queues, and explicit
per-worker concurrency. Tenant/session hashing keeps persistent sessions on
one worker while the configured worker list and order remain stable. Session
cookies and localStorage stay in that worker's RAM and expire or disappear
on worker restart. Clearing a session uses a separate bounded control queue.
The pool does not replay a failed browser invocation on another worker.
Without that replay, a read through workers does not step up a network
after a block; name `network` to use a dearer tier. Workers report the
network and transferred bytes of each attempt, delivered or not, so the
coordinator prices residential and ISP traffic as it would its own.
Status reports `engines.browser` and whether execution is local or distributed.
The API coordinator and SQLite directory still require one writer; this
provides browser distribution without coordinator high availability.
The same worker process can also hold Browser API sessions: set
`VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS` on it and list it in the Web
process's `VITNA_WEB_BROWSER_API_WORKERS`. See
[sessions on owned workers](#sessions-on-owned-workers).
### Operator metrics
Generate a separate operator token with `npm run web:metrics:key`. Retain the
token privately and configure only its SHA-256 hash as
`VITNA_WEB_METRICS_TOKEN_HASH`. Authenticated `GET /metrics` returns Prometheus
text; use the token as a bearer credential. This route rejects queries and
bodies and is disabled when no hash is configured. Customer keys cannot scrape it.
```bash
npm run web:metrics:key
```
Counters and histograms cover HTTP outcomes and full response transfer, current
requests, bounded queues, browser duration, engine invocations, resource bytes,
and ledger pricing observations. Labels use fixed operation, method, status,
and service values; URLs and customer identifiers are excluded. Counters reset
on process restart. `priced_micro_usd_total` includes known partial prices;
`unpriced_events_total` tracks incomplete pricing instead of treating it as zero.
These are process observations, not a replacement for the durable account ledger.
`deploy/vitna-api-alerts.yml` supplies starter availability, error-rate,
latency, queue, and metric-cardinality rules. Load it into your Prometheus
configuration and configure your own alert delivery. The file creates no
monitoring service or notification destination.
### Benchmark locally
Run a repeatable baseline after installing Chromium:
```bash
npm run web:benchmark -- --iterations 10 --concurrency 2
```
The runner measures every Web API product, both screenshot formats, and
the existing JS Unblocker through real local HTTP requests and real Chromium.
Each operation uses a fresh durable store. Pages and assets are fulfilled
locally, with delayed JavaScript, article content, product variants, and
company data. No external model, search provider, webhook, or proxy is used.
For a separate public-site coverage sample, run:
```bash
npm run web:benchmark -- --only coverage --report outputs/web-benchmark/public-coverage.json
```
This makes two sequential Reader requests each to Python, React, MDN,
GitHub, Hacker News, and an arXiv abstract, using automatic content readiness.
Each response must contain a specified content marker, a minimum content
length, and no truncation. It is a small diagnostic across six public sites,
not a representative sample of web coverage.
The JSON report and readable HTML table are saved under
`outputs/web-benchmark/`. Use `--report path/to/report.json` to choose another
location. One warmup per operation is excluded. The measured sample count
accepts 1 to 50 and client concurrency accepts 1 to 4; the service keeps its
default two browser slots. Latency includes browser startup, cleanup, durable
writes, and full JSON transfer and decoding. Successful latency and failure
counts are separate, and success requires verified output rather than only
an HTTP 200 response.
Crawl latency includes completion of three pages and polling every 100 ms.
Monitor measures unchanged selector checks against independent monitors;
baseline creation and deletion are excluded. Research measures collection
of one supplied source, without search or generation. A ten-request sample
is an initial baseline: its nearest-rank p95 equals its slowest observation.
To include two sequential reader requests to each of three public training
pages (static quotes, JavaScript quotes, and a book catalog), add `--live`:
```bash
npm run web:benchmark -- --iterations 10 --concurrency 2 --live
```
To repeat selected measurements, use `--only reader,crawl` with operation
IDs from the report, or `--only public` to run only the six public samples.
The report also measures synchronous storage-write time so slow jobs can
be distinguished from browser and network delays. Monitor setup is excluded,
and render-attempt concurrency includes startup and cleanup.
Public samples remain separate from the controlled results. This benchmark
does not establish representative website coverage, competitor performance,
hosted capacity, or external model and search-provider latency.
`npm run web:load` exercises actual local HTTP servers with a controlled
five-millisecond fixture renderer, bounded concurrency, output validation,
overload rejection, and a quota restart check. Its saved latency and throughput
measure the API transport and admission path, not Chromium or production
website capacity. `npm run search:relevance` separately evaluates authored
relevance and abstention fixtures. Keep these scopes separate when comparing
performance.
```bash
npm run web:load
npm run search:relevance
```