Get started
DocumentationWeb APIs

Vitna Web APIs

Source

Hosted pilot at https://web.vitna.ai (OpenAPI at /openapi.json, MCP manifest at /.well-known/mcp.json, Agent Card at /.well-known/agent-card.json), with keys by invitation while public signup is closed. Turn public web pages into Markdown, structured JSON, crawl datasets, change alerts (diffMode), structural redlines (/v1/diff), parsed PDF/Office documents (/v1/parse), company dossiers (/v1/entity, /v1/ir), cited enrichment matrices (/v1/grid), claim verifications (/v1/verify), structured registry firehose events (/v1/firehose/events), screenshots, and printable PDFs (pageRanges, scale, printBackground), alongside 16 official data sources (courtlistener, SEC EDGAR, Federal Register, CISA KEV, FRED, FINRA, Treasury, NY Fed, BLS, and more) and 25 MCP tools (/v1/mcp). Native retrieval, lexical reranking, observed field extraction, and bounded research analysis run in the Vitna process by default. The product page and request playground are at /apis. The native customer account console is at /apis/account.

To get a key, request an invitation at vitna.ai/start; a person reads the request. Activate the invitation on the account page, which shows an API key and an owner token once each. Nothing is billed during the pilot: usage is measured on a monthly statement, no rates are set, and payment collection is off. An org's platform key (vitna_key_) works too; see One key for the hosted APIs. Material for the people who operate the service is collected at the end, under Operators: running the service.

Authentication and the website relay

The hosted API answers at https://web.vitna.ai. Since 2026-10-07 a browser that opens it, or its /docs, is sent to /apis and this reference on vitna.ai, where the product's pages live: the operator sets VITNA_WEB_PAGE_URL and VITNA_WEB_DOCS_URL, and a self-hosted instance with neither set serves its own reference page.

Web operations use a vitna_web_ bearer key:

bash
curl http://127.0.0.1:8790/v1/read \
  -H "Authorization: Bearer $VITNA_WEB_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/","format":"markdown","render":true}'

The website can forward requests to the standalone service. Set VITNA_WEB_SERVICE_URL in the website environment, for example to http://127.0.0.1:8790. The relay forwards the caller key and maps /api/web/read to /v1/read. It also supports the job and monitor routes listed below. Every route carries a credential, so the relay sends nothing over plain HTTP except to a literal loopback address (127.0.0.1 or [::1]); any other service URL must be HTTPS, and the relay answers 503 otherwise.

The website relay is mounted as a catch-all route:

text
GET /api/web/[...path]
POST /api/web/[...path]
DELETE /api/web/[...path]

Replace [...path] with the supported operation or job path when making a request, such as read or jobs/JOB_ID.

With no service configured, the relay returns an explicit configuration error. The /apis playground starts with illustrative output; Execute live request sends a real request through the relay. Its password field keeps the key in component memory, with no browser storage.

API accounts and keys

An operator issues each invitation (see issuing invitations). It carries a one-time activation token, which lasts 24 hours by default. Redeem it once, on the account page or directly:

bash
curl http://127.0.0.1:8790/v1/accounts/activate \
  -H 'Content-Type: application/json' \
  -d '{"activationToken":"REPLACE_WITH_PRIVATE_ACTIVATION_TOKEN","label":"Application"}'

The HTTP 201 response contains apiKey, ownerToken, account metadata, and initial key metadata. Save the returned tokens privately; subsequent reads do not return them. API keys use vitna_web_; owner tokens use vitna_web_owner_. The owner token manages the account and keys, while an API key executes the web operations.

EndpointOwner-token operation
GET /v1/accountAccount, key metadata, and accounting status
POST /v1/account/rotateReplace the owner token; send an empty JSON object
GET /v1/keysList active, expired, and revoked key metadata
POST /v1/keysIssue a key with optional label and expiresAt
POST /v1/keys/:id/rotateReplace that key, invalidating its previous token
DELETE /v1/keys/:idRevoke that key
GET /v1/invoices/YYYY-MMMetered monthly usage statement

Use Authorization: Bearer $VITNA_WEB_OWNER_TOKEN for those routes. Creation and rotation bodies accept optional label and an ISO UTC expiresAt, such as 2026-12-01T00:00:00.000Z. Labels accept up to 80 characters. API keys expire within 366 days; up to 20 keys per account are retained. New requests reject revoked and expired tokens immediately. Owner rotation invalidates the old owner token. Activation requests are limited to 10 per minute per process; owner operations to 60 per minute per account. These management routes use JSON bodies up to 4,096 bytes and do not accept query parameters.

The SQLite store retains account metadata and credential hashes. Replaying the startup invitation configuration cannot redeem a used invitation again or restore a rotated credential. Customer payments are not collected.

Public signup is closed on the hosted pilot, and keys are issued by invitation. GET /v1/accounts/signup reports whether it is open; the operator's switch is described under optional public signup.

A lost owner token cannot be shown again. Ask through the contact form and an operator issues a new one, as described under offline operator recovery.

text
POST /v1/search

Search the local index with the same Web bearer key:

json
{
  "query": "product pricing",
  "maxResults": 5,
  "domains": ["example.com"],
  "maxAgeDays": 30
}

query accepts 1 to 1,000 characters; maxResults defaults to 5 and accepts 1 to 20. Optional index targets web or a configured vertical index, and optional effort (fast, balanced, deep) tunes native retrieval vs. inline excerpt depth. Optional domains and excludeDomains filter public hostnames; maxAgeDays accepts 1 to 3,650 and filters observed crawl age. Optional asOf (or contents.snapshotAsOf, accepting YYYY-MM-DD or ISO 8601 with a UTC offset) queries historical snapshots captured on or before that instant with zero lookahead; it requires lexical ranking and cannot accompany live scrape. timeoutMs accepts 1,000 to 60,000 milliseconds. Optional contents attaches stored indexed text, highlights, or excerpts (with optional maxTotalCharacters / maxTotalTokens and injection guard) in one native call without a live fetch. The response contains query, ranked results, engine information, and request meta. Titles, snippets, and crawl timestamps come from the indexed pages. By default, search makes no live page fetch, and no matching indexed pages returns an empty result list. See Vitna Library on vitna.ai for the full catalog of vertical indexes and native data sources.

To search and fetch each matching page in one call, set scrape: true or pass a scrape object with format, maxCharacters (default 20,000, up to 100,000), onlyMainContent, includeTags, excludeTags, render, waitFor, timeoutMs, retries, browser controls, highlights, question, extract, injection, or redact. Each result then carries a scrape object with status: "ok" and the page document, or status: "failed" and the error code when that single URL could not be read. When scrape.highlights is enabled without its own query, it defaults to the search query. Each scraped hit counts against the client's page-attempt quota.

Experimental local semantic ranking

Lexical retrieval remains the default. Operators can enable VITNA_WEB_SEARCH_SEMANTIC=true on the native backend, then request ranking:"semantic" on POST /v1/search. This fits a local TF-IDF/LSA model over at most 128 currently enabled indexed documents, bounded to 300 title characters and 3,500 plaintext characters per document. It uses eight retained dimensions by default and a cosine cutoff of 0.2. Training and ranking use no external provider or model download.

The model learns associations within that bounded corpus. It abstains on unobserved query terms and detected negation/exclusion. It cannot establish every logical constraint or factual implication. Strict lexical admission requires ranking:"lexical"; combining semantic ranking with strict mode is rejected. Scores are latent cosine similarities, not probabilities.

Operator host scope restricts the training corpus. Domain, site and age filters restrict returned IDs; the model still trains over the broader allowed-host corpus. Source policy and current plaintext are rechecked after asynchronous fitting, and a changed corpus makes the query abstain instead of returning a stale passage. Content changes refresh the cached fingerprint; timestamp-only refreshes need not retrain unchanged text.

Run the frozen authored evaluation locally:

bash
npm run search:semantic

On its 32-document, 26-query synthetic fixture, default eight-dimensional ranking raises held-out nDCG@5 from 0.566860 to 0.837790 and recall from 0.625000 to 1.000000. MRR falls from 0.916667 to 0.861111; negative accuracy falls from 5/8 to 4/8. Exact-detail queries include regressions, and observed cross-topic combinations still produce false positives. The frozen report at .search-data/reports/local-semantic.json records all predeclared dimensions and regressions. These measurements describe this authored fixture, not broad web relevance or a general semantic model.

Reader and scrape

text
POST /v1/read
POST /v1/scrape

These routes are aliases. Read a page as Markdown, readable text, or cleaned HTML. The reader prefers main/article content, removes page furniture and scripts, and includes page metadata and links.

json
{
  "url": "https://example.com/docs",
  "format": "markdown",
  "render": true,
  "waitFor": "main article",
  "timeoutMs": 30000,
  "maxCharacters": 50000
}
FieldDefaultMeaning
urlRequiredPublic HTTPS page, up to 4,096 characters
formatmarkdownmarkdown, text, or html
rendertrueExecute page JavaScript
waitForOmittedCSS selector for content readiness; requires rendering
timeoutMs30,000Request deadline, from 1,000 to 60,000 milliseconds
maxCharacters100,000Output limit, from 1 to 1,000,000 characters
onlyMainContenttruePrefer <main>/<article> and omit <nav>/<aside> chrome; false reads the full <body> (still omitting scripts and styles)
includeTagsOmitted1 to 20 CSS selectors (up to 100 chars each); render only matching subtrees in document order
excludeTagsOmitted1 to 20 CSS selectors (up to 100 chars each); omit matching subtrees everywhere, including inside includeTags
extractOmittedschema, selectors and prompt, as for Structured extraction; adds data from the same render
highlightsOmittedtrue or { query?, maxCharacters? } (up to 4,000, default 1,000); returns verbatim highlights with offsetUnit: "unicode_code_points"
questionOmittedA question (up to 1,000 characters) answered from the rendered page; returns answer and answerEngine. Runs locally on the native extractive model or on a configured language model
injectionOmittedscan reports what could steer a model reading the page; strip also removes invisible characters from content (see below)
redactOmittedtrue, or a list of email, phone, card, iban, ssn and ip; takes personal data out of the response

The HTML output is cleaned content, not the complete serialized browser DOM. Use Unblocker when the complete DOM is needed. Truncated reader output is marked in the response. The configured service deadline can be shorter than the requested deadline.

Personal data. redact: true takes email addresses, phone numbers, payment card numbers, IBANs, US Social Security numbers and IPv4 addresses out of the response, each replaced by a marker such as [redacted:email]. A list such as ["email", "phone"] takes only those kinds. The response's redacted counts what was replaced, by kind, for example { "email": 2, "phone": 1 }. Redaction covers content, title, metadata.description, links and extracted data, and /v1/extract and /v1/crawl accept the same field. The content is redacted before it is cut to maxCharacters, so a cut cannot leave half an address behind. A language model reading the page for extraction is given the redacted text, and its answer is redacted too, since a model can repeat what it has seen elsewhere. A page's own address (url, and the url of a failed page in a crawl) is how a result is matched to its page, so it stays as read. A crawl follows a page's links as the page wrote them and returns them redacted. An address a page splits with invisible characters, such as a zero-width space, is still found, and the replacement takes the invisible characters with it, so injection: "strip" cannot rejoin one. With injection, the findings' excerpts are redacted too: they quote the page's hidden parts, which the content leaves out.

Each kind is found by its shape and a check, never by guessing:

KindFound when
emailAn address with a domain. Image names such as [email protected] and remotes such as [email protected]:org/repo are not addresses
phoneA number with + and a country code, 8 to 15 digits; a tel: link; or a North American number with its punctuation, such as (212) 555-0147, 212-555-0147 or 212.555.0147
card13 to 19 digits, run together or in the usual groups, with the prefix and length of Visa, Mastercard, American Express, Diners Club, JCB, Discover or UnionPay, passing the Luhn check
ibanA country code, two check digits and up to 30 letters and digits, passing the mod-97 check
ssnThree, two and four digits with dashes, never starting 000, 666 or 900 to 999, with no all-zero group
ipFour numbers of at most 255. Loopback, multicast, reserved and documentation addresses are left, as are network blocks such as 10.0.0.0/8 and numbers after "Section", "§" or "version"

What redaction does not find. Names, street addresses, dates of birth and other personal data with no shape of its own; IPv6 addresses; Social Security numbers without dashes; and phone numbers outside North America written without their + country code. A page about one of these formats, such as an article on IP addressing, has its examples redacted. A reference number that passes a card's checks reads as a card, and its marker shows where. An asynchronous extraction keeps the pages it has read in its restart checkpoint until the job ends, with or without redaction.

Documents

A URL that serves a PDF, Word (.docx), Excel (.xlsx) or PowerPoint (.pptx) file is read as text, with no extra field. A document is recognised by its content type, or, when the server sends a generic type such as application/octet-stream, by its file name, and its first bytes must agree. The same reading applies wherever a page is read: Scrape, Extract, Crawl, Monitors and Research. A page that sends the browser on to a document, as a cleared challenge page or a script redirect can, is read as that document.

KindOutput
PDFA section per page, headed Page N, with each page's text in the order the page draws it
WordHeadings, paragraphs, bulleted lists, web links and tables
ExcelA table per sheet, headed Sheet: name; dates and times as their cell formats show them, in ISO 8601, and other numbers as stored
PowerPointA section per slide in the deck's order, headed Slide N and the slide's title

The response's metadata.document describes the file: kind, contentType, filename, bytes, and pages, sheets or slides, with truncated: true when the document passed the reading limit of 1,000,000 characters. Pictures, charts, formulas, speaker notes and other formatting are not read. Page controls (waitFor, actions) do not apply to a document, so a crawl that sets them still reads the documents it links to. /v1/screenshot and /v1/pdf capture what the browser shows and do not read documents. Linked documents downloads the files themselves.

CodeHTTPMeaning
document_too_large422The file exceeds 16 MiB
document_invalid422The bytes are not a readable file of the kind they claim
document_encrypted422The PDF is encrypted
document_no_text422The file has no text; a scanned PDF needs OCR, which is not available
document_unreadable422The file's text could not be read, for example a PDF whose fonts carry no map to letters
document_timeout504Reading did not finish within 20 seconds or the request deadline

A document is requested twice: once by the browser, which is stopped as soon as the response's headers show a document, and once to read it. A download link that works only once fails on the second request.

Prompt-injection scan. Set injection to scan on /v1/read, /v1/scrape or /v1/crawl to have each page checked for what could steer a language model that reads it. The scan uses rules, not a model, so it costs nothing beyond the read. It reports mechanisms and phrases, and does not judge intent: a page that writes about prompt injection is reported, because it holds the phrases.

FindingWhat it is
unicode_tagsInvisible tag characters spelling text a model can read; the finding shows what they spell. Flag emoji are not reported
invisible_run, invisible_charactersFour or more zero-width characters in a row, the shape of a hidden encoding, or sixteen or more in all. The joiners emoji, Indic and Persian text need are not reported
bidi_overrideA right-to-left override that makes text display other than it reads
ignore_instructions, new_instructions, role_token, system_prompt_probe, persona_shift, conceal_from_user, addressed_to_aiInstruction-like text, such as "ignore previous instructions", or the role tokens of a chat template, such as im_start and [INST]
exfiltration_linkA Markdown image whose address carries a placeholder for data to send out

Each finding says where it was found (text, characters, hidden_text, comment, attribute or meta), whether that text reaches the returned content (inContent), its severity, and an excerpt. Hidden places include elements the reader leaves out, comments, alt, title, aria-label and meta descriptions, and text hidden by inline styles the reader keeps, such as zero opacity or font size and off-screen positioning. Text hidden by a stylesheet class is not seen.

json
"injection": {
  "risk": "high",
  "findings": [
    { "kind": "unicode_tags", "severity": "high", "where": "characters", "inContent": true, "excerpt": "28 invisible tag characters spelling: ignore previous instructions" }
  ]
}

risk is the most serious finding that reaches the content, or low when every finding sits where the content does not reach. strip removes tag characters, zero-width characters and bidi controls from content, keeping flag emoji and joiners, and reports how many it removed as stripped. It never rewrites visible text. Whatever a request asks, those characters are removed from everything Vitna's own language model reads.

Batch scrape

text
POST /v1/batch

Scrape a known list of public HTTPS URLs with shared reader, browser, action, extraction, highlight, question, injection, and redaction options.

json
{
  "urls": [
    "https://example.com/pricing",
    "https://example.com/security"
  ],
  "format": "markdown",
  "maxCharacters": 20000,
  "highlights": { "query": "SOC 2", "maxCharacters": 800 },
  "extract": { "selectors": { "heading": "h1" } }
}

Synchronous requests accept 1 to 20 URLs and answer 200 with urls, pages, errors, counts (total, completed, failed), truncated, and meta. Pass async: true to run up to 100 URLs as a background task (202), polled with GET /v1/jobs/:id, cancelled with DELETE /v1/jobs/:id, and optionally delivered to webhookUrl. maxCharacters defaults to 20,000 per page (up to 100,000), and retries defaults to 0.

Each URL is isolated: a page that fails to render or extract is recorded in errors (or extractError / answerError when reading succeeded) without aborting the rest of the batch. Background batch jobs checkpoint after every URL and recover across restarts, recording an in-flight ambiguous URL as outcome_unknown rather than repeating work twice. Result payloads respect the job byte budget and set truncated: true if remaining pages would exceed it.

Browser controls

Page-reading requests also accept these options:

json
{
  "url": "https://example.com/",
  "country": "US",
  "session": "product-run-01",
  "language": "en-US",
  "viewport": { "width": 1440, "height": 960 },
  "headers": { "dnt": "1" }
}
FieldMeaning
countryTwo uppercase letters selecting a configured proxy route that serves that country
networkauto (the default) starts on the cheapest configured route and steps up a tier after a block; datacenter, isp, or residential uses only routes of that kind
session1 to 64 letters, numbers, underscores, or hyphens; sticky proxy selection
persistSessionExplicit boolean; true requires session and retains tenant-isolated cookies/localStorage
actionsUp to 50 scroll, wait, click, write or press actions; requires rendering. Bounds, the GET-only guard and failed-step reporting (error.action) are under Browser actions in the Unblocker request reference. /v1/read and /v1/scrape also take screenshot and scrape steps, below
languageBCP 47 locale, such as en-US, up to 35 characters
viewportWidth from 320 to 4,096; height from 240 to 4,096
headersOnly accept-language up to 200 ASCII characters and dnt equal to 0 or 1

A country request needs a matching route configured by the operator. An unavailable country fails explicitly, as does a network with no route of that kind (network_unavailable). When the operator configures residential or ISP provider routes, a blocked attempt's retry moves up a tier, and the bytes an attempt moves through those routes are metered for pricing; see proxy routes. When the operator runs the auto browser profile, a challenged attempt's retry also moves from Chromium to Camoufox, so a page behind a browser check can still be read with the default retries. So can a document: Camoufox takes the file from the response to the navigation that cleared the check, so /v1/read and each file `/v1/documents` downloads return it with the same 16 MiB limit and first-bytes check, and without asking for it a second time. A document Camoufox still cannot hand over, such as one whose bytes are not the file they claim to be, fails as the target_challenge that sent the read to Camoufox. A session keeps route selection deterministic while the route list stays unchanged. Every attempt still uses a fresh browser/context. With explicit persistSession: true, the renderer restores and saves that tenant's cookies and localStorage after successful delivery and cleanup. State stays in RAM with a 15-minute sliding TTL, 100 total sessions, 20 per tenant, and 256 KiB per session. It is lost on renderer restart. Same-session concurrent persistence returns session_busy. DELETE /v1/sessions/SESSION_ID clears the caller's session and cancels its active attempt. The caller cannot submit proxy credentials, cookies, authorization headers, or browser scripts.

Screenshot and scrape steps. On /v1/read and /v1/scrape, an action sequence can capture the page partway through: { "type": "screenshot" } images what the page shows at that step, and { "type": "scrape" } reads it. A list paged by a button reads as one request:

json
{
  "url": "https://example.com/filings",
  "actions": [
    { "type": "scrape" },
    { "type": "click", "selector": "button.next" },
    { "type": "scrape" },
    { "type": "screenshot", "fullPage": true }
  ]
}

The response's content is the page after the last step, as always, and actions holds the captures, each with the index of the step that took it:

json
{
  "actions": {
    "screenshots": [{ "index": 3, "contentType": "image/png", "base64": "iVBORw0KGgo...", "bytes": 48213 }],
    "scrapes": [
      { "index": 0, "url": "https://example.com/filings", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": ["https://example.com/filings?page=2"], "truncated": false },
      { "index": 2, "url": "https://example.com/filings?page=2", "title": "Filings", "format": "markdown", "content": "# Filings ...", "links": [], "truncated": false }
    ]
  }
}

A step waits for the page's text to settle before it captures, as the final read does, so a scrape after a click takes the next page rather than the one before it. Scraped pages use the request's format and maxCharacters. A screenshot shows the viewport unless fullPage is true, and is a PNG unless format is jpeg. A sequence may hold 10 screenshot and scrape steps, which together may capture 8 MiB (images, and the HTML of scraped pages). A step past that budget, or a full-page image larger than 16,384 pixels on a side or 32 million pixels in all, fails with action_output_too_large (HTTP 422) and names itself in error.action. Other routes, crawls included, refuse these steps, and /v1/status lists them under capabilities.actionOutputs. A scraped page is treated as the final one is: with injection it carries its own injection report, and with redact its personal data comes out and is counted in the response's one redacted. A screenshot is an image, which neither the scan nor redaction reads.

Without waitFor, the browser allows an initial 500-millisecond grace period, then waits for 200 milliseconds of stable meaningful visible content, within five seconds or the remaining request deadline. A persistent empty or loading state fails with target_incomplete. Set a populated content selector for delayed applications; page stability cannot establish that every requested product field has finished loading. A waitFor selector that never becomes visible fails with selector_timeout (HTTP 504) shortly before the deadline rather than request_timeout, and a retry waits within what is left of timeoutMs.

Encrypted sessions across restarts

Session state remains in RAM by default. For local browsers, pair VITNA_WEB_SESSION_DATA with VITNA_WEB_SESSION_KEY_HEX, an exact 32-byte hexadecimal operator key. Use a separate private directory from VITNA_WEB_DATA; with the supplied systemd unit it can be /var/lib/vitna-web-sessions. Requests still need a named session and persistSession:true. Browser-worker deployments configure persistence on each worker instead of the API coordinator.

Cookies and local storage are encrypted using AES-256-GCM with fresh nonces and authenticated tenant/session identity, expiry and byte bounds. The same stable key restores unexpired sessions after restart. A wrong key fails closed and preserves the prior database. A key that opens some rows but not others is the right key with damaged rows: each damaged row is set aside unchanged for inspection and counted, and the other sessions open normally. After startup a damaged row affects only its own session, which its owner can delete. TTL, per-client/global capacity, successful-state commit and explicit deletion continue to apply to durable rows. Deletion cannot be undone by a stale active lease. Renderer shutdown retains ciphertext and closes storage only after physical browser cleanup settles.

The helper rejects directory/data-file links and database hardlinks that could bypass the writer lock. Existing POSIX storage must belong to the service user and grant no group/other access. Windows deployments must configure equivalent private ACLs. Session keys are operator secrets; back up the key separately from a consistent stopped-service database backup. Key rotation and replicated session storage are not implemented.

Structured extraction

text
POST /v1/extract

Provide CSS selectors for known fields. A string selector reads text; an object with attribute reads an attribute from matching elements. This path runs locally without a model. Supply up to 32 selector fields. Supported syntax includes tags, IDs, classes, attribute matches, descendant and child combinators, and comma-separated groups. A supplied selector scopes its field; a missing match is not replaced with an unrelated structured-data value.

json
{
  "url": "https://example.com/products/desk",
  "selectors": {
    "name": "h1",
    "price": ".price",
    "image": { "selector": "img.product", "attribute": "src" }
  }
}

To shape published structured data, provide a supported JSON Schema. Extraction reads JSON-LD, microdata/RDFa, supported embedded application data, and any supplied selectors. Canonical URLs, explicit primary-entity references, product identifiers, and visible headings help select the page's company or product. An unresolved choice between multiple entities returns extraction_ambiguous (HTTP 422); explicit selectors can select the intended fields. Schema fields can use object, array, string, number, integer, and boolean types, including nullable unions with null. Supported keywords include properties, required, items, enum, additionalProperties, description, and title.

json
{
  "url": "https://example.com/products/desk",
  "schema": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "price": { "type": ["number", "null"] },
      "currency": { "type": ["string", "null"] }
    },
    "required": ["name", "price", "currency"],
    "additionalProperties": false
  }
}

Missing fields use the same bounded observed-fact parser as native Research. It recognizes explicit JSON objects, labelled literals, dotted/indexed fields, two-column tables, definition lists, and a finite set of unambiguous prose facts, including named company founding dates, exact employee counts, headquarters, and product prices/weights. It preserves the source wording, units, currencies, billing periods, and ranges internally. A measured value such as 2 kg needs a string field; it is not reduced to the number 2. The symbol $ alone does not establish the currency code USD.

The parser rejects conflicting requested observations with extraction_ambiguous (HTTP 422). Missing required fields return extraction_incomplete; invalid extracted values return extraction_invalid, also HTTP 422. If the configured native engine is called for remaining fields, its unsupported or conflicting observations return native_extraction_unsupported or native_ambiguous_data. An optional self-hosted compatible model can handle language-based fields beyond these rules. The final result must satisfy the schema, including nested objects, arrays, required fields, and enums. Schema responses include configured engine metadata in extractionEngine; extracted data follows the requested schema.

Extract with a prompt. Add prompt, up to 10,000 characters, to tell the model what to extract in plain language. With a schema, the prompt guides how the model fills it; without one, the model's JSON object is the data:

json
{
  "url": "https://example.com/investors/governance",
  "prompt": "List every board member with their role and the committees they chair."
}

A prompt needs a language model (VITNA_WEB_MODEL_BACKEND=self-hosted); the native engine returns model_not_configured (HTTP 503) rather than ignoring it, and /v1/status reports capabilities.promptExtraction. With a prompt, the model reads every field that no selector names, because published data cannot follow an instruction such as "prices in euros". Without one, published data, selectors and observed facts come first and the model fills only what they miss. A prompt cannot accompany a recipe, and selectors with a prompt need a schema that declares their fields.

Long pages and repairs. A language model reads the page in parts of 60,000 characters, cut at paragraph breaks, at most eight of them; each part is a metered model request. The answers combine in page order: a list gathers values from every part without repeats, and a single value is the first a part established. An answer that does not match the schema is sent back once with the problem named before the request fails with extraction_invalid. A response that used a language model reports how it read the page:

json
"extraction": { "parts": 3, "truncated": false, "repaired": 0 }

truncated means the text ran past eight parts. Data from a prompt without a schema is limited to 1 MiB (extraction_too_large) and 32 levels of nesting. The native engine reads a single pass, as before, since it refuses observations that conflict and must see them together.

Personal data. With redact, as described under Reader and scrape, a language model reads the pages with the named kinds already taken out, every string in data (and in rows) is redacted as well, and the response's redacted counts what came out of them.

Scrape and extract in one request. /v1/read and /v1/scrape accept an extract object with schema, selectors and prompt, under the same rules. The response carries the page's content and the extracted data from one browser render:

json
{
  "url": "https://example.com/products/desk",
  "format": "markdown",
  "extract": { "prompt": "The price in euros and whether it is in stock" }
}

Extract across pages

Supply urls instead of url to combine up to 20 pages into one result:

json
{
  "urls": ["https://example.com/about", "https://example.com/contact"],
  "schema": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "foundingDate": { "type": "string" },
      "employees": { "type": "integer" }
    },
    "required": ["name"],
    "additionalProperties": false
  },
  "async": true,
  "jobTimeoutMs": 120000
}

For site discovery, provide url and maxPages, up to 20, with the map discovery and path-filter options below. The default extraction page budget is 1. Arrays are combined without duplicate values; scalar fields use the first valid local value, and compatible nested objects can fill complementary fields. Flat product extraction combines pages only when their selected products have matching identity; one product's name cannot borrow another's price. Use an object array such as products for multiple product pages. Conflicting requested prose observations remain an error. Observed-fact parsing and the optional model fallback use at most 100,000 combined content characters. The parser also bounds source count, nesting, arrays, and fact count; it does not read every page of a domain automatically.

The result includes url, successfully read urls, combined data, and per-page errors. With async: true, the response is a job receipt instead. Poll /v1/jobs/:id for progress and the final result. jobTimeoutMs accepts 1,000 to 600,000 milliseconds for background work; timeoutMs remains the per-page browser deadline. An optional webhookUrl can receive a job completion notification. Fetch the result from the job endpoint.

Extraction recipes

text
POST /v1/recipes/draft
POST /v1/recipes
GET /v1/recipes
GET /v1/recipes/:id
DELETE /v1/recipes/:id

A recipe names the fields to read from one kind of page, and how to read each, once, so every page built from the same template can be extracted or watched by its id. A draft builds one from examples: name a sample page and, for each field, its value as the page shows it.

json
{
  "url": "https://shop.example.com/products/desk-lamp",
  "samples": ["https://shop.example.com/products/office-chair"],
  "fields": {
    "name": "Desk Lamp",
    "price": "$24.99",
    "image": { "example": "https://shop.example.com/img/lamp.jpg", "attribute": "src" },
    "reviews": { "examples": ["Great lamp", "Too bright"] },
    "sku": { "describe": "the product's SKU" }
  }
}
Field formMeaning
"value" or { "example": "value" }Text the page shows, matched regardless of case and spacing; an element whose text closely contains it also matches
{ "examples": ["a", "b"] }Two to 20 values of one repeated field, which becomes a list
{ "example": "value", "list": true }One value of a repeated field
{ "example": "value", "attribute": "src" }An attribute's value rather than text. Without attribute, an example not found as text is looked for in content, src, href, datetime, value, alt, title and data-src, and a relative URL matches its absolute form
{ "describe": "words" }No example: extraction finds the value from published data, observed facts and the operator's model when one is configured, and that value becomes the example

The draft renders the pages with the request's browser controls, finds the element holding each example in text a reader sees, never under a hidden ancestor, and keeps the selector that reaches it most steadily. An id comes first, then itemprop, data-testid and similar attributes, then classes, then a class or tag scoped by a named ancestor. Generated class names such as css-1x2y3z or Button_label__a8Kq3 are never used. A single field's selector must reach the example before anything else, because extraction keeps the first match; a list field's must reach every example. Each candidate is run on the page before it is chosen. The selector engine has no positions, so an element told from its siblings only by position gets no selector.

The response gives each field its selector, attribute and list, with basis (id, attribute, class, scoped or tag), the number of elements it matches, and whether the example matched as text, by contains or as an attribute. A field without a selector gives its reason: example_not_found, examples_disagree, no_stable_selector or, for a description, value_not_found. A described field's value is returned as described. recipe.fields is ready to save, and preview reads every page with the recipe:

json
{
  "preview": [
    { "url": "https://shop.example.com/products/desk-lamp", "data": { "name": "Desk Lamp", "price": "$24.99", "reviews": ["Great lamp", "Too bright", "Works fine"] }, "missing": [] },
    { "url": "https://shop.example.com/products/office-chair", "data": { "name": "Office Chair", "price": "$89.00" }, "missing": ["reviews"] }
  ]
}

This preview is illustrative. A draft is one request, and it reads the sample page and up to three samples through the same browser slots, page quota and accounting as any read. A sample that cannot be read appears in the preview with its error code; a first page that cannot be read fails the draft.

Save a recipe with a name, its fields, and optionally render and waitFor. Each field is a selector string, or an object with selector, attribute, list and describe, of which a selector or a description is required. A draft's recipe.fields can be sent as returned. A selector the engine cannot parse is refused when the recipe is saved. A key holds up to 50 recipes of up to 32 fields each, and only that key can read, run or delete them.

Run one by sending recipe in place of schema and selectors:

json
{
  "urls": ["https://shop.example.com/products/desk-lamp", "https://shop.example.com/products/office-chair"],
  "recipe": "RECIPE_ID"
}

A recipe run adds rows, one { "url", "data" } per page read, beside the combined data; a page whose own extraction fails keeps its row with an error code. A described field without a selector is found by extraction on each page. The recipe's render and waitFor are defaults that the request's own values override. POST /v1/monitors with recipe watches the recipe's selected fields; a recipe with none is refused with recipe_unmonitorable. An unknown or another key's recipe is recipe_not_found (404), and a key at its limit is recipe_limit (429).

Site map

text
POST /v1/map

Discover links, sitemap entries, and indexed search results. Include and exclude filters match case-sensitive pathname substrings, and excludes take priority. Discovery defaults to the starting hostname. Additional domains must be explicitly allowed.

json
{
  "url": "https://example.com/",
  "maxPages": 100,
  "maxBreadth": 100,
  "sitemap": "include",
  "maxSitemaps": 3,
  "includePaths": ["/products/"],
  "excludePaths": ["/account/"],
  "timeoutMs": 30000
}

The default maxPages is 50, with a maximum of 1,000 for maps. Filters can contain up to 20 paths per list. The seed page can be read to discover links even when it does not match an include filter. The response reports discovery counts and errors in addition to the URLs.

FieldDefaultMeaning
maxDepth20 for maps; 2 for crawlsLink depth from the seed, from 0 to 20
maxBreadth100Maximum discovery breadth, from 1 to 1,000
sitemapskipskip, include, or only
sitemapUrlsOmittedUp to 5 explicit public HTTPS sitemap URLs
maxSitemaps3Sitemap document budget, from 1 to 10
searchQueryOmittedOptional discovery query, up to 500 characters; uses the selected indexed Search backend
searchLimit20Search-result budget, from 1 to 100
includeSubdomainsfalseAllow public subdomains of the starting host
allowedDomains[]Up to 20 explicitly allowed public hostnames

For example, add searchQuery: "site:example.com products" to discover pages through the selected indexed Search backend. Search results and sitemap entries pass the same destination checks, host scope, and path filters as links found on a page. sitemap: "only" uses sitemap discovery without a full link walk. A map request remains bounded by its total timeoutMs, and by the service's request deadline when that is shorter. A map that runs out of time, by either deadline, fails with crawl_timeout (HTTP 504); only a caller that disconnects ends one with request_cancelled.

Crawl jobs

text
POST /v1/crawl
GET /v1/jobs/:id
DELETE /v1/jobs/:id

Collect cleaned pages in a background job:

json
{
  "url": "https://example.com/docs/",
  "maxPages": 100,
  "maxDepth": 4,
  "maxBreadth": 100,
  "format": "markdown",
  "includePaths": ["/docs/"],
  "excludePaths": [],
  "timeoutMs": 30000,
  "jobTimeoutMs": 120000
}

The create response includes a job id, status, pages, and errors. Poll GET /v1/jobs/:id using the same client key to retrieve progress and collected pages. DELETE cancels queued or running work.

The default page budget is 50 and the default depth is 2 links from the seed. The supported depth is 0 to 20; the page budget is 1 to 500. A page budget bounds attempted reads, while maxCharacters bounds each collected document. timeoutMs bounds each browser read. jobTimeoutMs bounds the whole background job, from 1,000 to 600,000 milliseconds, and defaults to timeoutMs when omitted. Partial page failures appear in errors.

redact takes personal data out of every collected page before the job stores it, with a redacted count on each page, as described under Reader and scrape. Crawls also accept extract, highlights, and question to run per-page structured extraction, passage selection, or page-grounded Q&A during the crawl; if extraction or Q&A fails on one page, that page keeps its content and records extractError or answerError while the crawl continues.

Crawls accept the same sitemap, search, domain, and path controls as maps. Retrieve a bounded slice of crawl results with query parameters:

text
GET /v1/jobs/JOB_ID?offset=0&limit=25&includeContent=false

offset accepts 0 to 500, limit accepts 1 to 100, and includeContent chooses whether page content accompanies the page metadata. The response includes pagination with offset, limit, total, and nextOffset.

Each job also caps returned content at two million characters and five MiB of serialized pages/errors. When that budget is reached, completed pages remain available and the job reports truncated: true with truncationReason: "result_limit". Up to 20 jobs per client are retained; older finished jobs are removed to admit new work. Queued jobs recover after restart. Crawl resumes its committed frontier and extraction resumes committed page snapshots; unresolved page attempts are recorded as outcome_unknown and skipped. An ambiguous final model invocation, running Research jobs, and running catalog and backfill jobs remain failed rather than repeating metered work. Recovery preserves the original absolute deadline, backs off, and permits at most three restarts. Extraction checkpoints share the five-MiB result bound; oversized checkpoints fail explicitly. Recovery metadata is public, while its snapshots and intents remain private. These leases are single-writer restart bookkeeping.

Jobs and monitors belong to the client identity associated with the key. A different client cannot inspect or cancel them.

Change monitors

text
POST /v1/monitors
GET /v1/monitors
GET /v1/monitors/:id
DELETE /v1/monitors/:id
POST /v1/monitors/:id/check

Watch readable page text, or selected fields, on an interval. Supply a public HTTPS webhook to receive changes:

json
{
  "url": "https://example.com/products/desk",
  "intervalSeconds": 3600,
  "selectors": { "price": ".price", "availability": ".availability" },
  "webhookUrl": "https://your-app.example/webhooks/vitna"
}

The interval defaults to 3,600 seconds and accepts 60 seconds to 30 days. The first successful check establishes a baseline. Subsequent checks compare normalized text or selector values. A missing field can be a change; it is not treated as a price of zero.

To score changes structurally instead of flat text diffs, pass diffMode (prose, pricing, policy, or risk_factors) and optional minMateriality (0 to 1). When the computed structural materiality score falls below minMateriality, the monitor records a below_materiality suppression entry, skips webhook delivery, and preserves its baseline; when it meets or exceeds the threshold, the event carries typed hunks, numeric deltas, and a SHA-256 diff.attestation digest.

To suppress minor noise on a page monitor, set meaningfulChange: true (default 5% changed-token ratio), { "minChangedRatio": 0.1 }, or { "prompt": "..." } (requires a configured language model). When a change is judged not meaningful, the monitor records a suppressed reason in history, sends no webhook, and keeps the previous baseline intact so gradual drift still crosses the threshold.

To watch search results for a query instead of a single URL, create a web monitor with kind: "web" (or omit url and pass query), along with optional maxResults (1 to 20, default 10), domains, excludeDomains, ranking, goal, dedup ("story" default or "url"), maxAgeDays (1 to 3,650), minSnippetCharacters (1 to 1,000), summarize (true for cited bullet summaries), and schema (for cited structured event payloads):

json
{
  "kind": "web",
  "query": "vitna release notes",
  "domains": ["example.com"],
  "goal": "pricing change or new enterprise plan",
  "dedup": "story",
  "maxAgeDays": 14,
  "summarize": true,
  "intervalSeconds": 3600,
  "webhookUrl": "https://your-app.example/webhooks/vitna"
}

Web monitors report added and removed result URLs against the previous snapshot. With dedup: "story", syndicated reprints with matching canonical URLs, normalized headlines, or 3-word shingle Jaccard similarity at or above 0.6 are suppressed across up to 200 seen stories without firing webhooks (suppressedCount is recorded in check history). When goal is set, alerts fire only when a newly added result matches the goal (judged by the configured language model when available, or deterministically by query-term match otherwise). Setting summarize: true or schema attaches cited summary, data, and citations fields to monitor.changed.

List or inspect monitors to see their current snapshot, recent changes, check times, and webhook delivery state. The manual /check route takes an empty JSON object and runs a check immediately. DELETE stops the monitor. Scheduling requires the service process to remain running.

Readiness selectors and retry settings are saved with each monitor. History is bounded to 20 checks. Webhook failures retain the same event ID across up to five delivery attempts. An event that fails all five leaves the queue, so a later successful delivery reports delivered again; the loss stays visible as webhook.failedEvents, a running count, and webhook.lastFailure, which names the event, its attempts and the last error.

Screenshots and PDFs

text
POST /v1/screenshot
POST /v1/pdf

Capture a browser view:

json
{
  "url": "https://example.com/",
  "format": "png",
  "fullPage": true,
  "width": 1440,
  "height": 960
}

Screenshots support png and jpeg. The default viewport is 1,440 by 960 pixels; width accepts 320 to 4,096 and height accepts 240 to 4,096. PDFs use Chromium print output and need only a URL for the default layout. On /v1/pdf, you can also bound long pages with pageRanges (page numbers or ranges between 1 and 1,000, such as "1-3" or "1-2,5"), scale (0.1 to 2), and printBackground (boolean, default true) to keep artifacts under the 10 MiB capture limit:

json
{
  "url": "https://example.com/",
  "pageRanges": "1-2",
  "scale": 0.75,
  "printBackground": false
}

Both routes return an artifact in a JSON envelope:

json
{
  "artifact": {
    "contentType": "image/png",
    "base64": "BASE64_ENCODED_BYTES",
    "bytes": 12345
  }
}

This envelope is illustrative. Decode base64 into bytes and save them with the returned content type. The playground provides a download button for live artifacts.

Linked documents

text
POST /v1/documents

Download the PDFs a page links to, as the files they are:

json
{ "url": "https://investors.example.com/financials", "maxFiles": 10 }

The page is read in the browser, as for Read, and every document link on it is listed. A link counts when its path ends in .pdf or its type attribute is application/pdf; a link whose text, title, label or class says "PDF" ("Annual report (PDF, 2 MB)") counts too and is listed after those, since it can lead to a page rather than a file. Links in embedded viewers (iframe, embed, object) count, fragments are dropped, and a link back to the page itself does not count. Each file is then fetched through the same browser and network path as the page, with its country, session, network and language, so a site that serves the page to the browser serves its files too. A url that is itself a document is the one file.

FieldDefaultMeaning
kinds["pdf"]Which documents to find: any of pdf, docx, xlsx, pptx
maxFiles10How many of the listed links to download, 1 to 20, in the order listed
downloadtruefalse lists the links without downloading any
timeoutMs30,000The whole request's deadline, the page and every file together

The browser fields (render, waitFor, actions, country, session and the rest of Browser controls) apply to the page; files take its network and identity but not its waitFor or actions.

json
{
  "url": "https://investors.example.com/financials",
  "found": [
    { "url": "https://cdn.example.com/2025-annual-report.pdf", "text": "2025 Annual Report" },
    { "url": "https://investors.example.com/static-files/0f1c", "text": "Q3 results (PDF)" }
  ],
  "documents": [
    {
      "url": "https://cdn.example.com/2025-annual-report.pdf",
      "text": "2025 Annual Report",
      "kind": "pdf",
      "contentType": "application/pdf",
      "filename": "2025-annual-report.pdf",
      "bytes": 2481503,
      "sha256": "9f2c...",
      "base64": "JVBERi0xLjcK..."
    }
  ],
  "errors": [{ "url": "https://investors.example.com/static-files/0f1c", "code": "not_a_document" }],
  "truncated": false
}

This response is illustrative. found lists up to 500 links in the order they are tried. documents holds each file downloaded, with finalUrl when a redirect served it from elsewhere, and errors each link that did not give one: not_a_document when it served a page, document_kind_excluded when it served a kind not asked for, documents_too_large when it would take the answer past 32 MiB, or the code a Read of it would fail with, such as target_http_error or document_too_large for a file over 16 MiB. A link in found but in neither list was not reached, and truncated is then true: maxFiles, the byte budget or the deadline stopped the downloads, or the page had more than 500 document links. The downloads already made are always returned; only a page that cannot be read fails the request.

The page and each file attempted count against the daily page quota, and each is metered as a browser read. Files are read one at a time, so ten files take about ten times as long as one; ask for fewer, or a longer timeoutMs, on a slow site.

Browser API

text
GET /v1/browser   (WebSocket upgrade)

Drive a real Chromium with Puppeteer, Playwright or any Chrome DevTools Protocol (CDP) client. Each connection launches one browser for that session and removes it, with its profile, when the connection closes. Scripts, clicks, typing, forms and screenshots are the client's to send. The endpoint is off unless the operator sets VITNA_WEB_BROWSER_API=true.

Connect with a vitna_web_ key in the Authorization header. The website relay does not carry WebSockets, so connect to the Web service's own origin:

js
import { chromium } from "playwright";

const browser = await chromium.connectOverCDP("wss://web.example.com/v1/browser", {
  headers: { Authorization: `Bearer ${process.env.VITNA_WEB_KEY}` },
});
const context = await browser.newContext();
const page = await context.newPage();
await page.goto("https://example.com/");
await browser.close();

Puppeteer's connect takes the same endpoint as browserWSEndpoint and the header in headers. The relay is tested with Playwright's connectOverCDP and with a raw CDP client. Add ?country=US to select an operator proxy route that serves that country, as page reads do, and ?network=residential (or isp, datacenter) to use only routes of that kind. A session has no retries to step up through, so without network it uses the cheapest configured route. Add ?session=NAME (1 to 64 letters, numbers, underscores or hyphens) to keep a route and its provider address from one session to the next, as session does for page reads: the provider is asked for the same address whenever that key opens a session of that name, and an unnamed session is given a fresh one. Whether the address is held is the provider's to decide. No other parameter is accepted.

The browser starts before the handshake completes, so every refusal is an ordinary HTTP answer with the JSON error body:

HTTPCodes
400invalid_request, country_unavailable, network_unavailable
401unauthorized
404not_found when the Browser API is not enabled
426upgrade_required for a request that is not a version 13 WebSocket upgrade
429session_limit, service_busy, daily_limit, rate_limited, with Retry-After
503browser_unavailable, service_unavailable

What a session may ask of its browser

Every command passes an allowlist before it reaches the browser. A refused command is answered with a CDP error, -32601 for a method that is not available and -32602 for parameters that are not, and the browser never sees it. The relay refuses what would reach past the session:

  • This machine's files: DOM.setFileInputFiles, Page.handleFileChooser, Input.dispatchDragEvent carrying files, Network.loadNetworkResource and DOM.getFileInfo. A navigation by Page.navigate, Target.createTarget or a rewritten Fetch.continueRequest must be to an https:, http: or data: URL, or about:blank. File uploads are therefore unavailable.
  • The network guard: Target.createBrowserContext with proxyServer, proxyBypassList or originsWithUniversalNetworkAccess, and Target.setRemoteLocations.
  • The machine itself: Tethering, SystemInfo, Extensions, PWA, Cast, Memory and Tracing, and every Browser method except getVersion, close, the window and permission methods, and setDownloadBehavior, which is rewritten as below.
  • The relay: Target.exposeDevToolsProtocol and Target.sendMessageToTarget. Attach with flatten: true, as Puppeteer and Playwright do.

Downloads are rewritten rather than refused: Browser.setDownloadBehavior and Page.setDownloadBehavior are forwarded with behavior: "deny", because clients ask for downloads into a directory on their own machine. A domain or method not on the allowlist, including one added in a later Chromium, is refused until it is reviewed.

The relay keeps one browser session for itself. It denies downloads in every new browser context before the context's id reaches the client, and it closes any page past the page limit. Its session id is refused in a client's commands and its events are not forwarded.

Network access is the same as for page reads: public HTTPS destinations on port 443 through the guarded proxy, each address checked and pinned, within the operator's VITNA_WEB_ALLOWED_HOSTS. Secure WebSockets are carried; QUIC and unproxied UDP are disabled. A plain HTTP page is answered with 403 by the guard itself.

Session limits and close codes

EnvironmentDefaultMeaning
VITNA_WEB_BROWSER_APIfalsetrue enables /v1/browser
VITNA_WEB_BROWSER_API_SESSIONS2Sessions open at once in this process, 1 to 20
VITNA_WEB_BROWSER_API_SESSIONS_PER_CLIENT1Sessions open at once for one client, at most the process limit
VITNA_WEB_BROWSER_API_SESSION_MS300,000Whole-session limit, 10,000 to 3,600,000
VITNA_WEB_BROWSER_API_IDLE_MS120,000Time without a client command, 5,000 to 3,600,000
VITNA_WEB_BROWSER_API_MAX_BYTES104,857,600Network transfer through the guard, both directions, 1 MiB to 10 GiB
VITNA_WEB_BROWSER_API_MAX_PAGES10Pages open at once; the relay closes the newest past it, 1 to 100

A session also allows 10 browser contexts, 2,000 commands awaiting the browser, 4 MiB per client message and 64 MiB per browser message. Output a client leaves unread is capped at 64 MiB. The connection closes with:

CodeReason
1000The client closed the connection, or sent Browser.close
1001service_shutdown
1007A message that was not valid UTF-8 or JSON
1008A message that was not a command, or client_not_reading
1009A client message over 4 MiB
1011The browser failed while the session ran
4001session_timeout
4002idle_timeout
4003transfer_limit
4004browser_exited, a browser that exited unasked
4005output_too_large, a browser message over 64 MiB

A session is one request against the client's per-minute and daily limits, or one shared-quota admission. Its browser time, from launch through cleanup, is recorded as browser milliseconds and priced at VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD, like every other browser operation. A session through a residential or ISP route also records its transferred bytes for per-GB pricing. The session's entry in /v1/usage and its receipt carry the operation browser, the close reason as its outcome, closeCode, network, browserProfile, commands, refusedCommands and transferredBytes. GET /v1/status reports capabilities.browserApi and, when enabled, limits.browserApi, whose execution says where sessions run: {"mode":"local"}, or the workers and their combined capacity. Its browserProfile is the profile the calling key's sessions launch with.

Browser profiles

A session's browser is native unless the operator grants its account the cloakbrowser profile, the Unblocker's profile of that name: an operator-installed CloakBrowser executable, launched with the same fingerprint flags and a 1920 by 947 desktop window. A named session keeps its fingerprint as it keeps its address, so it returns as the same device; an unnamed one draws a new fingerprint.

dotenv
VITNA_WEB_BROWSER_API_CLOAKBROWSER_CLIENTS=["acct_ID","pilot"]
VITNA_WEB_BROWSER_API_CLOAKBROWSER_PATH=/opt/cloakbrowser/chrome

The grant lists client ids, a static key's name or an account's id, 1 to 1,000 of them. Like a proxy-access grant it is the operator's check on the customer: confirm who the customer is and what the traffic is for before adding one. Public signup never adds an account to it. Every other key's sessions stay native, on VITNA_WEB_BROWSER_PATH or the bundled Chromium. The path is set here when sessions run on this machine, and on each worker otherwise (below); startup refuses a path without grants, or grants without a path to launch them. The CloakBrowser binary license is the operator's to review.

The relay does not solve challenges. A profile changes how the browser presents itself, and the network guard and allowlist are the same for every profile. Verification pages, CAPTCHAs and interstitials reach the client as they would reach any browser, and what to do with them is the client's. This differs from page reads, where the Unblocker waits on a recognised Cloudflare challenge for its client.

By default, sessions launch on the Web process's own machine, including when page reads use owned browser workers. The executable is VITNA_WEB_BROWSER_PATH, or Playwright's bundled Chromium installed by npm run web:install and npm run web:browser; a browser channel alone is not accepted. The browser runs with its sandbox on, so the host must support it, on Linux through unprivileged user namespaces or the setuid sandbox. A reverse proxy must pass the upgrade through: deploy/vitna-web.nginx.conf.example has a location = /v1/browser block that does.

Sessions on owned workers

To run sessions on machines you operate, list them in VITNA_WEB_BROWSER_API_WORKERS:

dotenv
VITNA_WEB_BROWSER_API=true
VITNA_WEB_BROWSER_API_SESSIONS=8
VITNA_WEB_BROWSER_API_WORKERS=[{"id":"browser-a","origin":"https://browser-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","sessions":8}]

Each worker is the npm run web:worker process described under owned browser workers, with VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS set to the sessions it may hold (1 to 20; unset or 0 leaves it off). It uses its own VITNA_WEB_BROWSER_PATH and VITNA_WEB_ALLOWED_HOSTS, with the same rule that a channel alone is not accepted. sessions defaults to 2, up to 20 per worker and 20 workers, and VITNA_WEB_BROWSER_API_SESSIONS may not exceed their total. These workers are separate from VITNA_WEB_BROWSER_WORKERS, so a machine can serve Browser API sessions without page reads, or both.

The Web process keeps the client's connection, the command allowlist and every limit above; the worker runs the browser and its network guard. They share one connection per session, an HTTP upgrade to /v1/cdp on the worker's server authenticated with its operator token, which carries the DevTools messages and the guard's transfer count. The operator proxy route a session selects is sent over that connection, credentials included, so give workers the same trust as the Web host. A worker answers the upgrade only once its browser is ready, so a worker that is full or cannot launch is passed over for the next, most free sessions first; a session already running is never moved. With every worker full, /v1/browser answers 429 service_busy. A worker closes a session's browser when its connection drops, when the worker stops, or 30 seconds past VITNA_WEB_BROWSER_API_SESSION_MS if the Web process never ends it. Its /healthz reports browserApi with its sessions, the profiles it can launch and how many are active.

A worker offers the cloakbrowser profile only with its own VITNA_WEB_BROWSER_WORKER_CLOAKBROWSER_PATH, an absolute path, and its native sessions keep using VITNA_WEB_BROWSER_PATH. Grants stay on the Web process. A worker asked for a profile it cannot launch answers 422 profile_unavailable and is passed over like a full one; when no worker offers the account's profile, /v1/browser answers 503 browser_unavailable.

Proxy access

text
GET    /v1/proxy
GET    /v1/proxy/keys
POST   /v1/proxy/keys
DELETE /v1/proxy/keys/KEY_ID

Datacenter proxies from Vitna's own egress gateways, for your own HTTP clients. The operator grants access account by account, with a monthly allowance. The account owner then mints proxy keys with the owner token. The key is shown once:

bash
curl -X POST https://web.example.com/v1/proxy/keys \
  -H "Authorization: Bearer $VITNA_WEB_OWNER_TOKEN" \
  -H "Content-Type: application/json" -d '{"label":"crawler"}'

The response carries username (the account), proxyKey, and the key's id and label. Connect through any gateway in GET /v1/proxy as an HTTPS proxy:

bash
curl -x "https://ACCOUNT:[email protected]:8791" https://example.com/

GET /v1/proxy returns access (granted, suspended, or not_granted), username, endpoints (each gateway's host, port, and country), limits (monthlyBytes, concurrency, tunnelsPerMinute), the month's usage.bytes, and the keys without their secrets. Revoking a key takes effect at every gateway within about a minute, including on tunnels already open. Proxy routes take the owner token, never an API key, and are relayed by the website like the other account routes.

Only HTTPS destinations are reachable: CONNECT to port 443 of a public address. A refusal names its reason in an X-Vitna-Error header, listed in the gateway reference. Bytes in both directions are metered and settled hourly into the account's statement as proxyBytes. They are priced per decimal GB at VITNA_WEB_BILLING_PROXY_GB_MICRO_USD. Settled and still-open hours both count against the month's allowance, which gateways enforce within about a minute of each other.

The operator configures the grants and gateways:

dotenv
VITNA_WEB_PROXY_ACCESS={"acct_ID":{"monthlyGb":50,"concurrency":10,"tunnelsPerMinute":600}}
VITNA_WEB_PROXY_GATEWAYS=[{"id":"egress-de","tokenHash":"GATEWAY_TOKEN_HASH","endpoint":"egress-de.example.com:8791","country":"DE"}]
VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000
Grant fieldDefaultMeaning
monthlyGbRequiredDecimal GB per UTC month, 1 to 100,000
concurrency10Tunnels open at once, 1 to 1,000
tunnelsPerMinute600Tunnels started per minute, 1 to 100,000
suspendedfalsetrue stops the account's access and closes its tunnels

A grant names an account this service holds, and public signup never creates one. A grant is the operator's check on the customer: confirm who the customer is and what the traffic is for before adding one. Changes take effect on restart, and gateways see them within a minute. Each gateway's tokenHash comes from npm run egress:gateway-key on that gateway, and its endpoint is the public host and port customers connect to.

Product and company data

text
POST /v1/product
POST /v1/company

Both accept a URL and the common browser options. Extraction reads published JSON-LD, microdata/RDFa, supported embedded application data, and page metadata, with bounded fallbacks for explicit labels and supported factual prose. These routes run locally without model configuration.

Product data contains name, description, price, currency, availability, sku, brand, images, offers, and variants. Published url, gtin, and mpn can also be returned. Records are matched using page identity, canonical URLs, identifiers, and primary-entity references. Related records for the same product can contribute offers and variants; unrelated product cards do not contribute a price to the selected product.

When offers publish different prices, price is null and priceRange can contain observed low and high bounds with currency. A published single bound remains a single bound. Prices in different currencies are not combined into a range. Offers preserve their own price, currency, availability, and url, plus published lowPrice, highPrice, offerCount, or seller. A variant price becomes the selected price only when the page identifies that variant; otherwise variant prices remain in variants and offers.

On a Shopify storefront, recognized by Shopify's own page markers, a product URL (/products/<handle>, under an optional locale or collection) also reads the store's own record from /products/<handle>.json. That record sits beside the page's as platform: name: "shopify", productId, handle, vendor, productType, tags, options, priceRange, and every variant with its sku, barcode, price, compareAtPrice, currency and options. Its values fill only the fields the page left empty: brand from the vendor, currency, images, and the sku and gtin of a product with one variant. The page's own price and availability stand. When the page's published data cannot be resolved, for example a graph past the 25,000-node bound, the store's record answers alone, with one offer per variant; when the store's record cannot be read, the page's record stands as on any other site. The record is a plain public HTTPS read with no browser and no further page quota.

Company data contains name, description, url, industry, locations, contactLinks, socialLinks, and products. Published legalName, foundingDate, employees, email, and telephone can also be returned. Multiple distinct contacts produce emails or telephones arrays. A scalar contact is returned when the page identifies a primary contact or publishes only one value. Public contact links and sameAs social profiles are retained; a directory's own footer contacts are not assigned to an external company it profiles. An unresolved choice between companies returns extraction_ambiguous (HTTP 422).

Unknown baseline scalar fields are null and baseline lists are empty. Additional fields appear only when supported by published values. Prices, dates, employee counts, locations, and contacts are not estimated from other entities, approximate prose, or missing data.

json
{ "url": "https://example.com/", "render": true }

Available fields depend on what the page publishes. These endpoints do not fill missing company facts or product details with invented values.

Store catalogs

text
POST /v1/catalog

List a Shopify or WooCommerce store's products from the store's own catalog:

json
{ "url": "https://shop.example.com/", "maxProducts": 250 }

url is any page of the store; its origin is the store. maxProducts accepts 1 to 1,000 and defaults to 100. The store's home page says which platform it runs on, through Shopify's or WooCommerce's own page markers or a WordPress REST link. When the home page names neither, as on a headless storefront, each platform's catalog is tried in turn, Shopify first, and each attempt counts as a page.

The response gives:

  • the store origin, after following at most two redirects (such as a myshopify.com address, or a bare domain, to the store's own address);
  • platform, shopify or woocommerce;
  • the store's currency;
  • products;
  • the catalog pages read;
  • truncated, when more products remain;
  • on WooCommerce, the store's total product count.

Each product carries id, handle, url, title, description, vendor, tags, options, images, priceRange, available (whether any variant is), and its variants. Each variant has id, title, sku, price, compareAtPrice, currency, available and options.

Shopify. The catalog is /products.json, read a page of 100 at a time. The currency comes from the storefront's own Shopify.currency script, and is null on a headless storefront without one. Every product has at least one variant. Products add productType, publishedAt and updatedAt, and variants add barcode.

WooCommerce. The catalog is the Store API that WooCommerce publishes under the WordPress REST root, read up to 100 products a page in product id order. The root is /wp-json/, or the one the home page names, such as /?rest_route=/ or a subdirectory install. A root on another origin is not followed.

Each page's variable products have their variations read beside it, 100 at a time and up to 250 per product, so each variant carries its own SKU, price and stock. A variant whose attribute accepts any value names no option for it. A simple product has no variants; its sku, price and available are its own. Prices are converted from the Store API's minor units, and vendor is the product's brand. Products add:

FieldValue
skuThe product's own SKU
typesimple, variable, grouped or external
categoriesCategory names
attributesThe product's attributes that do not define variants, each with its name and values
price, compareAtPriceThe product's one price and its regular price when on sale; null when its variations are priced differently
currencyThe store's currency code
onSale, onBackorderAs the store reports them
lowStockRemainingWhere the store shows a low-stock count
ratingaverage and count, where the product has reviews

The catalog is read with plain public HTTPS requests through the same guard as crawl discovery, with no browser. Every catalog read counts against the page quota: one per 100 Shopify products, and on WooCommerce one per page of products plus one per 100 variations. The request counts once against the request limits. A WooCommerce page may decode to 32 MiB; its compressed transfer stays within the 2 MiB bound of every public read.

A catalog request must finish within the service's request deadline, 30 seconds by default. WooCommerce stores often serve the Store API uncached, and a page of 100 products can take several seconds. For a large catalog on a slow store, set async: true to read it as a background job:

json
{ "url": "https://shop.example.com/", "maxProducts": 1000, "async": true, "jobTimeoutMs": 600000 }

The response is a job receipt (HTTP 202) with kind: "catalog". Poll GET /v1/jobs/:id: the job's progress counts each catalog read as it happens, and its result is the catalog answer described above. jobTimeoutMs (300,000 by default, up to 600,000) and webhookUrl work as on other background jobs, and both require async: true.

A job's result is stored whole, within five MiB. A catalog job therefore stops before the product that would take its result past that bound, and reads no more pages. It reports truncated: true with truncationReason: "result_limit", as a crawl job does. Products measured on live stores average about 4 KB, so 1,000 products usually fit. A catalog cut short by maxProducts names no truncationReason.

A service restart fails a running catalog job with job_interrupted; send it again. A queued catalog job survives a restart. A catalog dataset reads the same catalog on a schedule, with its 30-minute run deadline.

The store's robots.txt is read first, under the agent name VitnaWeb. A store that disallows its catalog to that agent, or to every agent, is refused with catalog_disallowed (403). The catalog is /products.json on Shopify and the Store API path on WooCommerce. The store's Crawl-delay, up to two seconds, is kept between reads.

These errors can also be returned:

  • catalog_unsupported (422): the site publishes neither catalog.
  • catalog_unavailable (502): a later WooCommerce page or variation read fails. A partial catalog is not returned.
  • destination_blocked (400): with VITNA_WEB_ALLOWED_HOSTS set, every read must stay in scope, including the redirects it follows.

Requests identify themselves as VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview).

Careers

text
POST /v1/careers

A company's open roles from its own job board on Greenhouse, Lever, Ashby, Workable, Recruitee or Personio, read from each platform's public job board API:

json
{ "url": "https://jobs.lever.co/acme", "maxJobs": 200, "descriptions": true }

url is the board (job-boards.greenhouse.io/NAME or boards.greenhouse.io/NAME, jobs.lever.co/NAME or jobs.eu.lever.co/NAME, jobs.ashbyhq.com/NAME, apply.workable.com/NAME, NAME.recruitee.com, NAME.jobs.personio.de or .com), one of its job pages, or a careers page that links or embeds exactly one board. maxJobs accepts 1 to 1,000 and defaults to 100. descriptions, true by default, includes each posting's description as plain text that keeps its lines; false leaves it out.

The response gives the board (platform, name, an EU Lever board's or a Personio board's region, and its public url), jobs, the pages read, truncated when more postings remain, and, for Greenhouse, the board's own total. Each job carries the same fields on every platform, null or empty where the board publishes nothing:

FieldMeaning
idThe board's id for the posting
title, url, applyUrlThe posting, its page, and where to apply
companyThe company name, where the board gives one (Greenhouse, Workable, Recruitee, Personio)
department, departments, teamThe board's own grouping
locations, countryEvery location the posting names
workplaceType, remoteremote, hybrid or onsite as the board states it, and whether the role is remote
employmentTypeThe board's own words, such as FullTime or Permanent
publishedAt, updatedAtISO timestamps, where published
compensationLever's or Recruitee's published salary range (min, max, currency, interval), or Ashby's summary where the company shows pay on its postings
descriptionThe posting's text, when descriptions is true
platformgreenhouse, lever, ashby, workable, recruitee or personio

Boards are read with plain public HTTPS requests through the same guard as store catalogs, with no browser, and the API host's robots.txt is read first under the agent name VitnaWeb. Every page read counts against the page quota: one for a Greenhouse, Ashby, Workable or Recruitee board, one per 50 postings on Lever (with Lever's Crawl-delay kept between them), and one for a careers page that has to be read to find its board. A Personio board is one read, or up to three with descriptions, since Personio's feed carried its description sections only when a language was named (measured 2026-10-07): it is read as it stands, then in English, then in German, until one carries them. With VITNA_WEB_ALLOWED_HOSTS set, the board's API host must be in scope as well as the url. Postings an Ashby board marks unlisted are left out, as are Recruitee offers not published. Workable says only whether a role is remote, so its other roles have no workplaceType. A renamed Recruitee company's board redirects to its new name, which is followed. A whole board's response may decode to 32 MiB; its compressed transfer stays within the 2 MiB bound of every public read.

HTTPCodes
403job_board_disallowed when robots.txt disallows the board's API, which Personio's does for a company it does not know
404job_board_not_found for a board that does not exist
422job_board_unsupported when no board is found at url; job_board_ambiguous when a careers page links more than one, naming each
502job_board_unavailable when the board answers with anything but its postings

The platforms document these APIs as public (retrieved 2026-10-07): the Greenhouse Job Board API, the Lever Postings API, Ashby's public job posting API, Recruitee's offers API (NAME.recruitee.com/api/offers/), and Personio's XML job feed; Workable's board widget API (apply.workable.com/api/v1/widget/accounts/NAME) is the one its own job widget reads. SmartRecruiters publishes a posting API too, but its API host's robots.txt disallows every agent but LinkedIn's, so it is not read. A careers page rendered only by script names no board in its HTML; send the board's own URL instead.

Investor Relations (POST /v1/ir)

text
POST /v1/ir

Discover and classify Investor Relations presentations, press releases, financial reports, earnings transcripts, and webcasts from a company URL or IR portal via Q4 feeds and rendered IR pages:

json
{ "url": "https://investor.nvidia.com/", "kinds": ["presentation", "press_release", "financial_report"], "maxItems": 25 }

Document parsing (POST /v1/parse)

text
POST /v1/parse

Parse a public HTTPS URL or base64-encoded PDF, DOCX, XLSX, or PPTX document into structured pages, typed blocks, extracted tables, Markdown, and character citation offsets:

json
{ "url": "https://example.com/report.pdf", "format": "both", "maxPages": 100 }

Claim verification (POST /v1/verify)

text
POST /v1/verify

Deterministically verify a factual claim against supplied URLs or native search results, returning per-segment support status, verbatim quotes with character offsets, content hashes, and a SHA-256 attestation digest:

json
{ "claim": "Example Domain is maintained for illustrative examples.", "urls": ["https://example.com/"], "maxSources": 3 }

Structural change detection (POST /v1/diff)

text
POST /v1/diff

Compare two public URLs (beforeUrl, afterUrl) or two inline text snapshots (beforeText, afterText) across prose, pricing, policy, or risk_factors modes with materiality scores and a SHA-256 attestation digest:

json
{ "beforeText": "Starter is $29/mo.", "afterText": "Starter is $39/mo.", "mode": "pricing" }

Entity alt-data dossiers (POST /v1/entity)

text
POST /v1/entity

Assemble a company dossier in parallel across requested modules (profile, careers, ir, infra, filings, news):

json
{ "domain": "nvidia.com", "ticker": "NVDA", "modules": ["profile", "careers", "ir", "infra", "filings", "news"] }

Enrichment matrices (POST /v1/grid)

text
POST /v1/grid

Enrich an entities by columns table synchronously (up to 10 entities, or asynchronously with async: true for up to 100 entities) using entity, search, verify, or source column strategies with cited source URLs, quotes, and attestation digests. Pass discover (with query, optional entityType, count, criteria, domains, excludeDomains, queries, index, and asOf) and optional hard budget controls (maxSearches, maxVerifications, maxDurationMs) to discover and verify matching entities in one call, with or without explicit entities or columns:

json
{
  "discover": {
    "query": "AI inference hardware startups",
    "entityType": "company",
    "count": 10,
    "criteria": ["founded after 2022", "builds custom silicon"]
  },
  "budget": { "maxSearches": 6, "maxVerifications": 20 },
  "columns": [{ "id": "org", "strategy": "entity", "field": "profile.name" }]
}

Pass the same discover, budget, entities, and columns definition inside source.grid on POST /v1/datasets to maintain it as a recurring living webset with incremental diffs, webhooks, and CSV/JSONL export.

Datasets

text
POST   /v1/datasets
GET    /v1/datasets
GET    /v1/datasets/DATASET_ID
DELETE /v1/datasets/DATASET_ID
POST   /v1/datasets/DATASET_ID/run
GET    /v1/datasets/DATASET_ID/records?run=latest&format=jsonl
GET    /v1/datasets/DATASET_ID/changes?after=RUN

A dataset is a collection you define once and Vitna keeps current. You give it a source, the field that identifies a record, and how often to run. Each run keeps a snapshot of the records and what changed since the run before.

json
{
  "name": "Competitor prices",
  "source": { "recipe": "RECIPE_ID", "urls": ["https://shop.example.com/p/1", "https://shop.example.com/p/2"] },
  "key": "url",
  "ignoreFields": ["checkedAt"],
  "intervalSeconds": 86400,
  "webhookUrl": "https://hooks.example.com/vitna"
}
SourceFieldsRecords
A recipe over pagesrecipe, urls (1 to 100)One per page: its url and the recipe's fields
A recipe over a siterecipe, url, maxPages (1 to 100, default 20), includePaths, excludePathsOne per page the site map finds
A store catalogcatalog (a Shopify or WooCommerce store's URL), maxProducts (1 to 1,000, default 250)One per product, as /v1/catalog returns it
A careers boardcareers (a board or careers page URL), maxJobs (1 to 1,000, default 250), descriptions (default true)One per posting, as /v1/careers returns it, keyed by id
Vitna's market recordmarket: model, tooling, extraction, speech, retrieval, or computeOne per figure on the class's tape, with an id naming its sku, seller, meter and any band
A grid (living webset)grid: discover, budget, entities, columns, concurrency (same contract as POST /v1/grid, without task fields)One flat record per matched or supplied entity (id, name, domain, ticker, url, optional verification, and each column id)

key names the field that identifies a record from one run to the next. It defaults to url (id for a market, a careers board, or a grid); records without it are skipped and counted. When a grid run hits a hard budget cap or a column cell error, the run is marked partial, preserves previously discovered records, and removes none. ignoreFields lists fields whose changes don't count, such as timestamps. intervalSeconds is 900 to 2,592,000, a day by default. The first run starts when the dataset is created. POST .../run runs it now; a dataset that is already running answers 409 dataset_running.

GET /v1/datasets/DATASET_ID lists the runs, newest first. Each gives its status, the counts of records, added, changed, and removed, the pages read, and the first 100 page errors with their total. A run's status is one of:

  • succeeded: it read every page.
  • partial: it missed a page or stopped at a limit. It keeps the records it could not reach as they were and removes nothing. A record leaves the dataset only on a run that read its whole source.
  • failed: its source could not be read at all. The previous records stay.

GET .../records returns a run's records as JSONL (format=jsonl, the default) or CSV (format=csv). CSV has one column per field, in the order fields first appear. Nested values are written as JSON, and text a spreadsheet would run as a formula gets a leading '. run is latest (the default) or a run number.

GET .../changes?after=RUN returns what each run after RUN added, changed, and removed, oldest first, whole runs at a time, up to 10,000 changes. Added and changed entries carry the record as it now is; removed entries carry the key. Read on with after set to the response's next; more says further runs follow. resync: true means runs after yours have been pruned, so read the records afresh. With webhookUrl, each run sends a dataset.run.completed event with its counts, retried with backoff up to five times.

Runs use the same browser slots, page quota, and accounting as API calls, one run at a time per service. A run that a restart interrupts starts again at the next tick. The limits:

LimitValue
Datasets per key5
Records per run5,000, each up to 16 KiB
Size per run8 MiB
Runs keptThe last retainRuns (1 to 30, default 10), within 32 MiB

The latest run is always kept, and so is the run the next one is compared against.

Published datasets

text
GET /v1/published
GET /v1/published/SLUG
GET /v1/published/SLUG/records?run=latest&format=jsonl
GET /v1/published/SLUG/changes?after=RUN

The operator can publish datasets that every key can read. Vitna's market record is the main example: each class's prices, kept current, with a feed of what moved. Published datasets take the same records and changes parameters. No key can run or remove them, and reading one counts as a request like any other.

dotenv
VITNA_WEB_PUBLISHED_DATASETS=[{"slug":"market-model","name":"AI model prices","source":{"market":"model"},"ignoreFields":["confirmedAt","fetchedAt","evidenceSha256","observationId"],"intervalSeconds":3600}]

Each entry is a dataset definition with a slug (up to 48 lowercase letters, numbers, and hyphens) and no webhook. A published dataset keeps its runs across restarts while its source and key stay the same. It starts afresh when either changes, and is removed when the configuration drops it. A market source reads the record's public tape at https://vitna.ai/api/v1/markets/{class}/tape; VITNA_WEB_RECORD_ORIGIN names another origin. Ignoring the confirmation and evidence fields, as in the example, means a price that is re-confirmed unchanged doesn't count as a change.

Firehose

text
GET /v1/firehose

A continuous stream of freshly crawled web pages: every news page in Common Crawl's news crawl (CC-NEWS), about 300,000 a day in many languages, read by cursor. Common Crawl publishes a news file every one to two hours. The service reads each file whole and keeps every HTTP 200 HTML page's title and text, in the order it read them, for a retention window of 72 hours by default.

text
GET /v1/firehose?after=0&limit=100&domain=reuters.com,de&language=en,de
ParameterDefaultMeaning
after0The cursor: records after this one. Send the previous answer's next
limit100Records per answer, 1 to 1,000
domainOmittedUp to 20 domains, separated by commas. A domain covers its subdomains, and a top-level domain such as de or a suffix such as co.uk covers every site under it
languageOmittedUp to 10 language codes, such as en or de, from the page's own lang attribute or Content-Language header
texttruefalse leaves each page's text out
json
{
  "records": [
    {
      "id": 4102,
      "url": "https://www.example.com/news/rates",
      "host": "www.example.com",
      "fetchedAt": "2026-10-07T14:58:25.000Z",
      "language": "en",
      "title": "Rates rise",
      "text": "The bank raised rates by a quarter point.",
      "htmlBytes": 280039,
      "digest": "sha1:NQ5RS4U4IVNJT3QEUFJYXWG45ZAZ2XDJ",
      "source": {
        "crawl": "CC-NEWS",
        "file": "crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz",
        "offset": 951,
        "length": 58004,
        "url": "https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz"
      }
    }
  ],
  "next": 4102,
  "more": true,
  "missed": false
}

Read on by sending next as after.

  • more is true while further records already match.
  • When more is false, the cursor has caught up and next holds its place, so polling with it returns only the pages read since.
  • missed: true means records past the cursor were pruned before they were read.

Each record's fetchedAt is when Common Crawl fetched the page. language is null when the page names none, and digest is the payload's SHA-1 as Common Crawl records it. source names the news file and the offset and length of the page's WARC record in it, so the page's HTML is one ranged read away:

bash
curl -r 951-58954 https://data.commoncrawl.org/crawl-data/CC-NEWS/2026/10/CC-NEWS-20261007145826-00176.warc.gz | gunzip

The firehose is off until the operator sets VITNA_WEB_FIREHOSE_PATH to its store's SQLite file. Until then it answers firehose_not_configured (503), and firehose_empty (503) while the store holds nothing yet. GET /v1/status reports capabilities.firehose and limits.firehoseRecords. While the firehose is on, it also reports firehose: the store's files, records, its newest latestFile, and the from and to of its pages' fetch times.

Each record returned adds one firehoseRecords to usage and to an account's statement. The ledger rate is VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD, and the operator estimate rate for static clients is VITNA_WEB_FIREHOSE_RECORD_USD. A read counts once against the request limits and uses no page quota.

Common Crawl publishes the crawl under its terms of use; the pages themselves remain their publishers'.

Monthly crawl backfill

text
POST /v1/firehose/backfill

A backfill reads one domain's pages from a Common Crawl monthly crawl (CC-MAIN), in the firehose's record shape, so a feed can start with a site's recent history. It is always a background job: the request answers HTTP 202 with a job of kind: "backfill". Poll GET /v1/jobs/:id; its result holds the pages.

json
{ "domain": "example.org", "path": "/news/", "language": "de", "maxPages": 200 }
FieldDefaultMeaning
domainRequiredA registered domain, such as example.org, with its subdomains
pathOmittedA path prefix, such as /news/, read on the domain itself and its www host
crawlThe newestA monthly crawl, such as CC-MAIN-2026-39
languageOmittedOne language code, such as de, as Common Crawl's own detection reports a page's languages
maxPages100Pages read, 1 to 500
jobTimeoutMs600,000The job's deadline, up to 600,000 milliseconds
webhookUrlOmittedA public HTTPS destination told when the job finishes

Common Crawl's URL index lists the domain's captures in URL order. A URL captured more than once in the crawl is read once, from its latest capture. Up to maxPages pages are read, each with one ranged request, and each record is checked against the index before it is kept: its URL, its length and its payload digest. Aim a backfill at a section with path, since URL order otherwise starts wherever the site's addresses sort first. A record's language is still the one the page declares, which can differ from the index's detection.

The result gives:

  • crawl and domain, and path or language when they were asked for;
  • records, each with source.crawl naming the monthly crawl, and no id, since a backfill's pages are not in the stream;
  • captures, the index rows read (up to five for each page asked for);
  • pages, the records read, and skipped, those that failed a check;
  • truncated, when more pages exist than were read.

A backfill can also stop early, keeping the pages it already read, with a truncationReason: result_limit when the job's five-MiB result is full, or commoncrawl_blocked or commoncrawl_unavailable when Common Crawl stops answering. Many publishers block Common Crawl's crawler, and a backfill of such a domain finds nothing (captures: 0).

Every Common Crawl request from every backfill takes its turn at one a second by default, so 100 pages take about two minutes. The host shares Common Crawl's allowance with Vitna Search's archive imports.

  • VITNA_WEB_FIREHOSE_BACKFILL_RATE raises the pace, up to 4 a second.
  • A busy index (a 429 or a 5xx) is asked again twice, 10 and 20 seconds later.
  • A 403 stops every backfill for 20 minutes. New ones answer commoncrawl_blocked (503) until then, and GET /v1/status reports firehoseBackfill.blockedUntil.

Each record returned is one firehoseRecords unit. Backfills are off until the operator sets VITNA_WEB_FIREHOSE_BACKFILL=true, and answer firehose_backfill_not_configured (503) until then. They do not need the stream's store. GET /v1/status reports capabilities.firehoseBackfill, limits.backfillPages and, while backfills are on, firehoseBackfill with its ratePerSecond. A service restart fails a running backfill with job_interrupted.

Structured topic events

text
GET /v1/firehose/events

Aggregates normalized public-registry and SEC filing events across up to eight structured topics (cisa-kev, federal-register, fda-drug-recalls, fda-device-recalls, treasury-debt, sec-8k, sec-13d, sec-form4) with a 5-minute in-memory TTL cache per topic:

text
GET /v1/firehose/events?topic=cisa-kev,sec-8k&since=2026-10-01&limit=20
ParameterDefaultMeaning
topicAll 8 topicsComma-separated or repeated topic names
sinceOmittedEarliest publication date formatted YYYY-MM-DD
limit20Maximum events returned across selected topics, 1 to 100

Ingestion

The Web service only reads the store. platform/web/firehose-run.mjs fills it, run on a timer (deploy/web-api/firehose.timer) as the Web service's own user, so both can open the file. Each run:

  1. Lists the current month's news files on data.commoncrawl.org, and the previous month's while the retention window reaches into it.
  2. Reads up to VITNA_FIREHOSE_FILES_PER_RUN files it does not yet hold, oldest first.
  3. Prunes pages past the window or the text budget, oldest file first.

A file is one request: about 1 GB and 26,000 pages, measured on 2026-10-07, read in about two minutes. That keeps far inside Common Crawl's request limits.

  • A 403 stops the run at once, with no further request, and the run exits with status 2 so the block shows in the journal.
  • A 429, a 5xx or a dropped connection stops the run until the next one.
  • A run that stops part-way keeps the pages it read and resumes at the next record with a ranged read, so no page is stored twice.
  • One run at a time: a lease in the store keeps a second run from starting while one is still going.
SettingDefaultMeaning
VITNA_FIREHOSE_PATHRequiredAbsolute path of the store's SQLite file; the Web service's VITNA_WEB_FIREHOSE_PATH names the same file
VITNA_FIREHOSE_RETAIN_HOURS72Hours of pages kept, by when their file was crawled, from 1 to 720
VITNA_FIREHOSE_MAX_TEXT_GB6Text kept at most, in GiB; past it the oldest files' pages go first
VITNA_FIREHOSE_MIN_FREE_GB10A run reads no file while the disk has less free space than this
VITNA_FIREHOSE_FILES_PER_RUN2Files read in one run, from 1 to 24

Pages average 4.2 KB of text, so 72 hours of news is about 4.5 GB.

The raw CC-NEWS crawl stream does not yet carry:

  • the monthly crawl as a stream, only as a backfill of one domain at a time;
  • each page's HTML in the answer;
  • delivery by webhook or to a storage bucket.

Structured public-registry event topics are served separately at GET /v1/firehose/events.

Research

text
POST /v1/research

Search and read bounded page sets, then optionally return extractive findings:

json
{
  "query": "What does this company offer?",
  "maxSources": 3,
  "answer": true
}

Discovery uses the local Vitna index by default; populate that index as described below, or supply your own urls array. maxSources defaults to 3 and accepts 1 to 20. With answer: true, the native engine selects observed source passages using lexical matches and returns numbered citations. Set answer to false to retrieve page content without an answer:

json
{
  "query": "What does this company offer?",
  "urls": ["https://example.com/"],
  "maxSources": 3,
  "answer": false
}

The response includes the query, source page content, an answer when requested, and page errors. Native passage answers begin with "Extractive findings from supplied sources". Supported comparisons and arithmetic begin with "Grounded comparison" or "Grounded calculation" and show observed operands, units, formulas, and numbered citations. Supported arithmetic includes sum, average, difference, ratio, percentage change, minimum, and maximum over named entities and explicit compatible attributes. Conflicting facts, units, billing periods, missing facets, and unsupported qualifiers refuse instead of guessing. generation carries engine: "vitna-native", mode: "extractive", planning: "intent_decomposition", and reasoning: "bounded_rules_v1". Optional analysis contains validated coverage and finite calculations with existing source IDs. This is a finite rule engine; general reasoning still requires a separately operated self-hosted model. Unsupported requests return HTTP 422 native_research_unsupported. Supplied URLs bypass indexed discovery. Disabling Search or the answer engine can return search_not_configured or model_not_configured when the request needs that engine.

Multiple searches and structured results

Use search rounds to refine discovery after earlier pages have been read:

json
{
  "query": "Compare available products and pricing for this company.",
  "maxSources": 10,
  "maxQueries": 6,
  "maxRounds": 3,
  "maxPages": 20,
  "planQueries": true,
  "answer": true,
  "async": true,
  "jobTimeoutMs": 300000
}
FieldDefaultMeaning
queryRequiredQuestion or research brief, up to 1,000 characters
maxSources3Readable source budget, from 1 to 20
maxQueriesmaxRoundsSearch-query budget, from 1 to 10
maxRounds1 (2 with followLinks)Search/read rounds, from 1 to 3
maxPagesmaxSourcesPage-attempt budget, from 1 to 40
followLinksfalseFollow query-relevant links discovered on read pages across rounds (same host when urls are supplied)
planQueriesfalseBounded query planning; deterministic source-based variants in native mode
schemaOmittedSupported JSON Schema for data; requires answer: true
streamfalseServer-Sent Events progress and final result
asyncfalseSubmit a durable background job
jobTimeoutMs300,000Background deadline, up to 600,000 milliseconds; requires async: true
webhookUrlOmittedPublic HTTPS completion destination; requires async: true

The input remains one query. The returned queries array records the searches actually executed. Up to 20 supplied urls bypass search discovery; they cannot be combined with planQueries: true. With followLinks: true, research scores unseen links found on each read page against the query terms and follows top-ranked candidates in subsequent rounds (scoped to the seed hosts when urls are supplied), bounded by maxRounds, maxPages, and maxSources. The response budget reports actual rounds, searches, page attempts, sources, model calls, context characters, limits, and the stopping reason.

Provide a schema to receive validated JSON in data alongside the answer and collected pages. Native mode extracts supported literal values; an optional self-hosted model can generate schema-shaped data:

json
{
  "query": "List the products this company sells.",
  "urls": ["https://example.com/"],
  "answer": true,
  "schema": {
    "type": "object",
    "properties": {
      "products": { "type": "array", "items": { "type": "string" } }
    },
    "required": ["products"],
    "additionalProperties": false
  }
}

Timelines and multi-attribute comparisons

Native Research can order observed historical events, select explicit earliest/latest events and calculate elapsed days between two cited dates. It accepts full ISO calendar dates, timezone-qualified ISO instants and unambiguous month-name/day/year dates. Year-only dates, numeric slashes, implicit timezones, mixed day/instant precision and crawl metadata are refused. Conflicting dates for the same named event remain ambiguous; requesting a year cannot resolve that contradiction.

Structured responses expose analysis.timeline with at most 24 ordered events, normalized dates, precision and source IDs. Elapsed calculations use cited epoch-millisecond operands and an explicit day result. Multi-attribute comparisons expose at most 24 analysis.comparisons cells with entity, attribute, observed display value and source ID. Explicit date filters apply before selecting historical numeric records; absent or conflicting cells are refused rather than silently filled. All emitted analysis must pass the finite public response contract.

The preserved native evaluation remains 44/44 cases. A separate authored timeline/comparison group passes 40/40, including 22 refusal diagnostics, in outputs/web-native-evaluation/report-v3.json. These fixtures exercise the bounded deterministic grammar, not open-ended general reasoning.

Stream progress

Set stream: true on a synchronous research request. The response content type is text/event-stream. Progress event names include started, round, planning, search, source, source_error, generating, and completed. Events include a monotonic sequence number and measured progress. The final result event contains the complete JSON response:

text
event: source
data: {"type":"source","sequence":3,"url":"https://example.com/"}

event: result
data: {"query":"What does this company offer?","answer":null,"mode":"sources","sources":[],"meta":{}}

This wire example is illustrative. A stream failure uses an error event with error.code and error.message. Inspect that event even if the HTTP status was 200. The synchronous stream deadline is at most 60 seconds and may be shorter under the deployment settings. Disconnecting cancels the request. The playground displays progress, the final result, and stream errors; cURL users should include -N to see events as they arrive.

Background jobs

Set async: true for research, extraction or a store catalog that needs a longer deadline. A firehose backfill is always a job. The create response uses HTTP 202 and includes id, kind, status, timestamps, jobTimeoutMs, progress, result, and error. Poll GET /v1/jobs/:id. Completed research, extraction, catalog or backfill output is in result; failed jobs carry error. DELETE /v1/jobs/:id cancels active work.

async and stream cannot both be true. Async jobs can use an optional completion webhook. Store the job ID even when using a webhook, so progress and delivery outcomes remain inspectable. Job scheduling and callbacks require the service process to keep running.

Sources

text
GET  /v1/sources
GET  /v1/sources/:provider
POST /v1/sources/find
POST /v1/sources/run

Sources are official data providers, federal agencies and registries with public APIs, that an application or agent can discover, read the contract of, and call with the same key as every other Web API. Each provider offers capabilities, and each capability publishes a contract: its input and output as JSON Schema, its price, and the terms its data comes under. Reading the list, a provider's contracts, or a find is free. A call through /v1/sources/run is metered once it has returned data.

Sources are off until the operator sets VITNA_WEB_SOURCES=true. GET /v1/status lists the enabled providers in capabilities.sources (empty while they are off), and every sources route answers sources_not_configured (503) until then.

ProviderCapabilitiesData
federal-registerdocuments/search, documents/getRules, proposed rules, notices and presidential documents, with agencies, dockets, comment deadlines and effective dates
ecfrsections/search, sections/textThe Code of Federal Regulations as in force today or on a past date, and a section's full text
usaspendingawards/search, agencies/listFederal contracts, grants and other awards, and each agency's budget authority and obligations
sec-edgarcompanies/lookup, filings/recent, filings/search, facts/conceptCompanies' CIKs, their recent EDGAR filings, full-text filing search, and XBRL financial facts
openfdadrugs/labels, drugs/reactions, recalls/search, devices/events, drugs/recalls, devices/recallsDrug labeling, adverse event counts, food/drug/device enforcement recalls, and MAUDE medical device events and CDRH recalls
clinicaltrialsstudies/search, studies/getRegistered clinical studies, their status, phase, eligibility and outcomes
cisa-kevcatalog/search, catalog/summaryCISA Known Exploited Vulnerabilities catalog search, ransomware campaign filter, and vendor summary
fredseries/observationsSt. Louis Fed FRED macroeconomic time series observations with period changes
finrashort-volume/ticker, short-volume/topDaily Reg SHO short sale volume by ticker and top shorted symbols
treasurydebt/daily, cash/tga, rates/averageTreasury Debt to the Penny, TGA operating cash balance, and average interest rates
nyfedrates/sofr, rates/effr, operations/rrpNew York Fed daily SOFR, EFFR reference rates, and overnight Reverse Repo operations
blsseries/observations, series/latestBureau of Labor Statistics CPI, unemployment, payroll, and price time series
courtlistenercourts/list, opinions/search, dockets/searchFree Law Project U.S. court directory, judicial opinions, and RECAP federal court dockets
fccfilings/search, proceedings/searchFCC ECFS telecommunications docket filings and rulemaking proceedings
eiaenergy/inventory, electricity/demandWeekly U.S. petroleum/natural gas inventories and hourly RTO electricity grid demand
dol-h1bemployers/approvals, lca/searchUSCIS H-1B Employer Data Hub approval/denial counts and DOL OFLC LCA wage filings

Find a capability

Describe what you need, and optionally narrow it to a category (government, finance, health or security) or a provider:

json
{ "query": "food recalls for listeria", "limit": 3 }

limit accepts 1 to 20 and defaults to 5. Each result is a whole contract, ranked by how well its title, keywords and description match the query, with its score:

json
{
  "query": "food recalls for listeria",
  "results": [
    {
      "score": 9,
      "provider": "openfda",
      "capability": "recalls/search",
      "title": "Search FDA recalls",
      "description": "Recall enforcement reports for drugs, foods or medical devices: ...",
      "input": { "type": "object", "properties": { "product": { "type": "string", "enum": ["drug", "food", "device"], "default": "food" } }, "required": [], "additionalProperties": false },
      "output": { "type": "object", "properties": { "total": { "type": "integer" } }, "required": ["total", "recalls"], "additionalProperties": false },
      "example": { "provider": "openfda", "capability": "recalls/search", "options": { "product": "food", "query": "listeria", "classification": "Class I", "limit": 5 } },
      "price": { "resource": "sourceRequests", "unit": "call", "microUsd": null, "discovery": "free" },
      "available": true,
      "terms": { "license": "CC0 1.0 Universal (public domain dedication)", "url": "https://open.fda.gov/license/", "attribution": "Source: openFDA, U.S. Food and Drug Administration.", "notices": ["Do not present this data as endorsed by the FDA."], "retrieved": "2026-10-07" }
    }
  ]
}

The schemas are shortened here. Without a query, find lists every capability in catalog order. GET /v1/sources lists the providers, and GET /v1/sources/:provider returns one provider with all of its contracts.

Contracts are closed: an option the input schema does not name is refused with invalid_options (400), and the message names the option and the rule it broke, such as options.title must be at most 50. Every object in an output schema is closed too, and a call whose mapped result does not match its output schema fails with source_contract (502) rather than reaching the caller in an undeclared shape.

Call a capability

Send the provider, the capability and its options. example in every contract is a body that works:

json
{ "provider": "ecfr", "capability": "sections/search", "options": { "query": "emergency exits", "title": 14, "limit": 5 } }
json
{
  "provider": "ecfr",
  "capability": "sections/search",
  "data": {
    "total": 101,
    "page": 1,
    "pages": 21,
    "sections": [
      {
        "citation": "14 CFR 25.807",
        "title": "14",
        "part": "25",
        "section": "25.807",
        "type": "Section",
        "heading": "Emergency exits.",
        "partHeading": "Airworthiness Standards: Transport Category Airplanes",
        "excerpt": "types of exits are defined as follows: (1) Type I. This type is a floor-level exit with a ...",
        "inForceFrom": "2016-12-30",
        "inForceUntil": null,
        "url": "https://www.ecfr.gov/current/title-14/section-25.807"
      }
    ]
  },
  "source": {
    "name": "Electronic Code of Federal Regulations",
    "publisher": "Office of the Federal Register (National Archives) and the U.S. Government Publishing Office",
    "url": "https://www.ecfr.gov/api/search/v1/results?query=emergency+exits&per_page=5&page=1&date=current&hierarchy%5Btitle%5D=14",
    "retrievedAt": "2026-10-07T05:00:00.000Z",
    "license": "U.S. Government work, not subject to copyright in the United States (17 U.S.C. 105)",
    "licenseUrl": "https://www.copyright.gov/title17/92chap1.html#105",
    "attribution": "Source: the Electronic Code of Federal Regulations (eCFR), Office of the Federal Register.",
    "notices": ["The eCFR is authoritative but unofficial. Legal research should be verified against the official annual CFR, the daily Federal Register and the List of CFR Sections Affected."],
    "modifications": "Vitna returns the fields this contract names, under its own field names, with markup removed. A text field is cut at a word boundary at the length its description states (4,000 characters unless it says otherwise)."
  }
}

Like every response it also carries meta. source.url is the upstream request the data came from, with any key removed. dataAsOf appears when the provider states how current its data is: openFDA's last update date, and ClinicalTrials.gov's processing timestamp exactly as its API writes it, with no time zone. Show attribution and notices wherever you display the data. ClinicalTrials.gov's terms ask redistributors to name it as the source, show the processing date, keep the data current and state modifications, which is what dataAsOf and modifications are for. Contact names, emails and phone numbers from study records are never returned.

Errors

CodeStatusMeaning
invalid_options400The options break the contract; the message names the option
source_not_found404No such provider or capability, or the provider has no such record
source_rejected400The provider refused the query, in its own words where it gives them
source_rate_limited429The provider is limiting requests; Retry-After says when to retry
source_capacity429Vitna has used the provider's daily allowance; it resets at 00:00 UTC
source_busy429Too many calls to that provider are already waiting
source_unavailable502 or 503The provider could not be reached or answered with an error (502), or it needs operator configuration (503)
source_response_too_large502The answer was over 2 MiB; narrow the request
source_contract502The provider answered in a shape that does not match the contract

Pace and allowances

Each provider has its own lane: its own concurrency, a minimum spacing between requests, and a daily ceiling where the provider publishes one. Vitna shares a provider's per-address allowance among all of its customers, so these are set below the providers' published limits (all read 2026-10-07):

  • SEC EDGAR allows 10 requests a second and requires every client to declare a user agent naming its operator and a contact address. Vitna sends 5 a second, and the provider is unavailable (source_unavailable, 503, with an unavailableReason in its contracts) until the operator sets VITNA_WEB_SOURCES_CONTACT.
  • openFDA allows 240 requests a minute and, without a key, 1,000 a day per address. Vitna keeps to 900 a day without a key, and 110,000 a day once the operator sets VITNA_WEB_SOURCES_OPENFDA_KEY, against openFDA's 120,000.
  • CourtListener allows keyless access to /courts/ (courts/list), while opinions/search and dockets/search use an API token supplied via VITNA_WEB_SOURCES_COURTLISTENER_KEY.
  • FCC ECFS and EIA v2 work out of the box with DEMO_KEY (900/day ceiling) and scale when VITNA_WEB_SOURCES_FCC_KEY or VITNA_WEB_SOURCES_EIA_KEY is set.
  • dol-h1b reads USCIS H-1B Data Hub employer CSVs directly for employers/approvals, and uses VITNA_WEB_SOURCES_DOL_KEY for DOL OFLC lca/search.
  • FederalRegister.gov and eCFR.gov allow automated access only through their APIs and publish no rate limit. Vitna sends 2 requests a second to each.
  • ClinicalTrials.gov publishes no rate limit. Vitna spaces its requests 1.2 seconds apart.
  • USAspending.gov publishes no rate limit for its search API. Vitna sends 4 requests a second, 2 at a time.

Requests identify themselves as VitnaWeb/0.1 (+https://vitna.ai/docs/web/overview), followed by the operator's contact address for SEC EDGAR. Reference lists (the SEC's ticker map, the eCFR title index, USAspending's agency list, ClinicalTrials.gov's processing date) are cached for an hour or a day once they load; a failed load is never cached.

Metering

GET /v1/sources, GET /v1/sources/:provider and POST /v1/sources/find are free. They count once against the request limits like any request. A POST /v1/sources/run that returns data adds one sourceRequests to usage and to an account's statement; a call that fails is not metered. The ledger rate is VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD, and every contract's price.microUsd shows it (null while it is unset). The operator estimate rate for static clients is VITNA_WEB_SOURCE_REQUEST_USD.

One call is one capability. Batches of calls, feedback that credits a call, and a terms acceptance step for providers whose terms need one are not yet available. None of the current providers asks for a separate acceptance step.

MCP

text
POST /v1/mcp

The Web API is also an MCP server, so an agent can find and call data sources, read pages and, where the operator offers it, search Google through the tools its client already understands. It uses the same vitna_web_ key or org key as every other route, sent as a bearer token:

bash
claude mcp add --transport http vitna https://web.vitna.ai/v1/mcp \
  --header "Authorization: Bearer $VITNA_WEB_KEY"

A client configured with JSON takes the same URL and header:

json
{ "mcpServers": { "vitna": { "type": "http", "url": "https://web.vitna.ai/v1/mcp", "headers": { "Authorization": "Bearer vitna_web_..." } } } }

Connect to the API host itself. The website relay at /api/web does not forward /v1/mcp or MCP's headers. GET /v1/status reports the endpoint and the protocol versions it speaks in capabilities.mcp, and unauthenticated GET /.well-known/mcp.json, GET /.well-known/agent-card.json, GET /openapi.json, and GET /llms.txt expose the machine-readable schema and full tool catalog before signup.

ToolWhat it doesRoute it calls
find_sourcesRanks source capabilities for a task, each with its input schema and a working examplePOST /v1/sources/find
describe_sourceA provider's contracts, or one capability's, with output schemas and termsGET /v1/sources/:provider
run_sourceCalls one capability and returns its data with its sourcePOST /v1/sources/run
read_pageReads one public page as markdown or text, cut at 20,000 characters unless maxCharacters says otherwise; also accepts onlyMainContent, includeTags, excludeTags, highlights, question, extract, injection and redactPOST /v1/read
batch_scrapeReads up to 20 public pages in one call with shared reader, DOM filter, highlights, question, extract, injection and redact options; per-URL errors are isolated in errorsPOST /v1/batch
extract_dataPulls fields from one page, up to 20 pages, or pages discovered from a starting URL, by selectors, a JSON Schema, a prompt (where a language model is configured), or a saved recipe; waits for the resultPOST /v1/extract
map_siteA site's URLs from its links and optionally its sitemaps, without reading themPOST /v1/map
crawl_siteStarts a background crawl and returns its job id: 25 pages of 20,000 characters each, for up to 5 minutes, unless asked otherwise; accepts per-page DOM filters, highlights, question, extract, injection and redactPOST /v1/crawl
get_crawlA crawl's status and a slice of its pages, 25 at a time and without their text unless includeContent asksGET /v1/jobs/:id
cancel_crawlStops a queued or running crawl; pages already read stay readableDELETE /v1/jobs/:id
research_webRuns bounded multi-page research across search hits or supplied urls, with optional followLinks and structured schema output; waits for the resultPOST /v1/research
list_open_rolesA company's open roles from its Greenhouse, Lever, Ashby, Workable, Recruitee or Personio board: the first 25 postings without their text unless maxJobs or descriptions says otherwisePOST /v1/careers
diff_pagesDeterministic structural change detection and redlines across two URLs or inline text snapshots with materiality scoring and SHA-256 attestationPOST /v1/diff
parse_documentParses PDF, DOCX, XLSX or PPTX into structured pages, typed blocks, tables, markdown and character citation offsetsPOST /v1/parse
verify_claimDeterministically verifies a factual claim against supplied URLs or search discovery with verbatim evidence offsetsPOST /v1/verify
entity_dossierBuilds a live company alt-data dossier across web profile, ATS roles, IR releases, DNS/SaaS infra, SEC filings and newsPOST /v1/entity
enrich_gridEvaluates a synchronous entities-by-columns enrichment matrix across entity, search, verify and official source strategiesPOST /v1/grid
discover_gridDiscovers and verifies up to 10 entities against explicit criteria with verbatim quotes and sha256: attestations under hard budget controls, with optional columns enrichmentPOST /v1/grid
investor_relationsDiscovers Investor Relations presentations, earnings releases, financial reports, transcripts and webcastsPOST /v1/ir
list_firehose_eventsReads normalized CISA KEV, Federal Register, FDA, Treasury, SEC 8-K, SC 13D and Form 4 event feedsGET /v1/firehose/events
capture_screenshotRenders a page in Chromium and returns a PNG or JPEG screenshotPOST /v1/screenshot
capture_pdfRenders a page in Chromium to PDF with pageRanges, scale and printBackground controlsPOST /v1/pdf
list_linked_documentsDiscovers or downloads PDF, DOCX, XLSX and PPTX files linked from a pagePOST /v1/documents
search_indexRanked pages from Vitna's own index, with optional domains, excludeDomains, maxAgeDays, asOf snapshot cutoff, ranking, effort, inline contents, and one-call live scrapePOST /v1/search
google_searchGoogle's results page for a query, in SerpApi's field names unless shape asks for Vitna's ownPOST /v1/serp on the search service

The source tools are listed only while sources are enabled, search_index and discover_grid only where a search engine is configured, and google_search only where the operator names a search service in VITNA_WEB_SERP_URL. read_page, batch_scrape, list_open_roles and the crawl tools ask for less than their routes do by default, because a whole page, job board or crawl can be more than a model should take in one result. A crawl tool's job deadline is the exception: it defaults to five minutes, where the route's defaults to one page's 30 seconds. extract_data does not offer the route's background mode; it waits for the result within the request deadline. Every tool is read-only except crawl_site and cancel_crawl, which start and stop a background crawl. A tool call is the REST request it stands for, sent with the caller's key, so it counts against the same request and page limits, is metered the same way (find_sources and describe_source are free, and a run_source call is one sourceRequests once it returns data), and appears in /v1/usage under the route's own operation.

google_search is the exception, because its route is not this service's. The call goes to the search service's POST /v1/serp with the caller's own org key, vitna_key_... (see one key), so the search service admits it, counts it against the org's search limits, applies its deadline, cache and fresh captures, and logs it, exactly as a direct request. It does not count against this service's request limit or appear in /v1/usage. A vitna_web_ key covers the Web APIs only and is not sent on: the tool answers platform_key_required.

Its arguments are SerpApi's, mapped onto the route's fields:

ArgumentRoute fieldAllowed values
qqueryRequired; 1 to 1,000 characters
glcountryTwo-letter country code, such as us
hllanguageLanguage tag, such as en or pt-BR
startoffset0 to 190; 0 unless given
nummaxResults1 to 10; 10 unless given
extrasextrastrue adds People Also Ask questions, related searches and ads
no_cachefreshtrue captures a new page rather than reuse the key's recent answer
deviceNonedesktop only. Vitna captures Google's desktop results page and has no mobile or tablet capture, so the tool refuses either as invalid_arguments
shapeshapeserpapi unless given, or vitna for the route's default fields

The result is the route's answer as structuredContent, in the shape described under SerpApi-shaped response. When Google does not answer and the search service hands the search to Bing, the result says so: meta.engine is bing, and in SerpApi's shape search_metadata gives bing_url instead of google_url. A search can take up to the search service's deadline, about a minute on the hosted service, so give the client's tool-call timeout at least that.

Results carry the route's response as structuredContent and as JSON text. When a route refuses a call, the tool result has isError: true and its text gives the route's code and message, such as invalid_options: options.title must be at most 50. or, for a rate limit, the number of seconds to wait. Arguments that break a tool's input schema are refused the same way, as invalid_arguments. An unknown tool or a malformed message is a JSON-RPC protocol error.

Protocol versions

The endpoint speaks protocol 2026-07-28 and the earlier 2025-11-25, 2025-06-18 and 2025-03-26, and keeps no session for any of them:

  • A 2026-07-28 request carries io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientCapabilities in params._meta, and mirrors its version, method and tool name into the MCP-Protocol-Version, Mcp-Method and Mcp-Name headers. A header that is missing or does not match the body is refused with -32020 (400), missing metadata with -32602 (400), and an unknown method with -32601 (404). server/discover reports the supported versions and the server's identity.
  • An earlier client opens with initialize and is answered with the version it asked for, or 2025-11-25 when it asked for one this server does not speak. Later requests name the version in MCP-Protocol-Version; a request without the header is read as 2025-03-26. No Mcp-Session-Id is issued, and one sent by the client is ignored.
  • A version the server does not speak is refused with -32022 (400), whose data.supported lists the versions it does.

Each POST carries one JSON-RPC message; batches are refused. Responses are always a single JSON object, never an SSE stream, so GET and DELETE answer 405. A request carrying an Origin header is refused with 403 unless the operator lists that origin in VITNA_WEB_MCP_ORIGINS (comma-separated), so only the browser origins the operator names can call it from a web page.

Capabilities and usage

text
GET /v1/status
GET /v1/usage

Use the same bearer key to inspect the configured capabilities and limits, or your client usage in the current UTC day. The status response indicates which engines and search discovery are configured, and capabilities.documents lists the document kinds read (limits.documentBytes and limits.documentCharacters bound them). Inspect engines, searchIndex, and Research generation metadata to distinguish native extractive behavior from a selected self-hosted model. The usage response includes request and page-attempt counts, limits, and recent request outcomes. Resource counters include browser milliseconds, render attempts, Search calls, answer/extraction-engine calls, the bytes moved through residential and ISP routes (residentialBytes, ispBytes), and the bytes an account's own proxy keys moved (proxyBytes), and the firehose records returned (firehoseRecords). searchRequests and modelRequests also count native in-process invocations; these names do not imply an external network request. Static client keys have no usage statement. API accounts can retain a metered usage ledger; payment collection is disabled.

Operator cost estimates

Optional rates let an operator estimate measured resource usage:

dotenv
VITNA_WEB_BROWSER_SECOND_USD=0.001
VITNA_WEB_SEARCH_REQUEST_USD=0.005
VITNA_WEB_MODEL_REQUEST_USD=0.01
VITNA_WEB_RESIDENTIAL_GB_USD=5
VITNA_WEB_ISP_GB_USD=2
VITNA_WEB_PROXY_GB_USD=3
VITNA_WEB_SOURCE_REQUEST_USD=0.002
VITNA_WEB_FIREHOSE_RECORD_USD=0.0002

These rates are illustrative configuration values, not Vitna prices. Each accepts a nonnegative USD rate per measured browser second, dispatched Search/model-engine invocation, source call that returned data, firehose record returned, or decimal GB (10^9 bytes) moved through a residential or ISP route or on an account's proxy keys. /v1/usage returns resources and a cost object with estimatedUsd, rates, basis: "operator_rates", and complete. When all rates are unconfigured, estimatedUsd is null. With only some rates configured, the estimate covers those resources and complete is false when an observed resource has no rate. It does not imply that the unpriced resources cost zero.

The estimate covers only the configured rate model. It is not a payment charge, a provider invoice, or a measurement of hosting and proxy bandwidth costs. Use provider billing records to reconcile external spend.

Account usage statements

Activated accounts retain idempotent resource observations and daily totals in the Vitna SQLite ledger. Browser durations, render attempts, Search calls, answer-engine calls, source calls that returned data, firehose records returned, bytes through residential and ISP routes, and bytes on the account's proxy keys are attributed to the account. Ledger pricing uses separate nonnegative integer microdollar rates, with one USD equal to 1,000,000 microdollars:

dotenv
VITNA_WEB_BILLING_BROWSER_SECOND_MICRO_USD=1000
VITNA_WEB_BILLING_SEARCH_REQUEST_MICRO_USD=5000
VITNA_WEB_BILLING_MODEL_REQUEST_MICRO_USD=10000
VITNA_WEB_BILLING_RESIDENTIAL_GB_MICRO_USD=5000000
VITNA_WEB_BILLING_ISP_GB_MICRO_USD=2000000
VITNA_WEB_BILLING_PROXY_GB_MICRO_USD=3000000
VITNA_WEB_BILLING_SOURCE_REQUEST_MICRO_USD=2000
VITNA_WEB_BILLING_FIREHOSE_RECORD_MICRO_USD=200

These are illustrative operator values. Browser observations round up to a whole millisecond, and each event charge rounds up to a whole microdollar. Route bytes are priced per decimal GB for every attempt that moved them, delivered or not, because the operator's provider bills them that way. The monthly /v1/invoices/YYYY-MM owner route returns usage, observed event counts, priced totals, rate completeness, and daily entries. Its kind is usage_statement and paymentCollection is false. pricedMicroUsd covers resources with a configured rate; totalMicroUsd stays null when an observed resource has no rate or accounting is incomplete. A current month remains accruing.

Check complete and billingStatus before treating a statement as settled. Failed or pending accounting can mark the statement incomplete even when an API operation returned successfully. The service retains pending observations for recovery; it does not silently report a complete bill for unrecorded usage. Ledger records have a bounded 400-day retention window; statements outside a fully retained month return invoice_unavailable. These statements do not charge a card, debit a prepaid balance, reconcile hosting bills, or deliver customer payment collection.

Native engines and optional endpoints

The default backends are in process:

dotenv
VITNA_WEB_SEARCH_BACKEND=native
VITNA_WEB_MODEL_BACKEND=native
VITNA_WEB_SEARCH_RERANK=true

Native retrieval uses Vitna's SQLite FTS5 index with title-weighted BM25 ranking and an optional local lexical reranking pass. Its file defaults to search.sqlite under VITNA_WEB_DATA; override it with VITNA_WEB_SEARCH_DB. Starting the service opens that index but does not crawl the web or add example documents. Empty or unrelated indexes can return no matches.

Populate selected public HTTPS sources explicitly:

bash
node --env-file=.env.web platform/web/native-search-crawl.mjs --seed https://example.com/ --max-pages 50 --max-requests 100

Repeat --seed for up to 100 seed URLs, or use --catalog file.json with the existing Vitna Search source catalogue format. Seed URLs cannot contain query strings. Crawls are bounded by --max-requests, --duration-ms, and --timeout-ms, respect the source scope, and add or refresh selected sources without removing other sites from the index. This crawler fetches public pages; it does not execute JavaScript. Use browser APIs for rendered pages. With configuration already in the environment, the equivalent script is available as:

bash
npm run web:index -- --seed https://example.com/

To refresh an explicit source list in the service, configure exactly one of VITNA_WEB_SEARCH_SEEDS (a JSON array of public HTTPS seed URLs) or VITNA_WEB_SEARCH_CATALOG (a local Search catalog file):

dotenv
VITNA_WEB_SEARCH_SEEDS=["https://example.com/docs/"]
VITNA_WEB_SEARCH_REFRESH=true
VITNA_WEB_SEARCH_REFRESH_MS=300000
VITNA_WEB_SEARCH_REFRESH_REQUESTS=30
VITNA_WEB_SEARCH_REFRESH_DURATION_MS=120000

Configured seeds or a catalog enable maintenance unless refresh is explicitly false. With neither supplied, startup makes no refresh requests. Due sources are selected by their oldest live page and catalog refreshHours, with bounded requests, duration, source batches, robots rules, conditional fetches, and the existing crawl lease. The default batch has three sources and seed sources have twenty pages each. searchIndex.maintenance exposes safe run summaries. Query text cannot enlarge this operator-defined source scope.

Native Web search returns at most 20 candidates per query. It supports site:hostname and -site:hostname filters. Retrieval and reranking remain lexical; they do not establish semantic understanding or a global web index. Native Search applies bounded evidence admission, with relevanceMode set to balanced by default or explicitly to strict. Balanced mode can retain useful partial wording but may miss unsupported constraints; strict mode requires all normalized content terms and can lose paraphrases. Native Research and schema extraction use no external model endpoint. They support observed facts, finite comparison rules, and compatible numeric calculations, and refuse missing or ambiguous evidence. They do not provide general language-model reasoning.

For a separate Vitna Search process, select the service backend explicitly:

dotenv
VITNA_WEB_SEARCH_BACKEND=service
VITNA_WEB_SEARCH_URL=http://127.0.0.1:8789
VITNA_WEB_SEARCH_KEY=YOUR_SEARCH_CLIENT_KEY
VITNA_WEB_SEARCH_MODE=index
VITNA_WEB_SEARCH_RERANK=true

Web discovery uses indexed search. The separate Search service retains its own SERP contract, but the Web engine does not select a paid provider or SERP fallback. The service adapter returns up to ten results per query; larger source budgets use multiple queries or supplied URLs. The Search service applies its own default retrieval: hybrid where its semantic retrieval is configured, keyword otherwise.

The MCP tool google_search is offered only when the operator names a Search service whose SERP route it calls, whichever backend serves index search:

dotenv
VITNA_WEB_SERP_URL=http://127.0.0.1:8789

Each call carries the caller's own org key there, never VITNA_WEB_SEARCH_KEY, so the URL must be https, or http on a loopback address. That Search service must have SERP retrieval enabled and accept org keys, which takes the same VITNA_PLATFORM_VERIFY_URL and VITNA_PLATFORM_SERVICE_TOKEN there as here.

For language-based extraction or synthesized research answers, select an operator-owned compatible model runtime. MODEL_URL is its full chat completions endpoint; model name and optional private key are separate:

dotenv
VITNA_WEB_MODEL_BACKEND=self-hosted
VITNA_WEB_MODEL_URL=http://127.0.0.1:8000/v1/chat/completions
VITNA_WEB_MODEL_KEY=YOUR_RUNTIME_KEY
VITNA_WEB_MODEL=YOUR_MODEL_NAME

The server requests JSON output, validates extraction against the schema, and checks that research answers reference the pages supplied to the model. Runtime credentials stay on the server. Setting endpoint variables requires the corresponding explicit backend; native mode does not silently call them. Each backend also accepts disabled to turn off that engine.

Deployment boundaries

The suite is an operator-configured service. Keep its client hash map, invitation hashes, and optional model/Search credentials in the service environment. Expose it through a TLS reverse proxy when remote access is needed. Website relay configuration belongs in the website environment.

The data directory configured with VITNA_WEB_DATA stores crawl jobs, monitors, quotas, and recent usage. Keep that directory across process restarts and run one service writer per directory. State is stored in state.sqlite using SQLite WAL transactions. A legacy state.json is imported once into a new database and retained as a recovery backup. Stop the service before copying the complete data directory for a filesystem backup, so the database and its journal files are captured consistently. The default request limits are 30 requests per minute, 1,000 requests per day, and 2,000 page reads per day, with two concurrent operations. Configure them using VITNA_WEB_RPM, VITNA_WEB_DAILY_LIMIT, VITNA_WEB_DAILY_PAGES, and VITNA_WEB_CONCURRENCY. VITNA_WEB_TIMEOUT_MS sets the default browser deadline, initially 30,000 milliseconds.

The browser service restricts requests to public HTTPS destinations and checks redirect destinations and resolved addresses. Rendering does not guarantee that a website will be reachable or that every delayed field will be ready. Choose a content selector with waitFor where appropriate.

See Unblocker setup for browser installation and proxy configuration, and Search setup for the separate search service.

The deployment package includes deploy/vitna-web.env.example, deploy/vitna-web.service, deploy/vitna-web.nginx.conf.example, and deploy/vitna-web.Dockerfile. These prepare a deployment; they do not publish a hosted endpoint. Browser settings use VITNA_WEB_BROWSER_PATH, VITNA_WEB_BROWSER_CHANNEL, and VITNA_WEB_BROWSER_PROFILE. Its auto reads in Chromium and moves a challenged read, page or document, to Camoufox, and it and camoufox take VITNA_WEB_CAMOUFOX_PYTHON, VITNA_WEB_CAMOUFOX_FINGERPRINTS and VITNA_WEB_CAMOUFOX_OS, with the meanings the Unblocker's settings of those names have; owned browser workers ignore all four. A service whose HOME does not last, as a systemd runtime directory does not, also needs XDG_CACHE_HOME set to the cache Camoufox's browser was fetched into; it reaches the fingerprint helper and no browser. The hosted Web API reads the hosted Unblocker's install: deploy/web-api/install-camoufox.sh gives /opt/vitna-camoufox the Web API user's group, starts Camoufox as that user under its unit's confinement, and prints the lines its env file needs (deploy/web-api/OVH-RUNBOOK.md, The Web API's auto browser profile). Each request's line in the service's log names the host it read, the engine that served the read (chromium, camoufox or cloakbrowser) and its attempts, with challenged when a challenge sent the read on; a request that read more than once, as /v1/documents does, also lists each read in reads (up to 20, a failed one with its code) and counts them in readCount. A job's reads are not on the line of the request that started it. vitna.ai keeps the host, engine and attempt count from an org's receipt, as it does from the Unblocker's. /v1/usage lists the requests alone. Proxy routes use VITNA_WEB_PROXIES or labelled VITNA_WEB_PROXY_ROUTES, and an optional exact-host scope uses VITNA_WEB_ALLOWED_HOSTS.

For labelled routes, configure private operator credentials:

dotenv
VITNA_WEB_PROXY_ROUTES=[{"id":"us-primary","country":"US","proxy":"https://user:[email protected]:8443"}]

The operator supplies and labels these routes. Country selection does not provision a proxy network or verify the provider location. Residential and ISP accounts from a provider the operator holds take the same list, with credential templates; see proxy routes. Keep proxy and integration credentials in private service configuration.

The supplied systemd unit and Docker image run one process with two browser slots by default. Configure queue, page, request, and job limits for your machine, and verify representative workloads before increasing them. SQLite persistence and larger budgets do not by themselves establish coordinator high availability or a hosted service. Owned browser workers can distribute rendering, as described under owned browser workers. The selected Vitna Search backend reads its populated index. The web suite does not provision a global web index. Owned CONNECT gateways can supply egress routes; see owned egress setup.

Operators: running the service

This section is for the operator of a Web API service and needs a checkout of Vitna's private repository. Nothing in it is needed to use the hosted pilot at https://web.vitna.ai, and it is not an offer of a self-hosted deployment. Until 2026-10-11 it opened this page as "Run locally", and the invitation, signup, recovery, quota, worker, metrics and benchmark material joined it from the customer half of this page the same day.

The web suite uses the browser runtime installed for Vitna Unblocker. It has its own client keys and service process.

bash
npm run web:install
npm run web:browser
npm run web:demo

The demo uses a real browser against local illustrative pages. It does not contact a target website or a paid model. To start your own service, mint a client key:

bash
npm run web:key -- pilot

Keep the printed token private. Put the printed client hash, rather than the token, into an ignored .env.web file:

dotenv
VITNA_WEB_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"}
VITNA_WEB_PORT=8790
VITNA_WEB_DATA=.web-data

Start the service with that configuration:

bash
node --env-file=.env.web platform/web/start.mjs

When the settings are already in the environment, use:

bash
npm run web:start

The default service address is http://127.0.0.1:8790. Browser operations use the installed Chromium runtime. Native engines make no model or search provider call at startup. Invite activation and owner key management are available through the account console when the website relay is configured. Customer payment collection and open self-service registration remain unavailable.

Issuing invitations

Static keys from web:key remain supported. For controlled account activation, create an operator invitation:

bash
npm run web:invite -- --client customer-01 --display-name "Customer application"

The command prints a one-time activationToken and an invites array. It does not contact the service or activate an account. Put only the returned hashed invites array in VITNA_WEB_INVITES, then start the service. Keep the activation token private. An account client identifier must differ from every static client identifier in VITNA_WEB_CLIENTS.

The CLI defaults to a 24-hour invitation, a 90-day API-key expiry, and a 366-day owner-token expiry, measured from invitation creation. Options --expires-hours, --key-days, and --owner-days can shorten those windows; the maximum invitation lifetime is 720 hours.

Optional public signup

Public signup is disabled by default. Enable it explicitly with VITNA_WEB_SIGNUP_ENABLED=true and a private, stable 32-byte hexadecimal VITNA_WEB_SIGNUP_SECRET. GET /v1/accounts/signup reports safe availability metadata. The account console at /apis/account offers signup when enabled.

Request a short-lived challenge with POST /v1/accounts/signup/challenge and an empty JSON object. The response specifies a SHA-256 proof over the UTF-8 bytes of challenge + ":" + nonce, with 18 leading zero bits and a bounded integer nonce. Submit the challenge, nonce and optional label to POST /v1/accounts/signup. This proof adds computational friction; it does not verify a human, email address or identity. No email or payment service is involved.

Account creation, signup counters and a hash-only replay receipt commit in one SQLite transaction. The successful response returns the API key and owner token once. Repeating the same successful signup returns public account metadata with duplicate:true and credentialsUnavailable:true; it cannot recover the credentials. Keep the original response privately.

Default issuance limits are three challenges per minute and 20 per UTC day per network bucket. Signup allows three accounts per network bucket per UTC day, 100 per coordinator per day and at most 1,000 retained accounts. IPv4 /24 and IPv6 /64 buckets are HMAC-hashed before storage. Failed proof attempts, retained network counters and replay receipts also have separate bounds.

Explicit positive integer envs can lower or adjust coordinator capacity: VITNA_WEB_SIGNUP_SIGNUPS_PER_DAY (1 to 100,000) and VITNA_WEB_SIGNUP_MAX_ACCOUNTS (1 to 1,000). The OVH pilot ran with 20 signups per UTC day and at most 100 accounts until the owner closed signup on 2026-10-11. These limits apply across network buckets and count existing native accounts; the account cap does not reset at midnight. Omitted variables preserve the defaults. Configuring capacities while signup is disabled, or using noncanonical/out-of-range integers, rejects startup. Set them together with explicit enablement.

The server uses its transport peer address. Headers supplied by a caller cannot change that address. Behind a reverse proxy, configure exact peer IPs in VITNA_WEB_TRUSTED_CLIENT_PROXIES and have that proxy overwrite X-Vitna-Client-IP with its observed client address. Missing or malformed identity from a trusted proxy fails closed. For website relays, also configure exact normalized egress IPs in VITNA_WEB_TRUSTED_CLIENT_RELAYS and the dedicated token SHA-256 in VITNA_WEB_CLIENT_RELAY_TOKEN_HASH. The relay sends X-Vitna-Relay-Client-IP plus X-Vitna-Relay-Token; the API checks its trusted socket, nginx-observed relay address and token before using the visitor IP. A configured relay without an authenticated assertion fails closed. The Next relay additionally requires a distinct nginx ingress token authenticated by VITNA_WEB_RELAY_INGRESS_TOKEN_HASH, and its private outbound VITNA_WEB_RELAY_CLIENT_TOKEN. It requires HTTPS beyond literal loopback. Nginx must overwrite ingress identity and establish trusted Cloudflare peers before reading Cloudflare identity. Caller X-Forwarded-For or CF-Connecting-IP headers alone are never trusted.

Offline operator recovery

If the owner token is lost or expired, or the activation response was lost after the invitation was redeemed, recover access through the operator's local account CLI. Stop the Web service first. The CLI opens the same single-writer SQLite data directory and refuses its live-writer lock; do not bypass that lock. Rotate the owner token by client identity:

bash
npm run web:account -- --data .web-data owner:rotate --client customer-01

The result contains a new ownerToken and ownerExpiresAt. Save the token privately, then restart the service so its credential caches reflect the change. The old token is invalidated. This recovery requires filesystem access to the data directory; it does not require the lost owner token or an HTTP admin credential. If --data is omitted, the CLI uses VITNA_WEB_DATA, then .web-data.

Other offline commands inspect account/key metadata or replace and revoke an API key:

bash
npm run web:account -- --data .web-data accounts:list
npm run web:account -- --data .web-data keys:list --client customer-01
npm run web:account -- --data .web-data key:rotate --client customer-01 --key REPLACE_WITH_KEY_UUID
npm run web:account -- --data .web-data key:revoke --client customer-01 --key REPLACE_WITH_KEY_UUID

Lists contain public metadata and hide credential hashes. key:rotate returns the replacement apiKey and key metadata once. Stop the service before these commands and restart it after mutations, preserving the single-writer boundary and refreshing cached credentials.

Shared quota admission

The owned quota authority coordinates admission across Web and Unblocker API processes over authenticated HTTP. Mint a separate operator credential and start a single authority with explicit authoritative service limits:

bash
npm run quota:key
npm run quota:start

The authority requires VITNA_QUOTA_TOKEN_HASH and VITNA_QUOTA_SERVICES; its private durable directory defaults to .quota-data, port to 8793 and host to loopback. deploy/vitna-quota.env.example documents service and per-client limit configuration. Non-loopback listening requires configured TLS certificate and key files. API processes explicitly select VITNA_WEB_QUOTA_BACKEND=shared, VITNA_WEB_QUOTA_URL and VITNA_WEB_QUOTA_TOKEN; standalone Unblocker uses the corresponding VITNA_UNBLOCKER_QUOTA_* variables. Partial configurations are rejected.

Authority limits override local quota settings. Each admitted API request and each browser attempt reserves its own atomic, idempotent operation. Retries consume page-attempt quota. Status/usage reads are free, scoped by service/client, and report the authority's limits and counters. Web resource timings and recent-operation lists remain scoped to their coordinator. Unreachable authorities fail closed before browser or search dispatch; uncertain admissions are never automatically replayed or failed over.

Deduplication IDs contain bounded timestamps and are retained for 24 hours. The same ID with a changed admission conflicts; an expired ID cannot become a new charge after pruning. A denied operation stays denied on replay even after quota rollover. New logical work needs a new ID. Backwards clocks cannot reset consumed windows. Counters/events, queues and HTTP bodies are bounded, and quota changes plus deduplication commit in one SQLite transaction.

The authority requires a dedicated private directory and rejects linked storage, reused account state and foreign database schemas before opening them for writes. Existing POSIX ownership/modes must be private; Windows operators must configure private ACLs. Run one writer per authority directory.

This adds shared admission, not full coordinator or authority high availability. Account credentials, usage statements, jobs and monitors still belong to one API coordinator; separate API directories do not replicate that state. Shared quotas do not make public signup accounts portable across coordinators. Session workers also retain their own writer directories and stable routing.

Owned browser workers

The default VITNA_WEB_BROWSER_BACKEND=native executes Chromium in the Web process. To separate browser execution onto machines you operate, run npm run web:worker on each worker and explicitly select the worker backend:

dotenv
VITNA_WEB_BROWSER_BACKEND=workers
VITNA_WEB_BROWSER_WORKERS=[{"id":"worker-a","origin":"https://worker-a.example.com:8792","token":"YOUR_SEPARATE_OPERATOR_TOKEN_AT_LEAST_32_CHARACTERS","concurrency":2}]

Workers require VITNA_WEB_BROWSER_WORKER_TOKEN_HASH. Generate its SHA-256 hash with npm run web:worker:hash, reading the token from private VITNA_WEB_BROWSER_WORKER_TOKEN configuration. Use a distinct operator token, not a customer API key. Nonlocal worker binds require the worker TLS key and certificate files. Loopback HTTP is supported for local development.

bash
npm run web:worker:hash
npm run web:worker

deploy/vitna-browser-worker.env.example, the systemd unit, and Dockerfile provide the full configuration and deployment templates.

The pool supports up to twenty fixed workers, bounded queues, and explicit per-worker concurrency. Tenant/session hashing keeps persistent sessions on one worker while the configured worker list and order remain stable. Session cookies and localStorage stay in that worker's RAM and expire or disappear on worker restart. Clearing a session uses a separate bounded control queue. The pool does not replay a failed browser invocation on another worker. Without that replay, a read through workers does not step up a network after a block; name network to use a dearer tier. Workers report the network and transferred bytes of each attempt, delivered or not, so the coordinator prices residential and ISP traffic as it would its own. Status reports engines.browser and whether execution is local or distributed. The API coordinator and SQLite directory still require one writer; this provides browser distribution without coordinator high availability.

The same worker process can also hold Browser API sessions: set VITNA_WEB_BROWSER_WORKER_CDP_SESSIONS on it and list it in the Web process's VITNA_WEB_BROWSER_API_WORKERS. See sessions on owned workers.

Operator metrics

Generate a separate operator token with npm run web:metrics:key. Retain the token privately and configure only its SHA-256 hash as VITNA_WEB_METRICS_TOKEN_HASH. Authenticated GET /metrics returns Prometheus text; use the token as a bearer credential. This route rejects queries and bodies and is disabled when no hash is configured. Customer keys cannot scrape it.

bash
npm run web:metrics:key

Counters and histograms cover HTTP outcomes and full response transfer, current requests, bounded queues, browser duration, engine invocations, resource bytes, and ledger pricing observations. Labels use fixed operation, method, status, and service values; URLs and customer identifiers are excluded. Counters reset on process restart. priced_micro_usd_total includes known partial prices; unpriced_events_total tracks incomplete pricing instead of treating it as zero. These are process observations, not a replacement for the durable account ledger.

deploy/vitna-api-alerts.yml supplies starter availability, error-rate, latency, queue, and metric-cardinality rules. Load it into your Prometheus configuration and configure your own alert delivery. The file creates no monitoring service or notification destination.

Benchmark locally

Run a repeatable baseline after installing Chromium:

bash
npm run web:benchmark -- --iterations 10 --concurrency 2

The runner measures every Web API product, both screenshot formats, and the existing JS Unblocker through real local HTTP requests and real Chromium. Each operation uses a fresh durable store. Pages and assets are fulfilled locally, with delayed JavaScript, article content, product variants, and company data. No external model, search provider, webhook, or proxy is used.

For a separate public-site coverage sample, run:

bash
npm run web:benchmark -- --only coverage --report outputs/web-benchmark/public-coverage.json

This makes two sequential Reader requests each to Python, React, MDN, GitHub, Hacker News, and an arXiv abstract, using automatic content readiness. Each response must contain a specified content marker, a minimum content length, and no truncation. It is a small diagnostic across six public sites, not a representative sample of web coverage.

The JSON report and readable HTML table are saved under outputs/web-benchmark/. Use --report path/to/report.json to choose another location. One warmup per operation is excluded. The measured sample count accepts 1 to 50 and client concurrency accepts 1 to 4; the service keeps its default two browser slots. Latency includes browser startup, cleanup, durable writes, and full JSON transfer and decoding. Successful latency and failure counts are separate, and success requires verified output rather than only an HTTP 200 response.

Crawl latency includes completion of three pages and polling every 100 ms. Monitor measures unchanged selector checks against independent monitors; baseline creation and deletion are excluded. Research measures collection of one supplied source, without search or generation. A ten-request sample is an initial baseline: its nearest-rank p95 equals its slowest observation.

To include two sequential reader requests to each of three public training pages (static quotes, JavaScript quotes, and a book catalog), add --live:

bash
npm run web:benchmark -- --iterations 10 --concurrency 2 --live

To repeat selected measurements, use --only reader,crawl with operation IDs from the report, or --only public to run only the six public samples. The report also measures synchronous storage-write time so slow jobs can be distinguished from browser and network delays. Monitor setup is excluded, and render-attempt concurrency includes startup and cleanup.

Public samples remain separate from the controlled results. This benchmark does not establish representative website coverage, competitor performance, hosted capacity, or external model and search-provider latency.

npm run web:load exercises actual local HTTP servers with a controlled five-millisecond fixture renderer, bounded concurrency, output validation, overload rejection, and a quota restart check. Its saved latency and throughput measure the API transport and admission path, not Chromium or production website capacity. npm run search:relevance separately evaluates authored relevance and abstention fixtures. Keep these scopes separate when comparing performance.

bash
npm run web:load
npm run search:relevance