Vitna Search
Private beta. A separate search API for agents with independent Google/Bing SERP retrieval and Vitna's own crawler and web index. SERP requests return actual search-engine organic listings; indexed search and cited research use successfully indexed pages. It includes a browser interface, dedicated client keys, and shared request limits. The private-key beta is hosted at https://search.vitna.ai, with an API reference at /docs. Its product page and a playground are at /search on vitna.ai. Keys are issued by invitation, requested at vitna.ai/start. Customer signup and search billing are not enabled.
The service
Use POST /v1/serp to retrieve actual Google or Bing text search result pages through Vitna's transport and parser, without a Bright Data call. The compatibility alias POST /request accepts the supported URL-based request shape. This alpha returns parsed organic listings or fetched HTML; it does not promise full vendor coverage for ads, shopping, maps, other SERP features, geographic egress, or challenges. See the SERP API contract for requests, operator setup, and limitations.
The native contract returns ranked URLs, titles, excerpts, source identity, and observed crawl timestamps, and on request each page's stored text and verbatim highlights (inline contents). It uses keyword retrieval with title-weighted BM25 ranking by default, removes duplicate URLs and fragments, and identifies result origins in provenance. Among its first 20 candidates, a result list leads with at most two pages per host (www. aside), and that host's further candidates follow the other sites'. Nothing is dropped, so a search confined to one site still fills its page. An instance configured with Vitna's own embedding engine adds semantic and hybrid retrieval and similar pages. A result carries a publication date only where the page itself declares one (declared publication dates); it never infers one, and it does not generate an answer. Query-guided expansion can discover and crawl a bounded set of pages, then make those pages searchable in Vitna's own index.
Exact single-brand queries prefer a matching indexed homepage under .com, .ai, or .org, including its www form. The homepage must still match the query and satisfy all filters; other results keep their BM25 ordering.
Running an instance is operator work, from a checkout of Vitna's private repository, and is described at the end of this page under Operator reference. Nothing there is needed to call the hosted API.
The process is separate from the site and the market collector. It has no access to the site's database and holds no signing key. Its optional SQLite search file belongs to this service alone. Provider search credentials belong only in this process's environment, not in the site's environment. Runtime search results and usage never become Witness price observations.
Google/Bing SERP
POST /v1/serp
POST /requestSend the search bearer key and JSON content type. For an actual Google result page:
{"engine":"google","query":"lunar science","country":"us","language":"en","maxResults":5,"format":"json"}The response contains general, input, organic, and meta, with meta.provenance: "search_engine" and meta.coverage: "organic_only". The organic entries expose rank, title, and link, with optional description, display_link, and observed sitelinks. Use engine: "bing" for Bing or format: "html" for the fetched HTML. SERP retrieval does not require an index, start a native crawl, or produce cited research. Read the complete SERP contract before migrating a URL-based client. For up to 100 distinct Google links across observed result pages, use the asynchronous multi-page jobs contract. Its target can finish partial; the single-page endpoint remains capped at ten. Collection requests default to fresh: false: exact same-client requests can join an active job or reuse a target-complete result for thirty seconds. Use fresh: true to bypass completed reuse. Capture timestamps remain unchanged, and cancelling a shared job affects its joined callers.
Enabled SERP retrieval defaults to auto: ordinary routes prefer a browser for Google and direct HTTPS for Bing, or pinned proxy browser routes when proxies are configured. A single deadline bounds distinct attempts, route concurrency, and cooldowns. Successful JSON results have a short process-local cache; send fresh: true for a new capture. Read the complete retrieval policy for operator settings and safe route/cache metadata. These controls do not establish production access to a search engine.
Search
POST /v1/searchSend Content-Type: application/json and a search bearer key. For example, with the issued credential in the caller's VITNA_SEARCH_KEY environment:
curl https://search.vitna.ai/v1/search \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"query":"agent search APIs","mode":"index","maxResults":5}'| Field | Default | Allowed values |
|---|---|---|
query | Required | 1 to 1,000 characters, no control characters |
maxResults | 5 | Integer from 1 to 100 |
mode | Explicit index recommended; browser default | index; mutually exclusive with provider and fallback; removed live and hybrid values return HTTP 400 |
provider | Legacy auto when mode is omitted | auto, vitna, brave, exa, tavily; cannot accompany mode |
fallback | Legacy false | Boolean; true only with legacy auto; cannot accompany mode |
domains / excludeDomains | Unset | 1–10 public hostnames; includes subdomains; index mode or legacy Vitna without external fallback |
maxAgeDays | Unset | Integer 1–3,650; observed crawl age; index mode or legacy Vitna without external fallback |
startPublishedDate / endPublishedDate | Unset | A date (YYYY-MM-DD, a whole UTC day) or an ISO 8601 date-time with a UTC offset, both inclusive; only pages that declare a publication date can match; index mode or legacy Vitna without external fallback |
asOf | Unset | Point-in-time snapshot cutoff (YYYY-MM-DD, end of UTC day, or ISO 8601 date-time with a UTC offset); searches indexed snapshots captured on or before that instant with zero lookahead, disables coverage expansion and paid fallback (as_of_requires_vitna), and echoes meta.asOf |
index | web | web or one of the 23 vertical indexes reported by GET /v1/indexes |
effort | Unset | fast, balanced, or deep; fast prefers keyword retrieval without reranking, balanced uses configured hybrid retrieval without default excerpts, and deep attaches contents.excerpts and applies native reranking when available; echoed in meta.effort |
retrieval | hybrid where semantic retrieval is configured, otherwise keyword | keyword, semantic, or hybrid; Vitna index without external fallback; semantic cannot accompany additionalQueries |
contents | Unset | text, highlights, and/or excerpts read from stored pages, optional maxTotalCharacters / maxTotalTokens budget, and injection guard; see inline contents; Vitna index without external fallback |
expand | Unset; permits automatic name/topic discovery | Boolean; true requests bounded expansion; false opts out of automatic discovery |
discover | Unset; permits automatic name/topic discovery | Boolean; explicit true requires expand: true; false disables site discovery |
urls | Unset | 1–8 public HTTPS page URLs without query strings; requires expand: true and must satisfy domain filters |
Unknown fields are refused. Request bodies are limited to 8,192 bytes. maxResults above 10 needs mode: "index" or legacy provider: "vitna"; a request that may reach an external provider keeps 1 to 10. The caller cannot set upstream provider endpoints, headers, or credentials. Domain, crawl-age and publication-date filters, and retrieval, require the Vitna index with external fallback disabled; otherwise the filters return HTTP 400 filters_require_vitna and retrieval returns HTTP 400 retrieval_requires_vitna. With discover: true, domains is limited to three hostnames and supplied URLs to three distinct origins; ordinary Index domain filters still allow ten.
Use mode: "index" for Vitna's stored corpus. It makes no paid search-engine call, including when it returns no matches. maxResults limits the immediate search response; it does not increase crawl work. A whole-request search deadline is separate from the background job's bounded crawler work.
An index search can return up to 100 results. A longer list extends a shorter one: its first results are those a 10-result search returns, with the same host spread among the first 20, and later results follow in relevance order. Every list keeps its duplicate suppression and filters. Caller-supplied additionalQueries are read as deep as the list asks, ten pages each or maxResults when larger, and pages found only below each query's top ten follow the rest; meta.index.retrieval.perQueryLimit reports the depth.
On a synthetic 50,000-page index with 384-dimension vectors, measured in one process on a Windows arm64 laptop and without the query embedding, 100 results took 30 ms median and 41 ms at the 95th percentile by keyword, against 11 ms and 13 ms for 10, and 61 ms and 77 ms hybrid, against 35 ms and 38 ms: far inside the default 10-second deadline. A repeated search is answered from the query workers' 30-second cache, which weighs a cached answer by its results, ten to a unit, so a 100-result answer takes the room of ten 10-result ones.
An empty native name/topic search, or a topic search with only partial-term matches, can automatically enqueue discovery when enabled, with no domains or supplied urls and no external fallback. For example, this illustrative name query needs no crawl flags:
{"query":"examplebrand","mode":"index"}When configured, automatic discovery first uses its own directory of observed names. Explicit expansion can also use matching known URLs. A narrow distinctive single ASCII brand can also produce up to three candidate hosts using .com, .ai, and .org. Those hosts are suggestions, not verified official identities. Name detection is heuristic; common topic words and multiword topic queries produce no guessed hostnames. Broad queries can match a publisher catalog's descriptions or, when configured, a local English Wikipedia title catalog to select candidate pages. Catalog entries are discovery starting points, not indexed coverage. A query without a suitable name, topic, or concept match can return no_candidates. This is not whole-web search. Current results return first with any job in meta.expansion; automatic admissions include meta.expansion.automatic: true.
Send expand: false or discover: false to prevent automatic discovery. For explicit target-only refresh, send expand: true, discover: false. For explicit site discovery, send expand: true, discover: true; domains remain optional guidance, not a prerequisite.
With expand: true, current indexed results return with meta.expansion: a durable job with a statusUrl, or an explicit no_candidates, queue_full, client_job_limit, discovery_unavailable, or unavailable status. Admission failure does not discard valid indexed results. With discover: false, the job crawls known relevant URLs and supplied targets with scope known_urls_and_supplied_targets.
Add discover: true to examine selected sites' homepages and sitemaps before choosing target pages. Optional domains guides site selection; without it, explicit discovery can use supplied URL origins, matching known URLs, observed names, narrow name candidates, topic-catalog matches, or configured concept-title matches. Automatic discovery skips incidental known-URL matches and uses the name or topic planners. The job has scope site_discovery, with at most three origins, 2,000 sitemap candidates, eight target pages, 40 total HTTP requests, and 60 seconds of crawler work. Candidate names do not establish official identity or promise whole-web coverage. An initialized empty index may accept expansion. See coverage expansion.
Bounded lexical and site hints, including names, candidate origins and their provenance, are encrypted in private jobs that expire after 24 hours. The raw original query and full request are not stored or logged. Deduplication uses a private HMAC digest. Plaintext queries and terms are absent from jobs and receipts; metadata receipts still record request outcomes and usage. The operator must configure a private VITNA_SEARCH_DEMAND_SECRET_FILE before accepting site-discovery jobs.
Authenticate GET /v1/search/jobs/<id> with the same client's key to receive {"job": {...}}. Status is queued, running, completed, or failed. Completed means the bounded pass finished; inspect its summary for indexed, blocked, failed, and unmet targets, then repeat the original search to retrieve fresh results. A manual rerun can send expand: false, discover: false to retrieve only the updated corpus. summary.discovery reports integer counters for origins, candidates, selected, and unavailable. Selected candidates are not indexed pages; inspect summary.indexed, summary.unchanged, and unmet targets for actual coverage. Job checks have a separate 60-per-minute limit and don't consume the search quota. Jobs expire after 24 hours. Unknown, expired, and other clients' jobs return the same 404 response.
Semantic retrieval
Semantic and hybrid retrieval are optional. They embed text with Vitna's own embedding engine: vitna-anchor's OpenAI-compatible POST /v1/embeddings, which runs on the search host as vitna-embed.service at 127.0.0.1:8795 for query embeddings and vitna-embed-batch.service at 127.0.0.1:8797 for vector maintenance, serving bge-small-en-v1.5 (384 dimensions). No hosted embedding, rerank, or search API is called. Both stay off unless VITNA_SEARCH_EMBED_URL and VITNA_SEARCH_EMBED_MODEL are set, and an unconfigured instance answers exactly as before.
retrieval | What runs |
|---|---|
keyword | The BM25 retrieval described above |
semantic | The pages whose stored vectors are nearest the query's embedding, by cosine similarity |
hybrid | Reciprocal rank fusion, with constant 60, of the top 20 keyword candidates and the top 20 nearest pages |
When the two hybrid lists swap a pair, their fused scores tie exactly and the embedding order decides. A request for more than 20 results reads both lists that deep; pages found only below both lists' top 20 follow the fused top 20, so the leading results stay those of a shorter request. A configured instance defaults to hybrid. Every mode applies the same enabled-source, domain and crawl-age filters, suppresses duplicate extracted content, honours maxResults, and labels results kind: "index" in provenance. The domain and crawl-age filters restrict the nearest-page scan itself, and the scan widens past copies of a nearer page, so neither empties a list that other admitted pages can fill. A matching brand homepage still leads only when it also matched the query's words. A page found only by its vector shows a query-focused excerpt when a query term occurs in it, otherwise its leading words.
meta.index.retrieval reports what ran. Illustrative values:
{"requested":"hybrid","used":"hybrid","model":"bge-small-en-v1.5","vectors":49012,"candidates":{"keyword":20,"semantic":20}}It is present whenever semantic retrieval is configured or a request names a mode. vectors counts pages with a stored vector, which can trail meta.index.documents until the maintenance job catches up. meta.index.match keeps its lexical meaning: it describes the keyword half of a hybrid search, for the first query as in keyword retrieval, and is absent from a semantic one. Automatic discovery follows the keyword half, so a hybrid search queues it exactly when a keyword search would have: when that half found nothing (candidates.keyword is 0) or matched only some terms (partial), even though nearest pages fill its results. Pages that additionalQueries found count as found, in either mode. An opted-in semantic search queues discovery only when empty.
The search never fails because of semantic retrieval. When the embedding call fails or exceeds VITNA_SEARCH_EMBED_TIMEOUT_MS, or no compatible vectors are loaded, the response carries keyword results with used: "keyword" and a fallback code: semantic_not_configured, embed_unavailable, embed_timeout, embed_paused, embed_rejected, embed_invalid_response, embed_model_mismatch, vectors_unavailable, vectors_incompatible, vectors_over_limit, or snapshot_as_of (when asOf runs lexical retrieval over historical snapshots). A fallback result is not cached. Receipts record the retrieval used and any fallback code, never the query.
After a query embedding times out or the engine cannot be reached, query embeddings pause for 30 seconds and searches report embed_paused, so an outage costs one timeout per pause rather than one per search. The first search after the pause tries the engine again while the rest stay paused, and a success ends the pause. Maintenance batches never pause. Because the embedding wait counts against the request deadline, startup refuses a VITNA_SEARCH_EMBED_TIMEOUT_MS above half of VITNA_SEARCH_TIMEOUT_MS, which leaves the keyword fallback at least half the deadline.
Page vectors live in their own SQLite file, vectors.sqlite beside the index unless VITNA_SEARCH_VECTORS names another, and never in index.sqlite. The file carries its own SQLite application id. The maintenance job refuses a vectors path that names the index, frontier, concept catalog or quota file, and the store refuses any file that belongs to another database; a vectors file from before the id existed is adopted, header only, on the job's next run. A file holds one model: its identifier and width are recorded once, and the API and the maintenance job refuse any other, so changing models means a new file. Each row keeps the fingerprint of the text embedded: by default the page title and its leading text, up to VITNA_SEARCH_EMBED_MAX_CHARACTERS (1,500) under a conservative token estimate, because single-window bge-small-en-v1.5 refuses texts over its 512-token window with HTTP 400 context_length_exceeded rather than truncating them. Operators can enable bounded overlapping passage chunking (VITNA_SEARCH_EMBED_MAX_CHUNKS, VITNA_SEARCH_EMBED_CHUNK_OVERLAP, VITNA_SEARCH_EMBED_MAX_DOCUMENT_CHARACTERS) and configure multilingual or long-context models (VITNA_SEARCH_EMBED_MAX_TOKENS, VITNA_SEARCH_EMBED_MULTILINGUAL); chunked matches deduplicate to the canonical document and pass matched passage byte/code-point offsets into contents.highlights. A maintenance job keeps the file current; operators run it as described under Maintaining the vectors file.
The query embedding is made on the API's main thread, so a slow engine never occupies a query worker, and its failure becomes keyword retrieval before any worker is involved. The nearest-neighbour scan runs in each query worker beside BM25. Each worker holds its own copy of the vectors: vectors × (width × 4 + 8) bytes, room for about 5% growth, and their URLs. For 49,063 pages at 384 dimensions that measured 79.5 MB per worker, with a 434 ms full load. VITNA_SEARCH_VECTORS_MAX_MB (512 per worker) bounds each copy. Workers apply new vectors and removals incrementally when the file changes: 500 changed vectors took 5.3 ms.
A file larger than that bound, such as one holding vectors for millions of imported Wikipedia articles, is searched through an approximate nearest-neighbour index instead, which the operator builds beside it (vectors.ivf beside vectors.sqlite). It is an inverted-file index: the vectors are split into up to 4,096 lists by two levels of spherical k-means, and each vector keeps one bit per dimension of its difference from its list's centre plus that difference's mean size, 52 bytes at 384 dimensions instead of 1,544. A search scores every list centre, estimates each vector in the VITNA_SEARCH_VECTORS_PROBES (64) nearest lists with one table read per eight dimensions, and rescores the best VITNA_SEARCH_VECTORS_RESCORE (400) estimates exactly from the floats stored in vectors.sqlite. Vectors stored or replaced after the index was built are held and searched exactly beside it, and a removed vector drops out because rescoring finds no row, so the index stays usable between rebuilds; the index and those newer vectors together must fit VITNA_SEARCH_VECTORS_MAX_MB. The index is plain JavaScript over typed arrays, like the exact scan, so the search host needs no native build. A file within the bound is always searched exactly. A file over the bound without an index built from it (a missing index, one from another vectors file or model, or more newer vectors than fit) answers with vectors_over_limit, and a damaged index with vectors_unavailable; either way the search falls back to keyword results as below. A query worker reads the index in the background, since at millions of vectors that takes longer than a query's deadline: after it starts, its semantic and hybrid searches fall back with vectors_unavailable until the index is in, and after a rebuild the index in use keeps answering until the new one replaces it.
Limitations:
- The default single-vector
bge-small-en-v1.5configuration embeds only English text well; multilingual retrieval requires deploying a multilingual profile (such asbge-m3ormultilingual-e5-small/multilingual-e5-basewithVITNA_SEARCH_EMBED_MULTILINGUAL=true). - Unless multi-passage chunking (
VITNA_SEARCH_EMBED_MAX_CHUNKS > 1) is enabled, only a page's title and leading text (withinVITNA_SEARCH_EMBED_MAX_CHARACTERSandVITNA_SEARCH_EMBED_MAX_TOKENS) are embedded, and text further down a page is found by keyword retrieval alone. - Nearest neighbours always exist, so
semanticreturns the closest pages even when none is relevant; there is no relevance threshold. On the authored negative queries, semantic and hybrid retrieval answered none with an empty result, where keyword retrieval answered 8% to 33% of them correctly. - Within the bound the scan is brute force and grows linearly with the number of vectors and their width. It measured 24.7 ms median for 49,063 vectors of 384 dimensions on a desktop Ryzen 7 3700X; the default 512 MB bound holds about 347,000. Past that, the approximate index answers instead, and can miss some of the exact nearest pages; see the measurement.
- A search embeds its query first: about 10 ms on an idle engine, longer while a maintenance batch is in progress, because the engine serves one request at a time.
- New and changed pages are found by keyword alone, or by their previous vector, until the next maintenance run.
- Source brief research and indexed text extraction remain keyword based.
The general web beta records the measured comparison of the three modes.
Legacy provider routing
Legacy gateway instances can omit VITNA_SEARCH_INDEX, explicitly select VITNA_SEARCH_PROVIDERS=brave,exa,tavily, and configure BRAVE_SEARCH_API_KEY, EXA_API_KEY, or TAVILY_API_KEY. Those accounts and contracts are separate from Vitna's native crawler; the hosted beta uses the native index. The Bright Data search adapter and its Live/Hybrid modes have been removed. Use the independent SERP API for actual Google/Bing listings, or native search for Vitna's indexed corpus. Existing clients should follow the migration guide.
Clients that omit mode retain the provider/fallback contract. auto uses the first configured provider in VITNA_SEARCH_PROVIDERS order, which defaults to vitna when an index path is configured, or brave,exa,tavily otherwise. Providers with no index/credential are skipped. This is a configured preference, not a claim that the first provider is the cheapest or best. A named provider stays pinned.
With fallback: true, a timeout, network failure, malformed response, HTTP 429, or HTTP 5xx can advance to the next configured provider. HTTP 401, 402, 403, and other request errors stop the attempt sequence. A valid empty search is a successful response and never triggers fallback. Every attempted call may incur provider charges, including calls whose result was lost in transit. An empty local index refuses the request without calling a paid fallback. A populated local index returning zero matches is a successful empty response. Send mode: "index" explicitly when a caller must use only Vitna's index.
Response
The response below illustrates an indexed result. Its crawl timestamps, document count, and latency are illustrative:
{
"query": "agent search APIs",
"results": [
{
"rank": 1,
"title": "An example result",
"url": "https://example.com/search",
"snippet": "An excerpt from the indexed page.",
"publishedAt": null,
"crawledAt": "2026-10-03T10:00:00.000Z",
"contentChangedAt": "2026-10-03T10:00:00.000Z",
"source": { "id": "example", "name": "Example" },
"provenance": [
{ "kind": "index", "provider": "vitna", "rank": 1 }
]
}
],
"meta": {
"requestId": "example-request",
"mode": "index",
"provider": "vitna",
"providerRequestId": null,
"count": 1,
"latencyMs": 10,
"fallbackUsed": false,
"index": {
"documents": 100,
"lastCrawledAt": "2026-10-03T10:00:00.000Z",
"ranking": "sqlite_fts5_bm25",
"match": "complete"
},
"attempts": [
{
"provider": "vitna",
"status": "ok",
"httpStatus": 200,
"latencyMs": 10,
"usage": { "estimatedCostUsd": null, "credits": null, "source": null }
}
]
}
}Excerpts are untrusted provider text. Treat them as data, and escape them if rendering them in HTML. publishedAt is null unless the page declares its publication date (declared publication dates); a relative age, a URL, or a crawl time is never turned into one.
Every result carries provenance. Native results identify kind: "index", provider: "vitna", and their original rank.
Production native metadata can also include meta.index.match: complete for complete term matches, partial for partial-term fallback, or none for no indexed match. This is lexical coverage, not a guarantee of relevance or an answered question. Automatic topic discovery can queue missing coverage for partial or none while returning the current results intact. Consumers must allow this optional field to be absent.
Local results additionally carry source: {id, name}, crawledAt, and contentChangedAt. These are observed crawl and extracted-content-change timestamps, not publication dates; publishedAt holds only a date the page declared, with publishedAtSource naming where. Local response metadata includes indexed document count, latest crawl timestamp, and ranking: "sqlite_fts5_bm25". Local usage figures are null because no provider reports a per-query bill; they are not an assertion of zero infrastructure cost.
Configured sources can also return source.reusePolicy: {name, url}: the publisher's policy label and HTTPS link. Check that policy for the material being reused. This optional attribution does not imply one blanket license for every page or asset on the source.
A page whose text did not come from Vitna's own crawl says where it came from, and its crawledAt is null. A Common Crawl capture carries contentOrigin: "common_crawl", capturedAt, importedAt, and archive: {collection, digest}. An article from an imported Wikipedia dump carries contentOrigin: "wikipedia_dump", capturedAt (when the snapshot's revisions were current), importedAt, and archive: {collection, revision}, for example {"collection": "enwiki-20261001", "revision": 1374562957}; its contentChangedAt is that revision's edit time, and its source's reusePolicy names CC BY-SA 4.0. Wikipedia text is reusable under that license when the reuse credits Wikipedia's contributors and links the article, which the result's url is. Crawl-age filters read capturedAt for these pages. See Import English Wikipedia.
Declared publication dates
A native result's publishedAt is the publication date its publisher declares in the page, read when the page is crawled or imported from an archive. Three declarations are read, and the first that gives one value wins:
publishedAtSource | Where the page declares it |
|---|---|
article_published_time | A <meta property="article:published_time"> (or name=) tag |
json_ld_date_published | datePublished on a JSON-LD article or posting, or failing that on the page itself (WebPage and its kinds), at the top level, in @graph, or as their mainEntity; a site's or a listed item's date is not read |
article_time_element | A <time datetime> inside an <article>: the one marked itemprop="datePublished" or pubdate, or else the only value the article's time elements give |
When a source gives two different values, such as several articles' dates on one listing page, it declares nothing and the next source is read. A value with a time and a UTC offset is returned as that instant in UTC, "2024-05-01T08:30:00.000Z". A bare date, or a time without an offset, stays the calendar day the page named, "2024-05-01", because the zone it was written in is unknown. A date more than a day after the crawl, a date before the year 1000, and the Unix epoch's first day, which content systems print for an empty field, are not stored. Pages without a declaration have publishedAt: null and no publishedAtSource, exactly as before. Nothing is inferred: not from a relative age such as "3 days ago", a URL path, visible text, a modification date, an HTTP header, or the crawl time. A page crawled before the index stored dates has none until it is crawled again, and an article imported from a Wikipedia dump has none: the dump declares revision times, not publication dates.
startPublishedDate and endPublishedDate filter on this declaration and apply only to pages that have one: a page without a declared date never matches a date filter, so a filtered search covers less of the index than an unfiltered one. Both bounds are inclusive. A date bound means the whole UTC day; a date-time bound needs a UTC offset and is compared as the instant it names. A page that declares only a date matches when that date falls within the bounds' dates. Semantic and hybrid retrieval restrict their nearest-page scan to dated pages in range; through the approximate index the range is applied as candidates are read back, as the crawl-age window is, so a narrow range there can return fewer pages than it admits.
{"query":"lunar missions","mode":"index","startPublishedDate":"2024-01-01","endPublishedDate":"2024-12-31"}The declaration is the publisher's claim, not a verified fact. A publisher can backdate a page, or declare an original print date for a page posted later.
Inline contents
A native search can return each result's stored page text and verbatim highlights with the results, so a caller needs no extraction request per URL. Send contents:
{"query":"Galilean moons","mode":"index","maxResults":5,"contents":{"text":{"maxCharacters":4000},"highlights":true}}| Field | Default | Allowed values |
|---|---|---|
contents.text | Off | true for up to 20,000 characters per result, or {"maxCharacters": n} / {"maxTokens": t} (1 to 100,000 characters, 4 characters per token, mutually exclusive) |
contents.highlights | Off | true for a 1,000-character passage budget per result, or {"maxCharacters": n, "maxTokens": t, "query": "..."} (1 to 4,000 characters, optional 1-1,000 character query) |
contents.excerpts | Off (auto on effort: "deep" for native index searches with extraction) | true for a 1,000-character contiguous excerpt window per result, or {"maxCharacters": n, "maxTokens": t, "query": "..."} (1 to 4,000 characters) |
contents.maxTotalCharacters / contents.maxTotalTokens | Unset | Positive integer response-wide character budget from 100 to 400,000 (maxTotalTokens from 25 to 100,000 at 4 characters per token, mutually exclusive); distributes budget across results and redistributes unused slack fairly |
contents.injection | Off | "scan" or "strip"; scans extracted text, highlights, and excerpts for prompt-injection phrases and invisible Unicode controls, attaching contents.injection (and stripping invisible controls when "strip") |
contents.snapshotAsOf | Unset | Point-in-time snapshot cutoff (YYYY-MM-DD or ISO 8601 date-time with UTC offset) for historical inline extraction; when paired with asOf, both values must match |
false leaves a part out, and a contents object that asks for no content part (text, highlights, or excerpts) returns an error. Other fields are refused. Contents require the Vitna index with external fallback disabled, otherwise HTTP 400 contents_requires_vitna, and an instance with page extraction, otherwise HTTP 503 extraction_unavailable. Both are checked before the request is admitted. When contents.maxTotalCharacters or contents.maxTotalTokens is requested, meta.contextBudget reports { requestedCharacters, returnedCharacters, estimatedTokens, truncated }. When contents.injection is "scan" or "strip", each extracted item's contents.injection reports { risk, findings } (plus stripped, the count of removed invisible Unicode characters when "strip" is used).
Contents come from the same stored text /v1/extract returns, read after ranking, under the stored source's and publisher's current policy. No page is fetched, and nothing is summarized or generated. They belong to the one admitted search and consume no further quota. Pages are read one after another, so a search with contents takes longer than one without. Each result then carries a contents object; these values are illustrative:
{
"status": "ok",
"contentChangedAt": "2026-10-03T10:00:00.000Z",
"totalCharacters": 5120,
"text": "Jupiter has dozens of known moons. …",
"characters": 4000,
"truncated": true,
"highlights": [
{ "text": "The four largest moons are called the Galilean moons.", "start": 812, "end": 865 }
],
"offsetUnit": "unicode_code_points"
}| Field | Meaning |
|---|---|
status | ok; page_not_indexed when the page is unavailable under current stored source or publisher policy; extraction_unavailable when the read failed. Otherwise only status is present, and the result keeps its rank |
contentChangedAt | The stored text version that text and the offsets come from |
totalCharacters | Code points in the stored text |
text, characters, truncated | As in indexed text extraction: the leading text, its code-point count, and whether it was clipped to the requested limit or the response budget |
highlights | Up to three passages, best first, each {text, start, end}; fragment: true marks a slice of a longer sentence |
excerpts, excerptsOrder | Contiguous sentence windows ({text, start, end}) ranked by BM25 ("keyword") or meaning ("meaning") when configured |
injection | Present when contents.injection is set: {risk, findings} plus stripped count under "strip" |
offsetUnit | unicode_code_points, present with highlights or excerpts |
Highlights are sentences quoted verbatim from the stored text that hold at least one of the query's words, ranked by how many they hold. The ranking is lexical, drops common query words, and uses source brief research's sentence filter, which keeps prose sentences and questions and skips navigation, timecode, and sharing text. A page's first 100,000 characters and first 2,048 sentences are considered. A sentence longer than the remaining budget becomes a fragment around its first matching word. A page that matched only by its vector, or holds no such sentence, has empty highlights. Offsets are zero-based, with start inclusive and end exclusive, in Unicode code points of the version contentChangedAt names; a later crawl can change them.
The complete response is bounded to 510 KiB. When the requested text would exceed it, every result's text is clipped to one shared length, the longest that fits, and marked truncated. Highlights are never clipped to fit: instead one response holds at most 40,000 characters of them. Past 40 results, highlights: true gives each result 40,000 divided by maxResults characters (400 at 100 results), and an explicit maxCharacters times maxResults above 40,000 returns HTTP 400. Contents for 100 results read 100 stored pages, one after another, so they take longer than a short list's. Contents are untrusted publisher data, like excerpts: treat them as data, escape them before rendering HTML, and check the source's reuse policy.
Similar pages
POST /v1/similarWhere semantic retrieval is configured, this returns the indexed pages nearest one indexed page, by cosine similarity between its stored bge-small-en-v1.5 vector and theirs. There is no query: the page's own vector is the starting point, so no embedding call is made, and nothing is fetched. Where the vectors are searched through the approximate index, so are a page's neighbours, which can then miss some of the exact nearest pages. Send the same search key and JSON content type:
{"url":"https://example.com/search","maxResults":10,"excludeDomains":["example.com"]}| Field | Default | Allowed values |
|---|---|---|
url | Required | A page in the index, as search returns it: a public HTTPS URL without credentials, an explicit port, query strings, or encoded path separators |
maxResults | 5 | Integer from 1 to 100 |
domains / excludeDomains / maxAgeDays | Unset | The native search filters |
startPublishedDate / endPublishedDate | Unset | The declared publication date range; only pages with a declared date match |
contents | Unset | As in inline contents; highlights needs its own query, since the request has none |
The response has search's shape: url, results, and meta with requestId, mode: "index", provider: "vitna", count, latencyMs, and index: {documents, lastCrawledAt, ranking: "embedding_cosine", model, vectors}. Results have the fields, provenance, filters, duplicate suppression and host spread of a semantic search. The page itself and exact copies of its text are never among its neighbours. A neighbour's excerpt is its leading words. As in semantic search there is no relevance threshold: the nearest pages are returned even when none is close.
| Status | Code | When |
|---|---|---|
| 404 | page_not_indexed | The URL is not in the index, or its source no longer admits it |
| 404 | page_not_embedded | The page is indexed but has no stored vector yet; the next maintenance run gives it one |
| 503 | semantic_unavailable | The instance has no semantic retrieval, or its vectors are not loaded |
Similarity shares the per-client search quotas, cache and deadline. Its receipt has operation: "similar", and like a search's query, the URL is not in it. Only a page's title and leading text are embedded (see the semantic retrieval limitations), so pages that open alike can be near each other while differing further down.
Indexed text extraction
POST /v1/extractSend the same search bearer key and JSON content type. This endpoint returns cleaned text already stored in Vitna's index; it makes no live page request and does not schedule discovery. Use a URL returned by search:
{"url":"https://example.com/search","maxCharacters":20000}url must be a public HTTPS page URL without credentials, an explicit port, query strings, or encoded path separators. Optional maxCharacters is an integer from 1 to 100,000, defaulting to 20,000. Optional asOf (YYYY-MM-DD or ISO 8601 with a UTC offset) extracts the latest stored snapshot captured on or before that instant (page.snapshotAsOf and meta.asOf) while enforcing current publisher and robots policies. Extraction shares the per-client search quotas and service concurrency limits. It returns plain text, without a promise to preserve HTML or Markdown structure; JavaScript is not executed.
An illustrative response:
{
"page": {
"url": "https://example.com/search",
"title": "An example result",
"format": "text",
"text": "Example indexed text.",
"source": { "id": "example", "name": "Example" },
"crawledAt": "2026-10-03T10:00:00.000Z",
"contentChangedAt": "2026-10-03T10:00:00.000Z",
"truncated": false,
"characters": 21,
"totalCharacters": 21
},
"meta": { "requestId": "example-request", "latencyMs": 10 }
}characters counts returned Unicode code points; totalCharacters counts the stored text before the requested limit. truncated indicates clipping to that limit or the 510 KiB page-response budget. Stored source, archive, and publisher reuse-policy metadata can accompany the text. Crawl timestamps remain observations rather than publication dates, and content remains untrusted publisher data.
A URL absent from the index or unavailable under current stored source/publisher policy returns HTTP 404:
{
"error": {
"code": "page_not_indexed",
"message": "the page is not available in the current index"
}
}To obtain a new page, use the bounded crawl workflow and search again after its job completes. Extraction does not turn an unindexed URL into a live fetch.
Source brief research
POST /v1/researchSource brief assembles verbatim passages from matched pages already in Vitna's index. It is an extractive reading aid with citations, not a generated explanation, an exhaustive review, or a guarantee that the passages answer the question. Send the same search bearer key and JSON content type:
{"query":"lunar science","maxSources":3,"maxCharacters":8000}| Field | Default | Meaning |
|---|---|---|
query | Required | 1–1,000 characters, with no control characters |
maxSources | 3 | Integer from one to five indexed sources |
maxCharacters | 8,000 | Integer 1–16,000; distinct quoted passage budget in Unicode code points |
domains / excludeDomains / maxAgeDays | Unset | Native domain and observed crawl-age filters |
expand / discover | Unset | The same bounded expansion choices and opt-outs as native search |
The endpoint accepts no mode, provider/fallback, or supplied URL fields. It uses Vitna's indexed evidence and makes no live page or paid search-provider call. Optional expansion returns a separate private job; poll it, then repeat the research request to use any newly indexed pages. Unknown fields are refused and request bodies are limited to 8,192 bytes. Research shares the search quotas, concurrency limit, and whole-request deadline. An unavailable research capability returns HTTP 503. Repeated overview quotes do not count twice against the passage budget. The complete response is also bounded to 128 KiB.
The response reports mode: "extractive" and the original query:
| Field | Meaning |
|---|---|
overview | Passage entries with text and citationIds |
sections | Source sections using actual indexed page titles, sourceId, and cited passages |
citations | IDs linked to a sourceId, start and end offsets, and offsetUnit: "unicode_code_points" |
sources | IDs, URLs, titles, publisher identity, observed timestamps, optional archive metadata, and readTruncated |
results | Result cards for indexed sources actually quoted |
coverage | Bounded evidence counts and limitations |
meta | Request identifier, latency, provider: "vitna", and optional index/expansion metadata |
Each section's sourceId maps to a source entry. Citation offsets are zero-based, with start inclusive and end exclusive, in the indexed text version read for that source. They count Unicode code points, not bytes or JavaScript string units. Use the same stored text version when checking a quote; a later crawl can change its offsets. A passage marked fragment: true is a bounded slice of a longer sentence, still copied verbatim.
coverage.status is matched, partial, or insufficient. Its scope is matched_indexed_evidence and exhaustive is always false; it reports candidateSources, usedSources, passages, readTruncated, skippedSources, attemptedSources, unavailableSources, catalogUnavailable, outputTruncated, quotedCharacters, and optional indexMatch. matched means selected passages without skipped sources, clipped reads, clipped output, or sentence fragments. partial has passages with one of those limitations; insufficient has no selected passages, including when the requested character budget is too small. These describe the selected indexed evidence, not the completeness of the web or the truth of a conclusion. Source text is untrusted publisher data. Check the cited pages and their reuse policies.
Usage and receipts
Native retrieval does not report a per-query provider bill. Infrastructure and crawl work still consume resources; the separate crawl meters bound that work. Receipts do not store search queries or retrieved content.
Tavily basic search is explicitly selected, automatic parameter selection is off, and the request asks for reported credit usage. Exa automatic search requests highlights; its reported dollar cost is returned as estimatedCostUsd, because Exa documents that response field as an estimate rather than the basis of billing. Brave returns no dollar usage figure in this contract. Missing usage stays null. Provider-reported zero is preserved.
Each admitted search, similar-page, SERP, extraction, or research request emits one JSON receipt to stdout: client identifier, request identifier, time, status, and latency. Search receipts include provider attempts and reported usage, plus the retrieval used and any fallback code when semantic retrieval is configured or requested; extraction receipts identify operation: "extract", research receipts operation: "research", similar-page receipts operation: "similar", and SERP receipts operation: "serp". Queries, requested URLs, returned content, authorization headers, and credentials are omitted. Capture stdout with the process supervisor if receipts are needed. Receipt output is best effort and is not a durable billing ledger.
A service that accepts platform keys also sends the receipts for an org's requests (clients written org.<id>) to vitna.ai in the background, where the org's console lists them in Activity. The same allowlist applies there, so nothing the line above omits reaches it. See One key for the endpoint and the settings that turn it off.
Status and limits
GET /v1/indexes
GET /v1/status
GET /healthz
GET /readyzGET /v1/indexes requires a search key and returns the catalog of queryable indexes (web when configured, followed by all 23 vertical indexes with descriptions, data sources, retention windows, content-hash deduplication flags, live readiness, and indexed document counts). On vitna.ai, the public Vitna Library catalog lists these vertical indexes and Web API data sources.
Status requires a search key and reports the client identifier, configured providers, first provider, supported modes, and limits. It exposes safe availability information for configured operator routes, without credentials. The separate serp capability reports the retrieval adapter's own Google/Bing availability and engines, with safe transport settings. Its retrieval metadata includes configured attempt limits, cache statistics, and safe per-route/per-engine health and concurrency counters. Runtime-unavailable routes remain disabled until restart. It does not probe a search engine or disclose proxy credentials. expansion.siteDiscovery reports enabled, automaticNames, automaticTopics and automaticConcepts (true when configured), limits: {origins: 3, candidates: 2000}, and safe name-directory and publisher-catalog aggregate counts. topicCatalog identifies source: "local_topic_descriptors" and reports publishers (unexpired descriptors), with observed and reviewed counts. candidates counts separate unverified domain metadata; imports records dataset attribution and retrieval metadata. None of these are indexed document counts. Optional conceptCatalog reports source: "wikipedia_titles", language: "en", titles, snapshot, importedAt, sourceUrl, license: {name, url}, and a changes description of normalized lookup keys with original page titles retained. Its title count measures metadata, not indexed articles. Redirect titles have no canonical-target mapping in this catalog; article text becomes usable only after an allowed page is fetched and indexed. Health is public and reports only that the HTTP process is up; it makes no paid provider probe. Status also reports extraction: {enabled, indexedOnly: true, limits: {defaultCharacters: 20000, maxCharacters: 100000}} and contents: {enabled, indexedOnly: true, limits}, whose limits are defaultTextCharacters, maxTextCharacters, defaultHighlightCharacters, maxHighlightCharacters, highlightsPerResult, highlightReadCharacters, and responseBytes. research reports enabled, mode: "extractive", and limits: {maxSources: 5, maxCharacters: 16000}. similar reports {enabled, indexedOnly: true, limits: {maxResults: 100}}, enabled where semantic retrieval is configured. Top-level snapshotAsOf: true indicates support for point-in-time asOf and contents.snapshotAsOf queries. Where semantic retrieval is configured, status adds retrieval: {default: "hybrid", modes, semantic}, whose semantic reports the model, the loaded vectors count, dimensions, the bytes one query worker holds, and a state of ready, missing, empty, incompatible, over_limit, or unavailable. When the vectors are searched through the approximate index, semantic.approximate adds indexed (vectors in the index), exact (vectors stored since its build), lists, probes, rescore, and builtAt. The index status counts importedDocuments beside liveDocuments and archivedDocuments, and a bulk source in sources reports its contentOrigin. The browser uses ordinary result search by default. Source brief is an explicit option when this capability is available; it shows only pages with usable quoted evidence and can return fewer cards than ordinary search. Readiness is also public and follows the default /v1/search route. When the native index is the default provider, readiness returns HTTP 503 while that index has no documents or its status cannot be read, whether or not SERP retrieval is enabled. A SERP-only instance reports its retrieval adapter's own availability, without probing a search engine. Authenticated status remains available with enabled SERP if an index status read fails, reporting index: {ready: false, unavailable: true}. Authenticated status includes local source counts, crawl freshness, and the latest crawl summary when the Vitna provider is configured. The root / serves a browser interface that keeps a pasted key only in that open page.
| Environment variable | Default | Meaning |
|---|---|---|
VITNA_SEARCH_HOST | 127.0.0.1 | Listening address |
VITNA_SEARCH_PORT | 8789 | Listening port |
VITNA_SEARCH_PROVIDERS | vitna | Unique ordered provider identifiers; external adapters require explicit selection and credentials |
VITNA_SEARCH_INDEX | .search-data/web/index.sqlite | Independent SQLite index; initialized if needed |
VITNA_SEARCH_SERP_ENABLED | false | Enables independent Google/Bing SERP retrieval without requiring a native index; also requires VITNA_SERP_RESALE_APPROVED=true |
VITNA_SERP_RESALE_APPROVED | false | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted |
VITNA_SEARCH_SERP_TRANSPORT | auto | Bounded direct/browser routes; proxies pin browser egress; explicit direct or browser also supported |
VITNA_SEARCH_SERP_TIMEOUT_MS | 10,000 | SERP retrieval timeout within the shared whole-request deadline |
VITNA_SEARCH_SERP_PROXIES | Unset | Private operator JSON array of browser proxy URLs |
VITNA_SEARCH_MODE | Unset | web selects the general beta's default index and frontier paths |
VITNA_SEARCH_FRONTIER | Unset outside web mode | Existing separate crawl-state file; adds authenticated crawl status |
VITNA_SEARCH_CONCEPTS | .search-data/web/concepts.sqlite in web mode; unset otherwise | Separate SQLite title-metadata catalog; optional native concept discovery |
VITNA_SEARCH_DEMAND_SECRET_FILE | Unset | Private file containing a 64-character lowercase hexadecimal key; enables encrypted site-discovery jobs in web mode |
VITNA_SEARCH_QUOTA | Unset | Separate writable SQLite quota file; persists per-client minute/day limits |
VITNA_SEARCH_RPM | 30 | Admitted search, SERP, extraction, and research requests per client per fixed minute |
VITNA_SEARCH_DAILY_LIMIT | 1,000 | Admitted search, SERP, extraction, and research requests per client per UTC day |
VITNA_SEARCH_CONCURRENCY | 4 | Simultaneous search, SERP, extraction, and research requests across the instance; a SERP search waiting for a busy route's slot counts, so keep it well above VITNA_SEARCH_SERP_OWNED_CONCURRENCY |
VITNA_SEARCH_TIMEOUT_MS | 10,000 | One whole-request deadline, including sequential legacy fallback attempts |
VITNA_SEARCH_EMBED_URL | Unset | Embeddings endpoint ending in /embeddings, HTTPS or loopback HTTP; with the model, enables semantic retrieval |
VITNA_SEARCH_EMBED_BATCH_URL | Unset | Optional second endpoint, held to the same rules, that vector maintenance uses instead, so a query never waits behind a batch; queries always use VITNA_SEARCH_EMBED_URL |
VITNA_SEARCH_EMBED_MODEL | Unset | Model identifier sent to the engine; a different echoed model is refused |
VITNA_SEARCH_EMBED_KEY | Unset | Optional bearer token for a private remote endpoint |
VITNA_SEARCH_EMBED_QUERY_PREFIX | Empty | Prepended to each query, for example Represent this sentence for searching relevant passages: for BGE models |
VITNA_SEARCH_EMBED_DOCUMENT_PREFIX | Empty | Prepended to each page's text and covered by its fingerprint |
VITNA_SEARCH_EMBED_DIMENSIONS | Unset | Expected vector width; responses of another width are refused |
VITNA_SEARCH_EMBED_TIMEOUT_MS | 2,000 | Query embedding deadline before the keyword fallback, 100–60,000 and at most half of VITNA_SEARCH_TIMEOUT_MS |
VITNA_SEARCH_EMBED_BATCH_SIZE | 4 | Pages per maintenance request, 1–256; an engine serves one request at a time, so without VITNA_SEARCH_EMBED_BATCH_URL a larger batch makes searches wait longer |
VITNA_SEARCH_EMBED_BATCH_TIMEOUT_MS | 60,000 | Deadline per maintenance request, 1,000–600,000 |
VITNA_SEARCH_EMBED_MAX_CHARACTERS | 1,500 | Characters of title and leading text embedded per page, 100–8,000 |
VITNA_SEARCH_VECTORS | vectors.sqlite beside the index | Separate vectors file, never the index, frontier, concept or quota file |
VITNA_SEARCH_VECTORS_MAX_MB | 512 | Bound on each query worker's copy of the vectors, 1–16,384; a larger file is searched through its approximate index, which with the vectors stored since its build must fit this bound |
VITNA_SEARCH_VECTORS_PROBES | 64 | Lists of the approximate index scanned per search, 1–4,096 |
VITNA_SEARCH_VECTORS_RESCORE | 400 | Approximate candidates rescored exactly per search, 10–4,096 |
VITNA_SEARCH_EMBED_BULK | false | true makes vector maintenance embed the pages of bulk imports too, after crawled pages |
Search, SERP, extraction, and research requests that fail after admission consume a quota slot; this includes extraction 404 responses. A legacy upstream request may already have been paid for. Shared API service_busy refusals and invalid requests consume no quota. SERP route-busy and cooling responses occur after admission and consume one slot. Fallback attempts are part of one admitted search; at most four providers can be attempted, including at most three external providers. Without VITNA_SEARCH_QUOTA, quotas live in process memory and reset on restart. With it, per-client minute/day counters persist in SQLite and accounting failures refuse searches. The web deployment template sets a quota file. Run one alpha instance; these limits are not a distributed customer billing ledger.
All responses use Cache-Control: no-store. Errors carry an error object with code and message. Completed attempt sequences also carry metadata.
| Status | Meaning |
|---|---|
| 400 | Invalid JSON or unsupported request shape, including removed Live/Hybrid modes (invalid_request) |
| 401 | Missing or invalid search credential |
| 404 | Unknown service path, or an extraction URL unavailable in the current index |
| 405 | Wrong HTTP method; Allow names the supported method |
| 413 | Request body exceeds the byte limit |
| 415 | Content type is not JSON |
| 429 | Minute limit, day limit, or service concurrency limit; includes Retry-After |
| 499 | Caller cancelled the request |
| 502 | No attempted provider completed the search |
| 503 | A provider, SERP, extraction, or research is unavailable, the required local index is empty, or quota accounting fails; also every SERP retrieval or parsing failure, a missed deadline included, told apart by its code |
Upstream error bodies are never forwarded. Legacy search-provider redirects are refused, and their response bodies are capped at 1 MiB. SERP uses its own bounded, validated search-engine retrieval. Disconnecting a caller aborts the current request and prevents another fallback attempt.
Compare retrieval outputs
Compare speed and result counts within the same task and freshness policy. These endpoints return different evidence:
| Route | What it returns | What a successful response establishes |
|---|---|---|
| Google SERP | Observed Google organic listings, engine snippets, page ranks, and resolved publisher links | A captured result page or bounded collection; snippets are not quotations fetched from the publisher pages |
| Indexed search | Keyword-ranked matches from Vitna's stored corpus, or hybrid-ranked where semantic retrieval is configured, with crawl and source metadata | Matches in the configured index; neither a lexical match nor vector similarity is a finding supported by source evidence |
| Native source brief | Verbatim passages from indexed pages with source references and Unicode citation offsets | Extractive indexed evidence with explicit coverage; it does not generate an answer or establish exhaustive research |
An owned-route provider fee of zero excludes cloud hosting, browser CPU, network transfer, storage, crawling, and operating costs. Optional external search charges are separate. Compare result relevance, evidence quality, coverage, and freshness alongside latency; a larger link count is not the same deliverable as a smaller cited source brief.
Provider protocol references
The adapters follow the providers' own interfaces, checked on 2026-10-03:
- Brave Web Search, ranked web results and subscription-token authentication.
- Exa Search, highlights, request identifiers, and estimated cost metadata.
- Tavily Search, bearer authentication, basic depth, and reported credit usage.
Before a commercial launch
The private beta serves configured clients and upstream accounts. Public signup, self-service key issuance and revocation, an invoice-grade usage ledger, customer billing, and public self-service deployment need a further implementation. The included search interface works on a configured instance. Search is a separate service from the existing routing subscriptions; no existing plan gains or is charged for search in this change. Check provider contracts before exposing their results commercially. Brave's standard terms prohibit resale/redistribution and replacing its service; ordinary API access does not authorize a public resale search service. Google and Bing results need the same decision: SERP retrieval starts only after the operator confirms, with VITNA_SERP_RESALE_APPROVED=true, that collecting and redistributing them is permitted.
Operator reference
The sections from here to the end are for the people who run a Search instance, which during the beta means Vitna's own operators, working from a checkout of Vitna's private repository. None of it is needed to call the hosted API at https://search.vitna.ai, and none of it is an offer to run a deployment of your own.
Try the complete offline demo with node platform/search/demo.mjs, then open http://127.0.0.1:8789. Its four pages are clearly labeled illustrative fixtures; the demo makes no external requests. See Run a focused search index for live crawling, OVHcloud preparation, and R2 snapshots. The general web beta adds persistent cross-site discovery, day/month crawl budgets, file-backed API quotas, and the Virginia deployment package.
Since 2026-10-07 a browser that opens the hosted instance, or its /docs, is sent to Search's page and this reference on vitna.ai, where the product's pages live: the operator sets VITNA_SEARCH_PAGE_URL and VITNA_SEARCH_DOCS_URL, and a self-hosted instance with neither set serves its own page and API reference.
Starting an alpha instance
Node.js 22.13 or later is required; Node 24 is recommended for the deployment package. Direct SERP retrieval and indexed search use built-in Node modules and need no npm dependencies or website build. The optional browser SERP transport, including browser attempts selected by auto, requires the separate Unblocker runtime. Indexed search uses the currently experimental node:sqlite. Startup opens .search-data/web/index.sqlite by default and initializes an empty local index when needed. Populate selected sources before serving useful native searches. SERP retrieval does not require populated indexed pages or crawler work.
Mint a credential for each alpha client:
npm run search:key -- pilotThe command prints a vitna_search_ token once, followed by a JSON entry mapping the client identifier to its SHA-256 hash. Give the token to the client privately. Merge the hash entry into VITNA_SEARCH_CLIENTS, a JSON object. The service accepts these search keys and an org's platform key (vitna_key_, see one key); push and router keys grant no search access. Remove a client's entry and restart to revoke its access.
For native retrieval, set VITNA_SEARCH_CLIENTS. The default provider is vitna, with an index at .search-data/web/index.sqlite; override its path with VITNA_SEARCH_INDEX. Crawl selected sources before searching their content. See the index setup. An operator can still select legacy external adapters explicitly, but native startup has no external provider fallback.
For actual Google/Bing results, set VITNA_SEARCH_SERP_ENABLED=true and VITNA_SERP_RESALE_APPROVED=true, and follow the SERP setup. The second setting records the operator's decision that collecting those results and redistributing them through Vitna's API is permitted; startup refuses the first without it. The same client key authenticates both capabilities; neither requires a Bright Data key or zone.
Start the API with:
npm run search:startFor a local environment file, keep a private .env.search in the checkout, using the settings described in deploy/vitna-search.env.example. Start it with Node's environment-file loader:
node --env-file=.env.search platform/search/start.mjsBy default the server binds to 127.0.0.1:8789. Boot refuses an empty client registry, invalid limits, duplicate credential hashes, no enabled retrieval capability, or SERP retrieval enabled without its redistribution confirmation. The local index is initialized if needed; an empty index can start but native retrieval, and with it readiness, remains not-ready until a crawl succeeds, even with SERP retrieval enabled. A SERP-only instance can be ready without an index. Starting the process makes no upstream search request and does not run a crawl.
Maintaining the vectors file
Where semantic retrieval is configured, maintain the vectors file with:
npm run search:embedThe job reads the index read-only, embeds new pages and pages whose embedded text changed, deletes vectors of pages that left the index, stops at --limit pages (default 25,000) or --budget-ms (default 480,000), and prints one JSON summary line. A vectors file holding another width or model stops the run with vectors_dimensions_mismatch or vectors_model_mismatch; any other error in the vectors file or the index exits 1 with the error on stderr and no summary. Every batch commits on its own, so an interrupted run can simply run again; a lease in the vectors file keeps two runs from overlapping. A text the engine refuses is retried with its character limit halved, and halved again while the engine keeps refusing (750, 375, then 187 characters for the default 1,500), stopping before the limit falls under 100 or once halving no longer shortens the text. It is then recorded and skipped until the page changes. The general web deployment packages it as a service and timer.
The job compares pages and vectors in document-id ranges of 100,000, so its memory stays bounded however large the index grows. Pages of a bulk import (an imported Wikipedia dump) are embedded only with VITNA_SEARCH_EMBED_BULK=true, and then after every crawled page that needs a vector; without it they are passed over and any vectors they already have are kept. The summary's documents counts the pages the job considered.
Once the vectors no longer fit VITNA_SEARCH_VECTORS_MAX_MB, build the approximate index from the vectors file, with the same embedding settings as the API:
npm run search:annIt reads the vectors file only, writes vectors.ivf beside it under a temporary name and renames it into place, and prints one JSON line with the vector count, lists, file size and duration. Query workers load the new index on their own. Vectors stored after a build are searched exactly beside the index, so rebuild when they no longer fit beside it, or when a large share of the file has changed.
