vitna.ai/docs/search/overview Machine view
Raw .md llms.txt
> Source: https://vitna.ai/docs/search/overview
> Raw Markdown: https://vitna.ai/docs/search/overview.md
> Corpus index: https://vitna.ai/llms.txt

# Vitna Search

Private beta. A separate search API for agents with independent
Google/Bing SERP retrieval and Vitna's own crawler and web index. SERP requests
return actual search-engine organic listings; indexed search and cited
research use successfully indexed pages. It includes a browser interface,
dedicated client keys, and shared request limits. The private-key beta is
hosted at `https://search.vitna.ai`, with an API reference at `/docs`. Its
product page and a playground are at `/search` on vitna.ai. Keys are issued
by invitation, requested at [vitna.ai/start](https://vitna.ai/start#search).
Customer signup and search billing are not enabled.

## The service

Use `POST /v1/serp` to retrieve actual Google or Bing text search result
pages through Vitna's transport and parser, without a Bright Data call.
The compatibility alias `POST /request` accepts the supported URL-based
request shape. This alpha returns parsed organic listings or fetched HTML;
it does not promise full vendor coverage for ads, shopping, maps, other SERP
features, geographic egress, or challenges. See the
[SERP API contract](serp.md) for requests, operator setup, and limitations.

The native contract returns ranked URLs, titles, excerpts, source identity,
and observed crawl timestamps, and on request each page's stored text and
verbatim highlights ([inline contents](#inline-contents)). It uses keyword retrieval with title-weighted
BM25 ranking by default, removes duplicate URLs and fragments, and identifies
result origins in `provenance`. Among its first 20 candidates, a result list
leads with at most two pages per host (`www.` aside), and that host's further
candidates follow the other sites'. Nothing is dropped, so a search confined
to one site still fills its page. An instance configured with Vitna's own
embedding engine adds [semantic and hybrid retrieval](#semantic-retrieval)
and [similar pages](#similar-pages). A result carries a publication date only
where the page itself declares one ([declared publication dates](#declared-publication-dates));
it never infers one, and it does not generate an answer. Query-guided
expansion can discover and crawl a bounded set of pages, then make those
pages searchable in Vitna's own index.

Exact single-brand queries prefer a matching indexed homepage under `.com`,
`.ai`, or `.org`, including its `www` form. The homepage must still match the
query and satisfy all filters; other results keep their BM25 ordering.

Running an instance is operator work, from a checkout of Vitna's private
repository, and is described at the end of this page under
[Operator reference](#operator-reference). Nothing there is needed to call
the hosted API.

The process is separate from the site and the market collector. It has no
access to the site's database and holds no signing key. Its optional SQLite
search file belongs to this service alone. Provider search credentials belong
only in this process's environment, not in the site's environment. Runtime
search results and usage never become Witness price observations.

## Google/Bing SERP

```text
POST /v1/serp
POST /request
```

Send the search bearer key and JSON content type. For an actual Google
result page:

```json
{"engine":"google","query":"lunar science","country":"us","language":"en","maxResults":5,"format":"json"}
```

The response contains `general`, `input`, `organic`, and `meta`, with
`meta.provenance: "search_engine"` and `meta.coverage: "organic_only"`.
The organic entries expose `rank`, `title`, and `link`, with optional
`description`, `display_link`, and observed `sitelinks`. Use `engine: "bing"` for Bing or
`format: "html"` for the fetched HTML. SERP retrieval does not require an
index, start a native crawl, or produce cited research. Read the complete
[SERP contract](serp.md) before migrating a URL-based client.
For up to 100 distinct Google links across observed result pages, use the
asynchronous [multi-page jobs](serp.md#multi-page-google-jobs) contract.
Its target can finish partial; the single-page endpoint remains capped at ten.
Collection requests default to `fresh: false`: exact same-client requests
can join an active job or reuse a target-complete result for thirty seconds.
Use `fresh: true` to bypass completed reuse. Capture timestamps remain
unchanged, and cancelling a shared job affects its joined callers.

Enabled SERP retrieval defaults to `auto`: ordinary routes prefer a browser
for Google and direct HTTPS for Bing, or pinned proxy browser routes when proxies are
configured. A single deadline bounds distinct attempts, route concurrency,
and cooldowns. Successful JSON results have a short process-local cache;
send `fresh: true` for a new capture. Read the complete
[retrieval policy](serp.md#retrieval-routes-and-retries) for operator settings
and safe route/cache metadata. These controls do not establish production
access to a search engine.

## Search

```text
POST /v1/search
```

Send `Content-Type: application/json` and a search bearer key. For example,
with the issued credential in the caller's `VITNA_SEARCH_KEY` environment:

```bash
curl https://search.vitna.ai/v1/search \
  -H "Authorization: Bearer $VITNA_SEARCH_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"query":"agent search APIs","mode":"index","maxResults":5}'
```

| Field | Default | Allowed values |
|---|---|---|
| `query` | Required | 1 to 1,000 characters, no control characters |
| `maxResults` | 5 | Integer from 1 to 100 |
| `mode` | Explicit `index` recommended; browser default | `index`; mutually exclusive with `provider` and `fallback`; removed `live` and `hybrid` values return HTTP 400 |
| `provider` | Legacy `auto` when `mode` is omitted | `auto`, `vitna`, `brave`, `exa`, `tavily`; cannot accompany `mode` |
| `fallback` | Legacy `false` | Boolean; `true` only with legacy `auto`; cannot accompany `mode` |
| `domains` / `excludeDomains` | Unset | 1–10 public hostnames; includes subdomains; `index` mode or legacy Vitna without external fallback |
| `maxAgeDays` | Unset | Integer 1–3,650; observed crawl age; `index` mode or legacy Vitna without external fallback |
| `startPublishedDate` / `endPublishedDate` | Unset | A date (`YYYY-MM-DD`, a whole UTC day) or an ISO 8601 date-time with a UTC offset, both inclusive; only pages that [declare a publication date](#declared-publication-dates) can match; `index` mode or legacy Vitna without external fallback |
| `asOf` | Unset | Point-in-time snapshot cutoff (`YYYY-MM-DD`, end of UTC day, or ISO 8601 date-time with a UTC offset); searches indexed snapshots captured on or before that instant with zero lookahead, disables coverage expansion and paid fallback (`as_of_requires_vitna`), and echoes `meta.asOf` |
| `index` | `web` | `web` or one of the 23 vertical indexes reported by `GET /v1/indexes` |
| `effort` | Unset | `fast`, `balanced`, or `deep`; `fast` prefers keyword retrieval without reranking, `balanced` uses configured hybrid retrieval without default excerpts, and `deep` attaches `contents.excerpts` and applies native reranking when available; echoed in `meta.effort` |
| `retrieval` | `hybrid` where [semantic retrieval](#semantic-retrieval) is configured, otherwise `keyword` | `keyword`, `semantic`, or `hybrid`; Vitna index without external fallback; `semantic` cannot accompany `additionalQueries` |
| `contents` | Unset | `text`, `highlights`, and/or `excerpts` read from stored pages, optional `maxTotalCharacters` / `maxTotalTokens` budget, and `injection` guard; see [inline contents](#inline-contents); Vitna index without external fallback |
| `expand` | Unset; permits automatic name/topic discovery | Boolean; `true` requests bounded expansion; `false` opts out of automatic discovery |
| `discover` | Unset; permits automatic name/topic discovery | Boolean; explicit `true` requires `expand: true`; `false` disables site discovery |
| `urls` | Unset | 1–8 public HTTPS page URLs without query strings; requires `expand: true` and must satisfy domain filters |

Unknown fields are refused. Request bodies are limited to 8,192 bytes.
`maxResults` above 10 needs `mode: "index"` or legacy `provider: "vitna"`; a
request that may reach an external provider keeps 1 to 10.
The caller cannot set upstream provider endpoints, headers, or credentials.
Domain, crawl-age and publication-date filters, and `retrieval`, require the
Vitna index with external fallback disabled; otherwise the filters return HTTP
400 `filters_require_vitna` and `retrieval` returns HTTP 400
`retrieval_requires_vitna`.
With `discover: true`, `domains` is limited to three hostnames and supplied
URLs to three distinct origins; ordinary Index domain filters still allow ten.

Use `mode: "index"` for Vitna's stored corpus. It makes no paid search-engine
call, including when it returns no matches. `maxResults` limits the immediate
search response; it does not increase crawl work. A whole-request search
deadline is separate from the background job's bounded crawler work.

An index search can return up to 100 results. A longer list extends a shorter
one: its first results are those a 10-result search returns, with the same
host spread among the first 20, and later results follow in relevance order.
Every list keeps its duplicate suppression and filters. Caller-supplied
`additionalQueries` are read as deep as the list asks, ten pages each or
`maxResults` when larger, and pages found only below each query's top ten
follow the rest; `meta.index.retrieval.perQueryLimit` reports the depth.

On a synthetic 50,000-page index with 384-dimension vectors, measured in one
process on a Windows arm64 laptop and without the query embedding, 100
results took 30 ms median and 41 ms at the 95th
percentile by keyword, against 11 ms and 13 ms for 10, and 61 ms and 77 ms
hybrid, against 35 ms and 38 ms: far inside the default 10-second deadline.
A repeated search is answered from the query workers' 30-second cache, which
weighs a cached answer by its results, ten to a unit, so a 100-result answer
takes the room of ten 10-result ones.

An empty native name/topic search, or a topic search with only partial-term
matches, can automatically enqueue discovery when enabled, with no `domains`
or supplied `urls` and no
external fallback. For example, this illustrative name
query needs no crawl flags:

```json
{"query":"examplebrand","mode":"index"}
```

When configured, automatic discovery first uses its own directory of observed
names. Explicit expansion can also use matching known URLs. A narrow distinctive single ASCII brand can also
produce up to three candidate hosts using `.com`, `.ai`, and `.org`. Those
hosts are suggestions, not verified official identities. Name detection is
heuristic; common topic words and multiword topic queries produce no guessed
hostnames. Broad queries can match a publisher catalog's descriptions or, when
configured, a local English Wikipedia title catalog to select candidate pages.
Catalog entries are discovery starting points,
not indexed coverage. A query without a suitable name, topic, or concept match can return
`no_candidates`. This is not whole-web search. Current results return first with any job in `meta.expansion`;
automatic admissions include `meta.expansion.automatic: true`.

Send `expand: false` or `discover: false` to prevent automatic discovery.
For explicit target-only refresh, send `expand: true, discover: false`.
For explicit site discovery, send `expand: true, discover: true`; domains
remain optional guidance, not a prerequisite.

With `expand: true`, current indexed results return with `meta.expansion`:
a durable job with a `statusUrl`, or an explicit `no_candidates`, `queue_full`,
`client_job_limit`, `discovery_unavailable`, or `unavailable` status. Admission failure does not discard
valid indexed results. With `discover: false`, the job crawls known relevant URLs
and supplied targets with scope `known_urls_and_supplied_targets`.

Add `discover: true` to examine selected sites' homepages and sitemaps before
choosing target pages. Optional `domains` guides site selection; without it,
explicit discovery can use supplied URL origins, matching known URLs,
observed names, narrow name candidates, topic-catalog matches, or configured
concept-title matches. Automatic
discovery skips incidental known-URL matches and uses the name or topic planners.
The job has scope `site_discovery`, with at most three origins, 2,000 sitemap
candidates, eight target pages, 40 total HTTP requests, and 60 seconds of
crawler work. Candidate names do not establish official identity or promise
whole-web coverage.
An initialized empty index may accept expansion. See
[coverage expansion](general-web.md#query-guided-coverage-expansion).

Bounded lexical and site hints, including names, candidate origins and their
provenance, are encrypted in private jobs that expire after 24 hours. The raw
original query and full request are not stored or logged. Deduplication uses a private HMAC
digest. Plaintext queries and terms are absent from jobs and receipts;
metadata receipts still record request outcomes and usage. The operator must configure a private
`VITNA_SEARCH_DEMAND_SECRET_FILE` before accepting site-discovery jobs.

Authenticate `GET /v1/search/jobs/<id>` with the same client's key to receive
`{"job": {...}}`. Status is `queued`, `running`, `completed`, or `failed`.
Completed means the bounded pass finished; inspect its summary for indexed,
blocked, failed, and unmet targets, then repeat the original search to retrieve
fresh results. A manual rerun can send `expand: false, discover: false` to
retrieve only the updated corpus. `summary.discovery` reports integer counters for `origins`,
`candidates`, `selected`, and `unavailable`. Selected candidates are not indexed
pages; inspect `summary.indexed`, `summary.unchanged`, and unmet targets for
actual coverage. Job checks have a separate 60-per-minute limit and don't consume
the search quota. Jobs expire after 24 hours. Unknown, expired, and other
clients' jobs return the same 404 response.

## Semantic retrieval

Semantic and hybrid retrieval are optional. They embed text with Vitna's own
embedding engine: vitna-anchor's OpenAI-compatible `POST /v1/embeddings`,
which runs on the search host as `vitna-embed.service` at
`127.0.0.1:8795` for query embeddings and `vitna-embed-batch.service` at
`127.0.0.1:8797` for vector maintenance, serving `bge-small-en-v1.5` (384
dimensions). No hosted
embedding, rerank, or search API is called. Both stay off unless
`VITNA_SEARCH_EMBED_URL` and `VITNA_SEARCH_EMBED_MODEL` are set, and an
unconfigured instance answers exactly as before.

| `retrieval` | What runs |
|---|---|
| `keyword` | The BM25 retrieval described above |
| `semantic` | The pages whose stored vectors are nearest the query's embedding, by cosine similarity |
| `hybrid` | Reciprocal rank fusion, with constant 60, of the top 20 keyword candidates and the top 20 nearest pages |

When the two hybrid lists swap a pair, their fused scores tie exactly and the
embedding order decides. A request for more than 20 results reads both lists
that deep; pages found only below both lists' top 20 follow the fused top
20, so the leading results stay those of a shorter request. A configured instance defaults to `hybrid`. Every mode applies the same
enabled-source, domain and crawl-age filters, suppresses duplicate extracted
content, honours `maxResults`, and labels results `kind: "index"` in
`provenance`. The domain and crawl-age filters restrict the nearest-page scan
itself, and the scan widens past copies of a nearer page, so neither empties a
list that other admitted pages can fill. A matching brand homepage still leads only when it also matched
the query's words. A page found only by its vector shows a query-focused
excerpt when a query term occurs in it, otherwise its leading words.

`meta.index.retrieval` reports what ran. Illustrative values:

```json
{"requested":"hybrid","used":"hybrid","model":"bge-small-en-v1.5","vectors":49012,"candidates":{"keyword":20,"semantic":20}}
```

It is present whenever semantic retrieval is configured or a request names a
mode. `vectors` counts pages with a stored vector, which can trail
`meta.index.documents` until the maintenance job catches up.
`meta.index.match` keeps its lexical meaning: it describes the keyword half of
a hybrid search, for the first query as in keyword retrieval, and is absent
from a `semantic` one. Automatic discovery follows the keyword half, so a
`hybrid` search queues it exactly when a keyword search would have: when that
half found nothing (`candidates.keyword` is 0) or matched only some terms
(`partial`), even though nearest pages fill its results. Pages that
`additionalQueries` found count as found, in either mode. An opted-in
`semantic` search queues discovery only when empty.

The search never fails because of semantic retrieval. When the embedding call
fails or exceeds `VITNA_SEARCH_EMBED_TIMEOUT_MS`, or no compatible vectors are
loaded, the response carries keyword results with `used: "keyword"` and a
`fallback` code: `semantic_not_configured`, `embed_unavailable`,
`embed_timeout`, `embed_paused`, `embed_rejected`, `embed_invalid_response`,
`embed_model_mismatch`, `vectors_unavailable`, `vectors_incompatible`,
`vectors_over_limit`, or `snapshot_as_of` (when `asOf` runs lexical retrieval
over historical snapshots). A fallback result is not cached. Receipts record the
retrieval used and any fallback code, never the query.

After a query embedding times out or the engine cannot be reached, query
embeddings pause for 30 seconds and searches report `embed_paused`, so an
outage costs one timeout per pause rather than one per search. The first
search after the pause tries the engine again while the rest stay paused, and
a success ends the pause. Maintenance batches never pause. Because the
embedding wait counts against the request deadline, startup refuses a
`VITNA_SEARCH_EMBED_TIMEOUT_MS` above half of `VITNA_SEARCH_TIMEOUT_MS`,
which leaves the keyword fallback at least half the deadline.

Page vectors live in their own SQLite file, `vectors.sqlite` beside the index
unless `VITNA_SEARCH_VECTORS` names another, and never in `index.sqlite`. The
file carries its own SQLite application id. The maintenance job refuses a
vectors path that names the index, frontier, concept catalog or quota file,
and the store refuses any file that belongs to another database; a vectors
file from before the id existed is adopted, header only, on the job's next
run. A
file holds one model: its identifier and width are recorded once, and the API
and the maintenance job refuse any other, so changing models means a new file.
Each row keeps the fingerprint of the text embedded: by default the page title and
its leading text, up to `VITNA_SEARCH_EMBED_MAX_CHARACTERS` (1,500) under a
conservative token estimate, because single-window `bge-small-en-v1.5` refuses
texts over its 512-token window with HTTP 400 `context_length_exceeded` rather
than truncating them. Operators can enable bounded overlapping passage chunking
(`VITNA_SEARCH_EMBED_MAX_CHUNKS`, `VITNA_SEARCH_EMBED_CHUNK_OVERLAP`,
`VITNA_SEARCH_EMBED_MAX_DOCUMENT_CHARACTERS`) and configure multilingual or
long-context models (`VITNA_SEARCH_EMBED_MAX_TOKENS`,
`VITNA_SEARCH_EMBED_MULTILINGUAL`); chunked matches deduplicate to the canonical
document and pass matched passage byte/code-point offsets into
`contents.highlights`. A maintenance job keeps the file current; operators run it as
described under [Maintaining the vectors file](#maintaining-the-vectors-file).

The query embedding is made on the API's main thread, so a slow engine never
occupies a query worker, and its failure becomes keyword retrieval before any
worker is involved. The nearest-neighbour scan runs in each query worker beside
BM25. Each worker holds its own copy of the vectors: vectors × (width × 4 + 8)
bytes, room for about 5% growth, and their URLs. For 49,063 pages at 384
dimensions that measured 79.5 MB per worker, with a 434 ms full load.
`VITNA_SEARCH_VECTORS_MAX_MB` (512 per worker) bounds each copy. Workers apply
new vectors and removals incrementally when the file changes: 500 changed
vectors took 5.3 ms.

A file larger than that bound, such as one holding vectors for millions of
imported Wikipedia articles, is searched through an approximate
nearest-neighbour index instead, which the operator
[builds](#maintaining-the-vectors-file) beside it (`vectors.ivf` beside
`vectors.sqlite`). It is an inverted-file index: the
vectors are split into up to 4,096 lists by two levels of spherical k-means,
and each vector keeps one bit per dimension of its difference from its list's
centre plus that difference's mean size, 52 bytes at 384 dimensions instead
of 1,544. A search scores every list centre, estimates each vector in the
`VITNA_SEARCH_VECTORS_PROBES` (64) nearest lists with one table read per eight
dimensions, and rescores the best `VITNA_SEARCH_VECTORS_RESCORE` (400)
estimates exactly from the floats stored in `vectors.sqlite`. Vectors stored
or replaced after the index was built are held and searched exactly beside
it, and a removed vector drops out because rescoring finds no row, so the
index stays usable between rebuilds; the index and those newer vectors
together must fit `VITNA_SEARCH_VECTORS_MAX_MB`. The index is plain
JavaScript over typed arrays, like the exact scan, so the search host needs no
native build. A file within the bound is always searched exactly. A file over
the bound without an index built from it (a missing index, one from another
vectors file or model, or more newer vectors than fit) answers with
`vectors_over_limit`, and a damaged index with `vectors_unavailable`; either
way the search falls back to keyword results as below. A query worker reads
the index in the background, since at millions of vectors that takes longer
than a query's deadline: after it starts, its semantic and hybrid searches
fall back with `vectors_unavailable` until the index is in, and after a
rebuild the index in use keeps answering until the new one replaces it.

Limitations:

- The default single-vector `bge-small-en-v1.5` configuration embeds only English text well; multilingual retrieval requires deploying a multilingual profile (such as `bge-m3` or `multilingual-e5-small`/`multilingual-e5-base` with `VITNA_SEARCH_EMBED_MULTILINGUAL=true`).
- Unless multi-passage chunking (`VITNA_SEARCH_EMBED_MAX_CHUNKS > 1`) is enabled, only a page's title and leading text (within `VITNA_SEARCH_EMBED_MAX_CHARACTERS` and `VITNA_SEARCH_EMBED_MAX_TOKENS`) are embedded, and text further down a page is found by keyword retrieval alone.
- Nearest neighbours always exist, so `semantic` returns the closest pages even when none is relevant; there is no relevance threshold. On the authored negative queries, semantic and hybrid retrieval answered none with an empty result, where keyword retrieval answered 8% to 33% of them correctly.
- Within the bound the scan is brute force and grows linearly with the number of vectors and their width. It measured 24.7 ms median for 49,063 vectors of 384 dimensions on a desktop Ryzen 7 3700X; the default 512 MB bound holds about 347,000. Past that, the approximate index answers instead, and can miss some of the exact nearest pages; see the [measurement](general-web.md#wikipedia-at-full-scale).
- A search embeds its query first: about 10 ms on an idle engine, longer while a maintenance batch is in progress, because the engine serves one request at a time.
- New and changed pages are found by keyword alone, or by their previous vector, until the next maintenance run.
- Source brief research and indexed text extraction remain keyword based.

The [general web beta](general-web.md#measure-coverage-before-expanding-access)
records the measured comparison of the three modes.

## Legacy provider routing

Legacy gateway instances can omit `VITNA_SEARCH_INDEX`, explicitly select
`VITNA_SEARCH_PROVIDERS=brave,exa,tavily`, and configure
`BRAVE_SEARCH_API_KEY`, `EXA_API_KEY`, or `TAVILY_API_KEY`. Those accounts and
contracts are separate from Vitna's native crawler; the hosted beta uses the
native index. The Bright Data search adapter and its Live/Hybrid modes have
been removed. Use the independent [SERP API](serp.md) for actual Google/Bing
listings, or native search for Vitna's indexed corpus. Existing clients
should follow the [migration guide](live-engine.md).

Clients that omit `mode` retain the provider/fallback contract. `auto` uses
the first configured provider in `VITNA_SEARCH_PROVIDERS` order,
which defaults to `vitna` when an index path is configured, or
`brave,exa,tavily` otherwise. Providers with no index/credential are skipped.
This is a configured preference, not a claim that the first provider is the
cheapest or best. A named provider stays pinned.

With `fallback: true`, a timeout, network failure, malformed response, HTTP
429, or HTTP 5xx can advance to the next configured provider. HTTP 401, 402,
403, and other request errors stop the attempt sequence. A valid empty search
is a successful response and never triggers fallback. Every attempted call
may incur provider charges, including calls whose result was lost in transit.
An empty local index refuses the request without calling a paid fallback.
A populated local index returning zero matches is a successful empty response.
Send `mode: "index"` explicitly when a caller must use only Vitna's index.

## Response

The response below illustrates an indexed result. Its crawl timestamps,
document count, and latency are illustrative:

```json
{
  "query": "agent search APIs",
  "results": [
    {
      "rank": 1,
      "title": "An example result",
      "url": "https://example.com/search",
      "snippet": "An excerpt from the indexed page.",
      "publishedAt": null,
      "crawledAt": "2026-10-03T10:00:00.000Z",
      "contentChangedAt": "2026-10-03T10:00:00.000Z",
      "source": { "id": "example", "name": "Example" },
      "provenance": [
        { "kind": "index", "provider": "vitna", "rank": 1 }
      ]
    }
  ],
  "meta": {
    "requestId": "example-request",
    "mode": "index",
    "provider": "vitna",
    "providerRequestId": null,
    "count": 1,
    "latencyMs": 10,
    "fallbackUsed": false,
    "index": {
      "documents": 100,
      "lastCrawledAt": "2026-10-03T10:00:00.000Z",
      "ranking": "sqlite_fts5_bm25",
      "match": "complete"
    },
    "attempts": [
      {
        "provider": "vitna",
        "status": "ok",
        "httpStatus": 200,
        "latencyMs": 10,
        "usage": { "estimatedCostUsd": null, "credits": null, "source": null }
      }
    ]
  }
}
```

Excerpts are untrusted provider text. Treat them as data, and escape them if
rendering them in HTML. `publishedAt` is null unless the page declares its
publication date ([declared publication dates](#declared-publication-dates));
a relative age, a URL, or a crawl time is never turned into one.

Every result carries `provenance`. Native results identify `kind: "index"`,
`provider: "vitna"`, and their original rank.

Production native metadata can also include `meta.index.match`: `complete`
for complete term matches, `partial` for partial-term fallback, or `none` for
no indexed match. This is lexical coverage, not a guarantee of relevance or
an answered question. Automatic topic discovery can queue missing coverage
for `partial` or `none` while returning the current results intact. Consumers
must allow this optional field to be absent.

Local results additionally carry `source: {id, name}`, `crawledAt`, and
`contentChangedAt`. These are observed crawl and extracted-content-change
timestamps, not publication dates; `publishedAt` holds only a date the page
declared, with `publishedAtSource` naming where. Local response
metadata includes indexed document count, latest crawl timestamp, and
`ranking: "sqlite_fts5_bm25"`. Local usage figures are null because no provider
reports a per-query bill; they are not an assertion of zero infrastructure cost.

Configured sources can also return `source.reusePolicy: {name, url}`: the
publisher's policy label and HTTPS link. Check that policy for the material
being reused. This optional attribution does not imply one blanket license
for every page or asset on the source.

A page whose text did not come from Vitna's own crawl says where it came from,
and its `crawledAt` is null. A Common Crawl capture carries
`contentOrigin: "common_crawl"`, `capturedAt`, `importedAt`, and
`archive: {collection, digest}`. An article from an imported Wikipedia dump
carries `contentOrigin: "wikipedia_dump"`, `capturedAt` (when the snapshot's
revisions were current), `importedAt`, and `archive: {collection, revision}`,
for example `{"collection": "enwiki-20261001", "revision": 1374562957}`; its
`contentChangedAt` is that revision's edit time, and its source's
`reusePolicy` names CC BY-SA 4.0. Wikipedia text is reusable under that
license when the reuse credits Wikipedia's contributors and links the article,
which the result's `url` is. Crawl-age filters read `capturedAt` for these
pages. See [Import English Wikipedia](general-web.md#import-english-wikipedia).

## Declared publication dates

A native result's `publishedAt` is the publication date its publisher
declares in the page, read when the page is crawled or imported from an
archive. Three declarations are read, and the first that gives one value
wins:

| `publishedAtSource` | Where the page declares it |
|---|---|
| `article_published_time` | A `<meta property="article:published_time">` (or `name=`) tag |
| `json_ld_date_published` | `datePublished` on a JSON-LD article or posting, or failing that on the page itself (`WebPage` and its kinds), at the top level, in `@graph`, or as their `mainEntity`; a site's or a listed item's date is not read |
| `article_time_element` | A `<time datetime>` inside an `<article>`: the one marked `itemprop="datePublished"` or `pubdate`, or else the only value the article's time elements give |

When a source gives two different values, such as several articles' dates on
one listing page, it declares nothing and the next source is read. A value
with a time and a UTC offset is returned as that instant in UTC,
`"2024-05-01T08:30:00.000Z"`. A bare date, or a time without an offset,
stays the calendar day the page named, `"2024-05-01"`, because the zone it
was written in is unknown. A date more than a day after the crawl, a date
before the year 1000, and the Unix epoch's first day, which content systems
print for an empty field, are not stored. Pages without a declaration have
`publishedAt: null` and no `publishedAtSource`, exactly as before. Nothing is
inferred: not from a relative age such as "3 days ago", a URL path, visible
text, a modification date, an HTTP header, or the crawl time. A page crawled
before the index stored dates has none until it is crawled again, and an
article imported from a Wikipedia dump has none: the dump declares revision
times, not publication dates.

`startPublishedDate` and `endPublishedDate` filter on this declaration and
apply only to pages that have one: a page without a declared date never
matches a date filter, so a filtered search covers less of the index than an
unfiltered one. Both bounds are inclusive. A date bound means the whole UTC
day; a date-time bound needs a UTC offset and is compared as the instant it
names. A page that declares only a date matches when that date falls within
the bounds' dates. Semantic and hybrid retrieval restrict their nearest-page
scan to dated pages in range; through the [approximate index](#semantic-retrieval)
the range is applied as candidates are read back, as the crawl-age window is,
so a narrow range there can return fewer pages than it admits.

```json
{"query":"lunar missions","mode":"index","startPublishedDate":"2024-01-01","endPublishedDate":"2024-12-31"}
```

The declaration is the publisher's claim, not a verified fact. A publisher
can backdate a page, or declare an original print date for a page posted
later.

## Inline contents

A native search can return each result's stored page text and verbatim
highlights with the results, so a caller needs no extraction request per URL.
Send `contents`:

```json
{"query":"Galilean moons","mode":"index","maxResults":5,"contents":{"text":{"maxCharacters":4000},"highlights":true}}
```

| Field | Default | Allowed values |
|---|---|---|
| `contents.text` | Off | `true` for up to 20,000 characters per result, or `{"maxCharacters": n}` / `{"maxTokens": t}` (1 to 100,000 characters, 4 characters per token, mutually exclusive) |
| `contents.highlights` | Off | `true` for a 1,000-character passage budget per result, or `{"maxCharacters": n, "maxTokens": t, "query": "..."}` (1 to 4,000 characters, optional 1-1,000 character query) |
| `contents.excerpts` | Off (auto on `effort: "deep"` for native index searches with extraction) | `true` for a 1,000-character contiguous excerpt window per result, or `{"maxCharacters": n, "maxTokens": t, "query": "..."}` (1 to 4,000 characters) |
| `contents.maxTotalCharacters` / `contents.maxTotalTokens` | Unset | Positive integer response-wide character budget from 100 to 400,000 (`maxTotalTokens` from 25 to 100,000 at 4 characters per token, mutually exclusive); distributes budget across results and redistributes unused slack fairly |
| `contents.injection` | Off | `"scan"` or `"strip"`; scans extracted `text`, `highlights`, and `excerpts` for prompt-injection phrases and invisible Unicode controls, attaching `contents.injection` (and stripping invisible controls when `"strip"`) |
| `contents.snapshotAsOf` | Unset | Point-in-time snapshot cutoff (`YYYY-MM-DD` or ISO 8601 date-time with UTC offset) for historical inline extraction; when paired with `asOf`, both values must match |

`false` leaves a part out, and a `contents` object that asks for no content part (`text`, `highlights`, or `excerpts`)
returns an error. Other fields are refused. Contents require the
Vitna index with external fallback disabled, otherwise HTTP 400
`contents_requires_vitna`, and an instance with page extraction, otherwise
HTTP 503 `extraction_unavailable`. Both are checked before the request is
admitted. When `contents.maxTotalCharacters` or `contents.maxTotalTokens` is
requested, `meta.contextBudget` reports `{ requestedCharacters, returnedCharacters, estimatedTokens, truncated }`.
When `contents.injection` is `"scan"` or `"strip"`, each extracted item's `contents.injection` reports `{ risk, findings }` (plus `stripped`, the count of removed invisible Unicode characters when `"strip"` is used).

Contents come from the same stored text `/v1/extract` returns, read after
ranking, under the stored source's and publisher's current policy. No page is
fetched, and nothing is summarized or generated. They belong to the one
admitted search and consume no further quota. Pages are read one after
another, so a search with contents takes longer than one without. Each result
then carries a `contents` object; these values are illustrative:

```json
{
  "status": "ok",
  "contentChangedAt": "2026-10-03T10:00:00.000Z",
  "totalCharacters": 5120,
  "text": "Jupiter has dozens of known moons. …",
  "characters": 4000,
  "truncated": true,
  "highlights": [
    { "text": "The four largest moons are called the Galilean moons.", "start": 812, "end": 865 }
  ],
  "offsetUnit": "unicode_code_points"
}
```

| Field | Meaning |
|---|---|
| `status` | `ok`; `page_not_indexed` when the page is unavailable under current stored source or publisher policy; `extraction_unavailable` when the read failed. Otherwise only `status` is present, and the result keeps its rank |
| `contentChangedAt` | The stored text version that `text` and the offsets come from |
| `totalCharacters` | Code points in the stored text |
| `text`, `characters`, `truncated` | As in [indexed text extraction](#indexed-text-extraction): the leading text, its code-point count, and whether it was clipped to the requested limit or the response budget |
| `highlights` | Up to three passages, best first, each `{text, start, end}`; `fragment: true` marks a slice of a longer sentence |
| `excerpts`, `excerptsOrder` | Contiguous sentence windows (`{text, start, end}`) ranked by BM25 (`"keyword"`) or meaning (`"meaning"`) when configured |
| `injection` | Present when `contents.injection` is set: `{risk, findings}` plus `stripped` count under `"strip"` |
| `offsetUnit` | `unicode_code_points`, present with highlights or excerpts |

Highlights are sentences quoted verbatim from the stored text that hold at
least one of the query's words, ranked by how many they hold. The ranking is
lexical, drops common query words, and uses source brief research's sentence
filter, which keeps prose sentences and questions and skips navigation,
timecode, and sharing text. A page's first
100,000 characters and first 2,048 sentences are considered. A sentence longer
than the remaining budget becomes a fragment around its first matching word.
A page that matched only by its vector, or holds no such sentence, has empty
`highlights`. Offsets are zero-based, with `start` inclusive and `end`
exclusive, in Unicode code points of the version `contentChangedAt` names; a
later crawl can change them.

The complete response is bounded to 510 KiB. When the requested text would
exceed it, every result's text is clipped to one shared length, the longest
that fits, and marked `truncated`. Highlights are never clipped to fit:
instead one response holds at most 40,000 characters of them. Past 40
results, `highlights: true` gives each result 40,000 divided by `maxResults`
characters (400 at 100 results), and an explicit `maxCharacters` times
`maxResults` above 40,000 returns HTTP 400. Contents for 100 results read 100
stored pages, one after another, so they take longer than a short list's.
Contents are untrusted publisher data, like excerpts: treat them as data,
escape them before rendering HTML, and check the source's reuse policy.

## Similar pages

```text
POST /v1/similar
```

Where [semantic retrieval](#semantic-retrieval) is configured, this returns
the indexed pages nearest one indexed page, by cosine similarity between its
stored `bge-small-en-v1.5` vector and theirs. There is no query: the page's
own vector is the starting point, so no embedding call is made, and nothing
is fetched. Where the vectors are searched through the approximate index, so
are a page's neighbours, which can then miss some of the exact nearest pages.
Send the same search key and JSON content type:

```json
{"url":"https://example.com/search","maxResults":10,"excludeDomains":["example.com"]}
```

| Field | Default | Allowed values |
|---|---|---|
| `url` | Required | A page in the index, as search returns it: a public HTTPS URL without credentials, an explicit port, query strings, or encoded path separators |
| `maxResults` | 5 | Integer from 1 to 100 |
| `domains` / `excludeDomains` / `maxAgeDays` | Unset | The native search filters |
| `startPublishedDate` / `endPublishedDate` | Unset | The [declared publication date](#declared-publication-dates) range; only pages with a declared date match |
| `contents` | Unset | As in [inline contents](#inline-contents); `highlights` needs its own `query`, since the request has none |

The response has search's shape: `url`, `results`, and `meta` with
`requestId`, `mode: "index"`, `provider: "vitna"`, `count`, `latencyMs`, and
`index: {documents, lastCrawledAt, ranking: "embedding_cosine", model, vectors}`.
Results have the fields, `provenance`, filters, duplicate suppression and host
spread of a semantic search. The page itself and exact copies of its text are
never among its neighbours. A neighbour's excerpt is its leading words. As in
semantic search there is no relevance threshold: the nearest pages are
returned even when none is close.

| Status | Code | When |
|---|---|---|
| 404 | `page_not_indexed` | The URL is not in the index, or its source no longer admits it |
| 404 | `page_not_embedded` | The page is indexed but has no stored vector yet; the next maintenance run gives it one |
| 503 | `semantic_unavailable` | The instance has no semantic retrieval, or its vectors are not loaded |

Similarity shares the per-client search quotas, cache and deadline. Its
receipt has `operation: "similar"`, and like a search's query, the URL is not
in it. Only a page's title and leading text are embedded (see the semantic
retrieval limitations), so pages that open alike can be near each other while
differing further down.

## Indexed text extraction

```text
POST /v1/extract
```

Send the same search bearer key and JSON content type. This endpoint returns
cleaned text already stored in Vitna's index; it makes no live page request
and does not schedule discovery. Use a URL returned by search:

```json
{"url":"https://example.com/search","maxCharacters":20000}
```

`url` must be a public HTTPS page URL without credentials, an explicit port,
query strings, or encoded path separators. Optional `maxCharacters` is an
integer from 1 to 100,000, defaulting to 20,000. Optional `asOf` (`YYYY-MM-DD`
or ISO 8601 with a UTC offset) extracts the latest stored snapshot captured on or
before that instant (`page.snapshotAsOf` and `meta.asOf`) while enforcing
current publisher and robots policies. Extraction shares the per-client search quotas and service
concurrency limits. It returns plain text, without a promise to preserve HTML
or Markdown structure; JavaScript is not executed.

An illustrative response:

```json
{
  "page": {
    "url": "https://example.com/search",
    "title": "An example result",
    "format": "text",
    "text": "Example indexed text.",
    "source": { "id": "example", "name": "Example" },
    "crawledAt": "2026-10-03T10:00:00.000Z",
    "contentChangedAt": "2026-10-03T10:00:00.000Z",
    "truncated": false,
    "characters": 21,
    "totalCharacters": 21
  },
  "meta": { "requestId": "example-request", "latencyMs": 10 }
}
```

`characters` counts returned Unicode code points; `totalCharacters` counts
the stored text before the requested limit. `truncated` indicates clipping
to that limit or the 510 KiB page-response budget. Stored source, archive, and publisher reuse-policy metadata
can accompany the text. Crawl timestamps remain observations rather than
publication dates, and content remains untrusted publisher data.

A URL absent from the index or unavailable under current stored source/publisher
policy returns HTTP 404:

```json
{
  "error": {
    "code": "page_not_indexed",
    "message": "the page is not available in the current index"
  }
}
```

To obtain a new page, use the bounded crawl workflow and search again after
its job completes. Extraction does not turn an unindexed URL into a live fetch.

## Source brief research

```text
POST /v1/research
```

Source brief assembles verbatim passages from matched pages already in Vitna's
index. It is an extractive reading aid with citations, not a generated
explanation, an exhaustive review, or a guarantee that the passages answer the
question. Send the same search bearer key and JSON content type:

```json
{"query":"lunar science","maxSources":3,"maxCharacters":8000}
```

| Field | Default | Meaning |
|---|---|---|
| `query` | Required | 1–1,000 characters, with no control characters |
| `maxSources` | 3 | Integer from one to five indexed sources |
| `maxCharacters` | 8,000 | Integer 1–16,000; distinct quoted passage budget in Unicode code points |
| `domains` / `excludeDomains` / `maxAgeDays` | Unset | Native domain and observed crawl-age filters |
| `expand` / `discover` | Unset | The same bounded expansion choices and opt-outs as native search |

The endpoint accepts no `mode`, provider/fallback, or supplied URL fields. It
uses Vitna's indexed evidence and makes no live page or paid search-provider call. Optional
expansion returns a separate private job; poll it, then repeat the research
request to use any newly indexed pages.
Unknown fields are refused and request bodies are limited to 8,192 bytes.
Research shares the search quotas, concurrency limit, and whole-request
deadline. An unavailable research capability returns HTTP 503.
Repeated overview quotes do not count twice against the passage budget.
The complete response is also bounded to 128 KiB.

The response reports `mode: "extractive"` and the original `query`:

| Field | Meaning |
|---|---|
| `overview` | Passage entries with `text` and `citationIds` |
| `sections` | Source sections using actual indexed page titles, `sourceId`, and cited `passages` |
| `citations` | IDs linked to a `sourceId`, `start` and `end` offsets, and `offsetUnit: "unicode_code_points"` |
| `sources` | IDs, URLs, titles, publisher identity, observed timestamps, optional archive metadata, and `readTruncated` |
| `results` | Result cards for indexed sources actually quoted |
| `coverage` | Bounded evidence counts and limitations |
| `meta` | Request identifier, latency, `provider: "vitna"`, and optional index/expansion metadata |

Each section's `sourceId` maps to a source entry. Citation offsets are
zero-based, with `start` inclusive and `end` exclusive, in the indexed text
version read for that source. They count Unicode code points, not bytes or
JavaScript string units. Use the same stored text version when checking a
quote; a later crawl can change its offsets. A passage marked `fragment: true`
is a bounded slice of a longer sentence, still copied verbatim.

`coverage.status` is `matched`, `partial`, or `insufficient`. Its scope is
`matched_indexed_evidence` and `exhaustive` is always `false`; it reports
`candidateSources`, `usedSources`, `passages`, `readTruncated`, `skippedSources`,
`attemptedSources`, `unavailableSources`, `catalogUnavailable`, `outputTruncated`,
`quotedCharacters`, and optional `indexMatch`. `matched` means selected passages
without skipped sources, clipped reads, clipped output, or sentence fragments.
`partial` has passages with one of those limitations; `insufficient` has no
selected passages, including when the requested character budget is too small.
These describe the selected indexed evidence,
not the completeness of the web or the truth of a conclusion. Source text is
untrusted publisher data. Check the cited pages and their reuse policies.

## Usage and receipts

Native retrieval does not report a per-query provider bill. Infrastructure
and crawl work still consume resources; the separate crawl meters bound that
work. Receipts do not store search queries or retrieved content.

Tavily basic search is explicitly selected, automatic parameter selection is
off, and the request asks for reported credit usage. Exa automatic search
requests highlights; its reported dollar cost is returned as
`estimatedCostUsd`, because Exa documents that response field as an estimate
rather than the basis of billing. Brave returns no dollar usage figure in
this contract. Missing usage stays null. Provider-reported zero is preserved.

Each admitted search, similar-page, SERP, extraction, or research request emits one JSON receipt to stdout:
client identifier, request identifier, time, status, and latency. Search
receipts include provider attempts and reported usage, plus the retrieval used
and any fallback code when semantic retrieval is configured or requested; extraction receipts
identify `operation: "extract"`, research receipts `operation: "research"`,
similar-page receipts `operation: "similar"`, and SERP receipts `operation: "serp"`.
Queries, requested URLs, returned content,
authorization headers, and credentials are omitted. Capture stdout with the
process supervisor if receipts are needed.
Receipt output is best effort and is not a durable billing ledger.

A service that accepts platform keys also sends the receipts for an org's
requests (clients written `org.<id>`) to vitna.ai in the background, where the
org's console lists them in Activity. The same allowlist applies there, so
nothing the line above omits reaches it. See [One key](../one-key.md) for the
endpoint and the settings that turn it off.

## Status and limits

```text
GET /v1/indexes
GET /v1/status
GET /healthz
GET /readyz
```

`GET /v1/indexes` requires a search key and returns the catalog of queryable
indexes (`web` when configured, followed by all 23 vertical indexes with
descriptions, data sources, retention windows, content-hash deduplication flags,
live readiness, and indexed document counts). On vitna.ai, the public
[Vitna Library](/library) catalog lists these vertical indexes and Web API data
sources.

Status requires a search key and reports the client identifier, configured
providers, first provider, supported `modes`, and limits. It exposes safe
availability information for configured operator routes, without credentials.
The separate `serp` capability reports the retrieval adapter's own Google/Bing
availability and engines, with safe transport settings. Its `retrieval`
metadata includes configured attempt limits, cache statistics, and safe
per-route/per-engine health and concurrency counters. Runtime-unavailable routes remain disabled until
restart. It does not probe a search engine or disclose proxy credentials.
`expansion.siteDiscovery` reports `enabled`, `automaticNames`, `automaticTopics`
and `automaticConcepts` (true when configured), `limits: {origins: 3, candidates: 2000}`,
and safe name-directory and publisher-catalog aggregate counts.
`topicCatalog` identifies `source: "local_topic_descriptors"` and reports
`publishers` (unexpired descriptors), with `observed` and `reviewed` counts.
`candidates` counts separate unverified domain metadata; `imports` records
dataset attribution and retrieval metadata. None of these are indexed document
counts. Optional `conceptCatalog` reports `source: "wikipedia_titles"`,
`language: "en"`, `titles`, `snapshot`, `importedAt`, `sourceUrl`,
`license: {name, url}`, and a `changes` description of normalized lookup keys
with original page titles retained.
Its title count measures metadata, not indexed articles. Redirect titles have
no canonical-target mapping in this catalog; article text becomes usable only
after an allowed page is fetched and indexed. Health is public and reports only that
the HTTP process is up; it makes no paid provider probe. Status also reports
`extraction: {enabled, indexedOnly: true, limits: {defaultCharacters: 20000, maxCharacters: 100000}}`
and `contents: {enabled, indexedOnly: true, limits}`, whose limits are
`defaultTextCharacters`, `maxTextCharacters`, `defaultHighlightCharacters`,
`maxHighlightCharacters`, `highlightsPerResult`, `highlightReadCharacters`, and
`responseBytes`.
`research` reports `enabled`, `mode: "extractive"`, and
`limits: {maxSources: 5, maxCharacters: 16000}`. `similar` reports
`{enabled, indexedOnly: true, limits: {maxResults: 100}}`, enabled where
semantic retrieval is configured. Top-level `snapshotAsOf: true` indicates
support for point-in-time `asOf` and `contents.snapshotAsOf` queries. Where semantic retrieval is
configured, status adds `retrieval: {default: "hybrid", modes, semantic}`,
whose `semantic` reports the `model`, the loaded `vectors` count, `dimensions`,
the `bytes` one query worker holds, and a `state` of `ready`, `missing`,
`empty`, `incompatible`, `over_limit`, or `unavailable`. When the vectors are
searched through the approximate index, `semantic.approximate` adds
`indexed` (vectors in the index), `exact` (vectors stored since its build),
`lists`, `probes`, `rescore`, and `builtAt`. The index status counts
`importedDocuments` beside `liveDocuments` and `archivedDocuments`, and a bulk
source in `sources` reports its `contentOrigin`. The browser uses ordinary result
search by default. Source brief is an explicit option when this capability is
available; it shows only pages with usable quoted evidence and can return fewer
cards than ordinary search.
Readiness is also public and follows the default `/v1/search` route. When
the native index is the default provider, readiness returns HTTP 503 while
that index has no documents or its status cannot be read, whether or not SERP
retrieval is enabled. A SERP-only instance reports its retrieval adapter's
own availability, without probing a search engine. Authenticated status
remains available with enabled SERP if an index
status read fails, reporting `index: {ready: false, unavailable: true}`.
Authenticated
status includes local source counts, crawl freshness, and the latest crawl
summary when the Vitna provider is configured. The root `/` serves a browser
interface that keeps a pasted key only in that open page.

| Environment variable | Default | Meaning |
|---|---|---|
| `VITNA_SEARCH_HOST` | `127.0.0.1` | Listening address |
| `VITNA_SEARCH_PORT` | 8789 | Listening port |
| `VITNA_SEARCH_PROVIDERS` | `vitna` | Unique ordered provider identifiers; external adapters require explicit selection and credentials |
| `VITNA_SEARCH_INDEX` | `.search-data/web/index.sqlite` | Independent SQLite index; initialized if needed |
| `VITNA_SEARCH_SERP_ENABLED` | `false` | Enables independent Google/Bing SERP retrieval without requiring a native index; also requires `VITNA_SERP_RESALE_APPROVED=true` |
| `VITNA_SERP_RESALE_APPROVED` | `false` | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted |
| `VITNA_SEARCH_SERP_TRANSPORT` | `auto` | Bounded direct/browser routes; proxies pin browser egress; explicit `direct` or `browser` also supported |
| `VITNA_SEARCH_SERP_TIMEOUT_MS` | 10,000 | SERP retrieval timeout within the shared whole-request deadline |
| `VITNA_SEARCH_SERP_PROXIES` | Unset | Private operator JSON array of browser proxy URLs |
| `VITNA_SEARCH_MODE` | Unset | `web` selects the general beta's default index and frontier paths |
| `VITNA_SEARCH_FRONTIER` | Unset outside web mode | Existing separate crawl-state file; adds authenticated crawl status |
| `VITNA_SEARCH_CONCEPTS` | `.search-data/web/concepts.sqlite` in web mode; unset otherwise | Separate SQLite title-metadata catalog; optional native concept discovery |
| `VITNA_SEARCH_DEMAND_SECRET_FILE` | Unset | Private file containing a 64-character lowercase hexadecimal key; enables encrypted site-discovery jobs in web mode |
| `VITNA_SEARCH_QUOTA` | Unset | Separate writable SQLite quota file; persists per-client minute/day limits |
| `VITNA_SEARCH_RPM` | 30 | Admitted search, SERP, extraction, and research requests per client per fixed minute |
| `VITNA_SEARCH_DAILY_LIMIT` | 1,000 | Admitted search, SERP, extraction, and research requests per client per UTC day |
| `VITNA_SEARCH_CONCURRENCY` | 4 | Simultaneous search, SERP, extraction, and research requests across the instance; a SERP search waiting for a busy route's slot counts, so keep it well above `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` |
| `VITNA_SEARCH_TIMEOUT_MS` | 10,000 | One whole-request deadline, including sequential legacy fallback attempts |
| `VITNA_SEARCH_EMBED_URL` | Unset | Embeddings endpoint ending in `/embeddings`, HTTPS or loopback HTTP; with the model, enables semantic retrieval |
| `VITNA_SEARCH_EMBED_BATCH_URL` | Unset | Optional second endpoint, held to the same rules, that vector maintenance uses instead, so a query never waits behind a batch; queries always use `VITNA_SEARCH_EMBED_URL` |
| `VITNA_SEARCH_EMBED_MODEL` | Unset | Model identifier sent to the engine; a different echoed model is refused |
| `VITNA_SEARCH_EMBED_KEY` | Unset | Optional bearer token for a private remote endpoint |
| `VITNA_SEARCH_EMBED_QUERY_PREFIX` | Empty | Prepended to each query, for example `Represent this sentence for searching relevant passages: ` for BGE models |
| `VITNA_SEARCH_EMBED_DOCUMENT_PREFIX` | Empty | Prepended to each page's text and covered by its fingerprint |
| `VITNA_SEARCH_EMBED_DIMENSIONS` | Unset | Expected vector width; responses of another width are refused |
| `VITNA_SEARCH_EMBED_TIMEOUT_MS` | 2,000 | Query embedding deadline before the keyword fallback, 100–60,000 and at most half of `VITNA_SEARCH_TIMEOUT_MS` |
| `VITNA_SEARCH_EMBED_BATCH_SIZE` | 4 | Pages per maintenance request, 1–256; an engine serves one request at a time, so without `VITNA_SEARCH_EMBED_BATCH_URL` a larger batch makes searches wait longer |
| `VITNA_SEARCH_EMBED_BATCH_TIMEOUT_MS` | 60,000 | Deadline per maintenance request, 1,000–600,000 |
| `VITNA_SEARCH_EMBED_MAX_CHARACTERS` | 1,500 | Characters of title and leading text embedded per page, 100–8,000 |
| `VITNA_SEARCH_VECTORS` | `vectors.sqlite` beside the index | Separate vectors file, never the index, frontier, concept or quota file |
| `VITNA_SEARCH_VECTORS_MAX_MB` | 512 | Bound on each query worker's copy of the vectors, 1–16,384; a larger file is searched through its approximate index, which with the vectors stored since its build must fit this bound |
| `VITNA_SEARCH_VECTORS_PROBES` | 64 | Lists of the approximate index scanned per search, 1–4,096 |
| `VITNA_SEARCH_VECTORS_RESCORE` | 400 | Approximate candidates rescored exactly per search, 10–4,096 |
| `VITNA_SEARCH_EMBED_BULK` | `false` | `true` makes vector maintenance embed the pages of bulk imports too, after crawled pages |

Search, SERP, extraction, and research requests that fail after admission consume a quota slot;
this includes extraction 404 responses. A legacy upstream request may already
have been paid for. Shared API `service_busy` refusals and invalid requests
consume no quota. SERP route-busy and cooling responses occur after admission
and consume one slot. Fallback attempts are part of one admitted search; at most
four providers can be attempted, including at most three external providers.
Without `VITNA_SEARCH_QUOTA`, quotas live in process memory and reset on restart.
With it, per-client minute/day counters persist in SQLite and accounting failures
refuse searches. The web deployment template sets a quota file. Run one alpha
instance; these limits are not a distributed customer billing ledger.

All responses use `Cache-Control: no-store`. Errors carry an `error` object
with `code` and `message`. Completed attempt sequences also carry metadata.

| Status | Meaning |
|---|---|
| 400 | Invalid JSON or unsupported request shape, including removed Live/Hybrid modes (`invalid_request`) |
| 401 | Missing or invalid search credential |
| 404 | Unknown service path, or an extraction URL unavailable in the current index |
| 405 | Wrong HTTP method; `Allow` names the supported method |
| 413 | Request body exceeds the byte limit |
| 415 | Content type is not JSON |
| 429 | Minute limit, day limit, or service concurrency limit; includes `Retry-After` |
| 499 | Caller cancelled the request |
| 502 | No attempted provider completed the search |
| 503 | A provider, SERP, extraction, or research is unavailable, the required local index is empty, or quota accounting fails; also every SERP retrieval or parsing failure, a missed deadline included, told apart by its code |

Upstream error bodies are never forwarded. Legacy search-provider redirects
are refused, and their response bodies are capped at 1 MiB. SERP uses its
own bounded, validated search-engine retrieval. Disconnecting a caller aborts
the current request and prevents another fallback attempt.

## Compare retrieval outputs

Compare speed and result counts within the same task and freshness policy.
These endpoints return different evidence:

| Route | What it returns | What a successful response establishes |
|---|---|---|
| Google SERP | Observed Google organic listings, engine snippets, page ranks, and resolved publisher links | A captured result page or bounded collection; snippets are not quotations fetched from the publisher pages |
| Indexed search | Keyword-ranked matches from Vitna's stored corpus, or hybrid-ranked where semantic retrieval is configured, with crawl and source metadata | Matches in the configured index; neither a lexical match nor vector similarity is a finding supported by source evidence |
| Native source brief | Verbatim passages from indexed pages with source references and Unicode citation offsets | Extractive indexed evidence with explicit coverage; it does not generate an answer or establish exhaustive research |

An owned-route provider fee of zero excludes cloud hosting, browser CPU,
network transfer, storage, crawling, and operating costs. Optional external
search charges are separate. Compare result relevance,
evidence quality, coverage, and freshness alongside latency; a larger link
count is not the same deliverable as a smaller cited source brief.

## Provider protocol references

The adapters follow the providers' own interfaces, checked on 2026-10-03:

- [Brave Web Search](https://api-dashboard.search.brave.com/app/documentation/web-search/get-started), ranked web results and subscription-token authentication.
- [Exa Search](https://exa.ai/docs/reference/search), highlights, request identifiers, and estimated cost metadata.
- [Tavily Search](https://docs.tavily.com/documentation/api-reference/endpoint/search), bearer authentication, basic depth, and reported credit usage.

## Before a commercial launch

The private beta serves configured clients and upstream accounts. Public signup,
self-service key issuance and revocation, an invoice-grade usage ledger, customer
billing, and public self-service deployment need a further implementation. The
included search interface works on a configured instance. Search
is a separate service from the existing routing subscriptions; no existing
plan gains or is charged for search in this change.
Check provider contracts before exposing their results commercially. Brave's
[standard terms](https://api-dashboard.search.brave.com/documentation/resources/terms-of-service)
prohibit resale/redistribution and replacing its service; ordinary API access
does not authorize a public resale search service. Google and Bing results
need the same decision: SERP retrieval starts only after the operator confirms,
with `VITNA_SERP_RESALE_APPROVED=true`, that collecting and redistributing them
is permitted.

## Operator reference

The sections from here to the end are for the people who run a Search
instance, which during the beta means Vitna's own operators, working from a
checkout of Vitna's private repository. None of it is needed to call the
hosted API at `https://search.vitna.ai`, and none of it is an offer to run a
deployment of your own.

Try the complete offline demo with `node platform/search/demo.mjs`, then open
`http://127.0.0.1:8789`. Its four pages are clearly labeled illustrative fixtures;
the demo makes no external requests. See [Run a focused search index](self-hosted.md)
for live crawling, OVHcloud preparation, and R2 snapshots.
The [general web beta](general-web.md) adds persistent cross-site discovery,
day/month crawl budgets, file-backed API quotas, and the Virginia deployment package.

Since 2026-10-07 a browser that opens the hosted instance, or its `/docs`, is
sent to Search's page and this reference on vitna.ai, where the product's
pages live: the operator sets `VITNA_SEARCH_PAGE_URL` and
`VITNA_SEARCH_DOCS_URL`, and a self-hosted instance with neither set serves
its own page and API reference.

## Starting an alpha instance

Node.js 22.13 or later is required; Node 24 is recommended for the deployment
package. Direct SERP retrieval and indexed search use built-in Node modules
and need no npm dependencies or website build. The optional browser SERP
transport, including browser attempts selected by `auto`, requires the
separate Unblocker runtime. Indexed search uses the
currently experimental `node:sqlite`. Startup opens
`.search-data/web/index.sqlite` by default and initializes an empty local
index when needed. Populate selected sources before serving useful native
searches. SERP retrieval does not require populated indexed pages or crawler work.

Mint a credential for each alpha client:

```bash
npm run search:key -- pilot
```

The command prints a `vitna_search_` token once, followed by a JSON entry
mapping the client identifier to its SHA-256 hash. Give the token to the
client privately. Merge the hash entry into `VITNA_SEARCH_CLIENTS`, a JSON
object. The service accepts these search keys and an org's platform key
(`vitna_key_`, see [one key](../one-key.md)); push and router keys grant no
search access. Remove a client's entry and restart to revoke its access.

For native retrieval, set `VITNA_SEARCH_CLIENTS`. The default provider is
`vitna`, with an index at `.search-data/web/index.sqlite`; override its path
with `VITNA_SEARCH_INDEX`. Crawl selected sources before searching their
content. See the [index setup](self-hosted.md). An operator can still select
legacy external adapters explicitly, but native startup has no external
provider fallback.

For actual Google/Bing results, set `VITNA_SEARCH_SERP_ENABLED=true` and
`VITNA_SERP_RESALE_APPROVED=true`, and follow the
[SERP setup](serp.md#run-without-an-index). The second setting records the
operator's decision that collecting those results and redistributing them
through Vitna's API is permitted; startup refuses the first without it. The
same client key authenticates both capabilities; neither requires a Bright
Data key or zone.

Start the API with:

```bash
npm run search:start
```

For a local environment file, keep a private `.env.search` in the checkout,
using the settings described in `deploy/vitna-search.env.example`. Start it
with Node's environment-file loader:

```bash
node --env-file=.env.search platform/search/start.mjs
```

By default the server binds to `127.0.0.1:8789`. Boot refuses an empty client
registry, invalid limits, duplicate credential hashes, no enabled retrieval
capability, or SERP retrieval enabled without its redistribution confirmation.
The local index is initialized if needed; an empty index can start but native
retrieval, and with it readiness, remains not-ready until a crawl succeeds,
even with SERP retrieval enabled. A SERP-only instance can be ready without
an index. Starting the
process makes no upstream search request and does not run a crawl.

## Maintaining the vectors file

Where [semantic retrieval](#semantic-retrieval) is configured, maintain the
vectors file with:

```bash
npm run search:embed
```

The job reads the index read-only, embeds new pages and pages whose embedded
text changed, deletes vectors of pages that left the index, stops at `--limit`
pages (default 25,000) or `--budget-ms` (default 480,000), and prints one JSON
summary line. A vectors file holding another width or model stops the run with
`vectors_dimensions_mismatch` or `vectors_model_mismatch`; any other error in
the vectors file or the index exits 1 with the error on stderr and no summary.
Every batch commits on its own, so an interrupted run can simply
run again; a lease in the vectors file keeps two runs from overlapping. A text
the engine refuses is retried with its character limit halved, and halved
again while the engine keeps refusing (750, 375, then 187 characters for the
default 1,500), stopping before the limit falls under 100 or once halving no
longer shortens the text. It is then recorded and skipped until the page
changes. The
[general web deployment](general-web.md#semantic-retrieval-deployment)
packages it as a service and timer.

The job compares pages and vectors in document-id ranges of 100,000, so its
memory stays bounded however large the index grows. Pages of a bulk import
(an imported Wikipedia dump) are embedded only with
`VITNA_SEARCH_EMBED_BULK=true`, and then after every crawled page that needs a
vector; without it they are passed over and any vectors they already have are
kept. The summary's `documents` counts the pages the job considered.

Once the vectors no longer fit `VITNA_SEARCH_VECTORS_MAX_MB`, build the
approximate index from the vectors file, with the same embedding settings as
the API:

```bash
npm run search:ann
```

It reads the vectors file only, writes `vectors.ivf` beside it under a
temporary name and renames it into place, and prints one JSON line with the
vector count, lists, file size and duration. Query workers load the new index
on their own. Vectors stored after a build are searched exactly beside the
index, so rebuild when they no longer fit beside it, or when a large share of
the file has changed.
Human Machine Shift+M