> Source: https://vitna.ai/docs/search/general-web
> Raw Markdown: https://vitna.ai/docs/search/general-web.md
> Corpus index: https://vitna.ai/llms.txt

# Run the general web beta

Vitna Search can grow its own index across public websites, keep a durable crawl
queue, and serve authenticated searches without a paid search provider. This is
a bounded beta with limited coverage. The API is deployed at
`https://search.vitna.ai` on a reused Virginia OVH server, with private R2
backups. No additional OVH instance was purchased.

Index is the browser's default search mode. Searches return the current
corpus immediately. Empty name/topic queries and topic queries with only
partial-term matches can automatically queue discovery of a few candidate
sites, then crawl selected pages into the same
index. Domain input is optional. This uses Vitna's own
crawler and makes no paid search-engine call. It does not promise whole-web
coverage or immediate access to pages that have not been indexed.
Source brief can assemble cited, verbatim passages from the matched indexed
pages; it does not generate an explanation.

## Current evidence

Three local live runs through 2026-10-03 used 460 HTTP requests and recorded about
31.3 MB of response payload. They indexed 65 real pages, registered 250 origins,
and left 27,066 URLs pending. Registered origins and queued URLs are discoveries,
not searchable coverage. Some publishers refused requests, timed out, or had
unavailable robots files; those failures remain recorded for bounded retries.

A staging import of all of English Wikipedia (6,030,988 articles, snapshot
20261001) answered 67.7% of NQ-open's questions in keyword retrieval's top
ten, against 16.1% on the production index; it is not deployed. See
[Wikipedia at full scale](#wikipedia-at-full-scale).

A separate synthetic test inserted 100,000 documents in 16.2 seconds and
measured 79 ms median / 91 ms 95th-percentile retrieval on this PC over 100
queries. Its database was about 454 MB. These measurements concern the keyword
index, not web relevance, real page sizes, or OVH performance.

A later local HTTP test against 100,000 synthetic pages completed 100/100
requests with four simultaneous callers, two query workers, and caching off.
Median search latency was 314 ms; the 95th percentile was 1,382 ms. Public
health checks had a 1.1 ms 95th percentile. This is a short local smoke test,
not sustained throughput or a production capacity guarantee.

```bash
npm run search:benchmark -- --documents 100000
npm run search:benchmark -- --documents 100000 --http --concurrency 4
```

`--vectors 384` adds one synthetic 384-dimension vector per page. On
2026-10-06, `--documents 49063 --vectors 384 --http --concurrency 2` on a
desktop Ryzen 7 3700X measured a query worker's full load of the 49,063
vectors at 434 ms and 79.5 MB, the top-40 scan at 24.7 ms median and 25.4 ms
at the 95th percentile, and 500 changed vectors applied in 5.3 ms. Over HTTP
with two workers, hybrid searches took 119 ms median against 72 ms for
keyword, with 95th percentiles of 650 and 564 ms. The benchmark makes its
query vectors in process, so those hybrid times exclude the engine call, about
10 ms on an idle engine.

## Crawl and preview locally

Node 24 is recommended. The service uses built-in modules and needs neither
an npm install nor a website build. Leave at least 10 GiB free on the data disk.

Initialize local databases and a private pilot credential with:

```bash
npm run search:setup
```

This creates `.env.search` and `.search-data/private/pilot.key` without printing
the token. It preserves an existing environment file and key. Both are excluded
from Git and release archives. The new environment selects the Vitna index and
loopback port 8791. Start the authenticated local API with:

```bash
node --env-file=.env.search platform/search/start.mjs
```

There is no crawl or cloud deployment during setup. An empty index stays
not-ready until a successful crawl. Windows does not enforce POSIX file modes;
store these private files in an account-restricted workspace. For the browser
preview with its own temporary credential, use:

```bash
npm run search:web -- --max-requests 80
npm run search:web:preview
```

The first command makes real public HTTP requests. Its override can only lower
the configured run limit. The second opens a loopback-only service at
`http://127.0.0.1:8790`, with a temporary key usable only by that preview. It
searches the real local index and performs no crawl. Its quota database is in
memory; production uses the file specified below.

Live runs can exit with a failure status while preserving successfully indexed
pages. Read the summary's `failed`, `errors`, and `stopReason`. Requests stopped
by a run limit resume from the queue on the next invocation. The preview never
creates provider calls or reads the site's database.

## Crawl scope and limits

The starter `platform/search/web-seeds.json` contains 28 public sites spanning
science, health, books, parks, government, art, economics, and software. Links
can add other public HTTPS origins within the configured cap. Coverage and
permission are evaluated as each origin is visited; a seed is not a promise
that its site will be indexed. Pages in languages other than English can enter
through discovered links.

`platform/search/web-profile.json` sets these defaults:

| Control | Default |
|---|---:|
| Total searchable documents | 100,000 |
| Documents per origin | 2,000 |
| Documents per bulk-imported source (a Wikipedia dump) | 8,000,000 |
| Registered origins | 1,000 |
| Queue entries | 250,000 |
| Concurrent origin jobs | 4 |
| HTTP requests per run | 1,000 |
| Jobs per run / maximum duration | 4,000 / 15 minutes |
| Requests per UTC day / month | 25,000 / 600,000 |
| Metered payload per UTC day / month | 4 GiB / 120 GiB |
| Daily/monthly request and payload capacity reserved for demand work | 20% |
| Raw and decompressed page size | 1 MiB each |
| Raw and decompressed discovery file size | 2 MiB each |
| Sitemap files per origin / nesting depth | 6 / 2 |
| Advertised feeds per page / feed page refresh | 2 / 24 hours |
| Minimum origin spacing | 1 second, increased for robots rules |
| Regular page refresh / robots cache | 7 days / 24 hours |
| Discovered link depth | 4 |
| Failed attempts before manual retry | 5 |
| Minimum extracted words | 40 |
| Minimum free disk space | 10 GiB |

The queue, origin delays, robots cache, retry state, and day/month meters are
stored in SQLite. Every HTTP request, including robots, discovery, and
redirects, reserves budget before starting. Successful reads refund unused
payload reservation; crashes leave the full reservation charged. These meters
bound crawl work and payload, not transport headers, compute charges, or an
account-wide cloud bill. Changing the profile does not prune existing queue
entries or documents to a smaller cap. Validate a smaller profile against
existing counts before activating it.

The first two caps count crawled pages only (Common Crawl imports included).
A [bulk import](#import-english-wikipedia) fills its own source, which has
its own cap, `maxDocumentsPerBulkSource`, so millions of imported articles
neither stop the crawler at its 100,000 nor count against
`en.wikipedia.org`'s crawl allocation, and the crawler's caps are no looser
than before. The crawler's source sync never trims or removes a bulk source.

Background work stops when shared daily or monthly usage reaches 80% of its
request or payload limit, reserving the remainder for user-requested jobs.
Both lanes charge the same durable totals; the reserve does not create an
additional budget or guarantee immediate crawl capacity.

At a full frontier, background discovery updates known entries without
registering new sites that cannot receive queue space. Each run may reclaim up
to 64 automatically registered origins that have never been visited and have
no queued URL, robots cache, lease, delay, or indexed document. Configured seeds
and current demand targets remain protected.

Newly reviewed starting sources also receive bounded queue admission when the
frontier is full. Each pass admits at most eight missing seed pages across
sources, replacing only untouched discoveries; indexed pages, crawl history,
active leases and configured seeds stay protected. The catalog now includes
28 starting sources spanning science, government, health, general reference,
travel, programming, reporting and textbooks. These are starting points for
the crawl, rather than a claim of complete coverage on those sites.

## Query-guided coverage expansion

In configured web mode, an empty native name/topic search or a topic query
with only partial-term matches can queue site discovery automatically when no domains or URLs are supplied
and external fallback is disabled. A plain request works without domain input
or crawl flags:

```json
{"query":"examplebrand","mode":"index"}
```

The browser enables this option by default. Untouched requests omit the
`expand` and `discover` flags; disabling the option sends both as `false`.
Either explicit `expand: false` or `discover: false` prevents automatic
discovery. Automatic admissions include `meta.expansion.automatic: true`.
Production `meta.index.match` can report `complete`, `partial`, or `none`;
`partial` describes partial-term fallback. Discovery can address that weak
coverage while leaving the current results intact. The field is optional,
and complete term matches do not guarantee that a result answers the query.
Name detection is heuristic. Common topic words and multiword topic queries
do not generate guessed hostnames. Broad queries instead use publisher
description matches from reviewed or observed publisher descriptors. A catalog domain is a
discovery candidate, not an indexed document or a promise of access. A query
without a suitable directory/name, publisher, or configured concept-title match
can return `no_candidates`.

Automatic name lookup uses Vitna's own directory of names observed on fetched
pages; only explicit expansion also matches known URLs or caller-supplied
sites. A narrow distinctive
single ASCII brand can additionally suggest up to three hosts formed with
`.com`, `.ai`, and `.org`. These are unverified candidates, not a claim that
any of them is the official site or matches the intended entity.

Topic discovery matches query words against publisher descriptions, then
examines at most three selected publishers' homepages and sitemaps before
choosing up to eight target pages. It shares the same request, time, policy,
and ownership limits as name discovery. The catalog does not contribute
search results until the crawler successfully indexes pages.

When configured, a separate local English Wikipedia title catalog can match
concepts to candidate article URLs. It contains title metadata, including
redirect titles, rather than article bodies or canonical redirect mappings.
Each candidate must still pass crawler policy and be fetched and indexed
before its text is searchable or available to Source brief. Matching a title
does not establish current page availability or whole-web coverage.

Automatic name lookup uses exact directory names and bounded candidates;
incidental matches in article addresses do not establish a site identity.
Before a candidate enters the index, its HTTPS page must respect robots and
`noindex`, contain useful readable text, and identify the requested name.
Parking pages, challenges and unrelated redirects are rejected. A successful
check establishes a self-identified content match, rather than verified ownership.

The directory learns public names from indexed homepages, retains observations
for 180 days, and stores up to 10,000 sites with eight aliases each. Failed probes
are retained as private HMAC digests for at most 24 hours, preventing repeated
failed requests without saving guessed names. Both tables live in the frontier
database and are covered by the existing backup.

To learn from homepages already in an initialized index, run this optional
local command with the same index and frontier environment settings:

```sh
npm run search:sites -- --backfill
```

It inspects at most 1,000 live homepages, respects enabled source scope and
cached publisher rules, makes no network requests, and prints aggregate counts.

Callers can also request work explicitly. Current indexed results return
first with a separate `meta.expansion` job. Send `expand: true, discover: false`
for target-only refresh. The service selects up to eight relevant known URLs
using words in
their addresses, plus supplied `urls`. `domains` can supply domain roots when
no relevant known URL is available. This job's scope is
`known_urls_and_supplied_targets`.

Add `discover: true` to examine selected sites' homepages and sitemaps before
choosing target pages. It requires `expand: true` and a configured private
discovery key. Optional `domains` guides site selection; without it, sites
come from matching known URLs, supplied URL origins, observed names, narrow
brand-name candidates, topic-catalog matches, or configured concept-title
matches. Automatic discovery skips
incidental known-URL matching. With discovery,
callers can provide at most three `domains` or three distinct supplied URL
origins; ordinary native domain filters retain their ten-host limit.

Site-discovery jobs have scope `site_discovery`. Each examines at most three
origins, including robots files, homepages, and up to two sitemap files per
site, and considers at most 2,000 sitemap candidates. It selects at most eight
target pages within 40 total HTTP requests and 60 seconds of crawler work.
Discovery requests and target fetches share those budgets. URL matches can be
weak, publishers can refuse crawling, and selected candidates may never be
indexed. This is bounded site discovery, not whole-web search.

Jobs are durable in the frontier database, private to the authenticated client,
and expire after 24 hours. Admission allows 64 pending jobs in total and four
per client. Repeating the same normalized query, filters, discovery choice,
and supplied targets returns the same job until expiry. Deduplication uses a
private HMAC digest. Bounded lexical and site hints, including names, candidate
origins and their provenance, are encrypted in private expiring discovery
jobs. The raw original query and full request are not stored or logged.
Plaintext terms are not
persisted in jobs or receipts. Target URLs
remain in the job and normal crawl state.

`GET /v1/search/jobs/<id>` returns progress with the same bearer key. A completed
job means its bounded pass finished; inspect `summary.indexed`, `summary.failed`,
and `summary.demand.unmet` for actual coverage. `summary.discovery` adds integer
counters `origins`, `candidates`, `selected`, and `unavailable`; provisional
name jobs also report `probed`, `accepted`, and `cached`. Candidate and
selection counts do not mean searchable pages; indexed/unchanged counts and
unmet targets describe the crawl outcome. Search again to retrieve newly
indexed pages; the browser's completed-job rerun sets `expand: false` to avoid
repeating the work. Polling allows at most 60 checks per minute and does not consume
the ordinary search allowance. An unknown, expired, or other client's job
returns 404.

Each job uses at most eight target attempts, 40 HTTP requests, and 60 seconds of
crawl work. It follows the crawler's existing public-address validation, robots,
noindex, origin spacing, free-disk guard, and shared daily/monthly budgets. It
does not recursively crawl links or feeds. Ordinary target-only jobs do not
fetch sitemaps; site-discovery jobs add only the bounded pass above. The same global index lease
protects background and demand work. Never-fetched pending queue records may
be replaced to admit explicit targets at the frontier cap; fetched history and
active leases are preserved.

The background runner releases its lease between 100-request or one-minute
batches and checks demand jobs between batches. The demand timer checks every
30 seconds while the background pass is inactive. A busy crawler, publisher
delay, or exhausted budget can postpone work; queued jobs report retry timing.

```bash
curl https://search.vitna.ai/v1/search \
  -H "Authorization: Bearer $VITNA_SEARCH_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"query":"lunar science","mode":"index","expand":true,"discover":true,"domains":["nasa.gov"]}'
```

Set `expand: true, discover: false` to crawl only known and supplied targets.
For name discovery without domains, send `expand: true, discover: true`.
Supplied `urls`
remain public HTTPS page addresses without credentials or query strings and
must satisfy domain restrictions.

Run one bounded demand pass manually:

```bash
npm run search:demand
```

The installer
includes `vitna-search-demand.service` and `.timer`; enable the timer alongside
the API and background crawl units. Jobs are included in existing frontier
backups. This deployment uses Vitna's own crawler and index.

## Initialize the topic catalog

Run these explicit local commands with the existing `VITNA_SEARCH_FRONTIER`
and `VITNA_SEARCH_INDEX` settings loaded into the process environment:

```bash
npm run search:topics -- --init
npm run search:topics -- --review-seeds
npm run search:topics -- --backfill
```

Without those settings, the defaults are `.search-data/web/frontier.sqlite`
and `.search-data/web/index.sqlite`. Initialization creates the frontier/catalog
schema. Seed review records 28 curated publisher descriptions. Backfill reads
at most 1,000 eligible already indexed public homepages, respecting current
source scope and cached publisher rules. These commands make no network
requests and neither register crawl origins nor start crawling. Startup does
not automatically seed or download the catalog.

An optional explicit command downloads a bounded domain-candidate dataset:

```bash
npm run search:topics -- --import-majestic --limit 10000
```

It uses the fixed [Majestic Million CSV](https://downloads.majestic.com/majestic_million.csv),
retaining 10,000 candidates by default and at most 100,000. Reads are bounded
to 100 MiB and 60 seconds. A durable 24-hour cooldown includes failed download
attempts; no force flag or alternate URL is accepted. Failed refreshes preserve
the previous candidate snapshot. Stored attribution identifies
[Majestic Million](https://majestic.com/reports/majestic-million) and its
[CC BY 3.0 Unported license](https://creativecommons.org/licenses/by/3.0/),
retrieval information, and normalization to a bounded subset of HTTPS roots.
That dataset license does not establish permission to reuse publishers'
pages or assets.

Imported candidates remain separate unverified metadata. Importing them does
not create publisher descriptions, register origins, enqueue a crawl, or add
searchable documents. Topic discovery matches reviewed or observed publisher
descriptions; the candidate count does not measure its indexed coverage.

## Import a concept-title snapshot

`VITNA_SEARCH_CONCEPTS` selects a separate SQLite catalog. Web mode defaults
to `.search-data/web/concepts.sqlite`; the API and demand worker must use the
same path, distinct from index, frontier, and quota files. Startup performs no
title download. Import a chosen published English Wikipedia snapshot explicitly:

```bash
npm run search:concepts -- --import-wikipedia --snapshot 20261003
```

The date is an example published snapshot. Select a valid `YYYYMMDD` snapshot
from the [official title dumps](https://dumps.wikimedia.org/other/pagetitles/).
The importer constructs that fixed Wikimedia source URL; it accepts no
alternate download URL or prefix-sized title subset. It reads the complete
gzip stream, validates title records and database integrity, and publishes a
staged replacement only after successful completion. A failed import preserves
the existing catalog.

| Import bound | Limit |
|---|---:|
| Compressed / decoded payload | 160 MiB / 2 GiB |
| Title records / database size | 25 million / 8 GiB |
| Minimum available disk space | 12 GiB |
| Import deadline | 15 minutes |

Original English main-namespace titles and redirect titles are retained with
separate normalized lookup keys. There are no redirect-target mappings or
article bodies. Status records the snapshot, import time, source URL, payload
counts, and contributor/license attribution; consult that attribution and the
publisher's policy when reusing material. This catalog cannot supply search
results or Source brief passages by itself. Candidate pages must first be
fetched and indexed within the crawler's policy and budgets.

The R2 backup described below includes the title catalog once it holds
titles, when `VITNA_SEARCH_CONCEPTS` is set in `web-backup.env`. Without that
setting, preserve it separately or repeat its recorded snapshot import when
restoring concept discovery.

## Private discovery jobs

Configure `VITNA_SEARCH_DEMAND_SECRET_FILE` in the shared
`/etc/vitna-search/web.env`, for example:

```dotenv
VITNA_SEARCH_DEMAND_SECRET_FILE=/etc/vitna-search/demand.key
```

Create that private file with a cryptographically random 32-byte key encoded
as 64 lowercase hexadecimal characters, with an optional final newline. Use
a regular file, not a symbolic link. On Linux, give it to `vitna-search` with
mode 0600; group or public permissions are refused. Use private account ACLs
on Windows. Keep the key out of chat, source control, release archives, and
receipts. The API, background web crawler, and demand unit
must read the same file through their shared environment. The key is needed
only for new site discovery; ordinary retrieval and target-only expansion
remain available without it.

Authenticated `GET /v1/status` exposes
`expansion.siteDiscovery` with `enabled`, `automaticNames`, `automaticTopics`
and `automaticConcepts` (true when configured), `limits: {origins: 3, candidates: 2000}`, and safe
name-directory and publisher-catalog aggregate counts.
`topicCatalog` reports `source: "local_topic_descriptors"`, `publishers`
(unexpired descriptors), `observed`, `reviewed`, `candidates` (separate
unverified domain metadata), and `imports` (dataset attribution/retrieval
records). These do not count searchable documents.
`conceptCatalog` identifies `source: "wikipedia_titles"`, `language: "en"`,
title count, snapshot/retrieval information, source URL, license attribution,
and normalization changes. Its title count is also metadata, not indexed pages.
Status also reports `extraction` availability, `indexedOnly: true`, and its
default/maximum text limits, plus `research` availability and extractive limits.
Check `enabled` before submitting `discover: true`. The key's contents and
decrypted query terms are never returned through public job status.
Unavailable discovery is reported in `meta.expansion`; current indexed
results remain available. A job whose encrypted context cannot be opened
fails with `discovery_context_unavailable`, rather than silently changing
into a different crawl.

## Crawler policies

The crawler checks public DNS addresses and pins the checked address into the
TLS connection. It refuses IP literals, private addresses, credentials, custom
ports, query strings, login/cart paths, and download links. It identifies as
`VitnaSearch`, honors robots allow/disallow rules, `Content-Signal: search=no`,
noindex, and nofollow. Newly forbidden indexed pages are removed when the
publisher's robots policy is refreshed. Other-origin robots redirects fail
closed. HTTP 429/503 Retry-After pauses that origin; exhausted jobs require:

```bash
npm run search:web -- --retry-failed
```

Publisher sitemaps, RSS/Atom feeds, and `llms.txt` add scoped same-origin page
URLs. Nested sitemap indexes and gzip files share the origin's six-file budget;
cycles are skipped and nesting stops at two levels. XML entity declarations
are rejected. HTML-advertised feeds are fetched only when following links is
allowed. Feed pages, seed pages, and imported archive URLs receive a 24-hour refresh interval; the
queue and crawl budget can delay that further. Optional feed failures cannot
discard an already fetched page. Discovery URLs are enqueued in batches.
The crawler extracts HTML, plain text, and Markdown. JavaScript rendering, PDFs,
model reranking, spam scoring, image/news search, and generated answers are
outside this beta. Semantic embeddings are an optional
[deployment](#semantic-retrieval-deployment) on Vitna's own engine. Weekly refresh is a minimum interval,
not a freshness guarantee: queue size, failures, and caps can delay it further.

## API and production accounting

Run the API with `VITNA_SEARCH_MODE=web`, `VITNA_SEARCH_PROVIDERS=vitna`, and
dedicated client hashes in `VITNA_SEARCH_CLIENTS`. `search:key` creates a token
and its hash; deliver the token privately and place only the hash in the
service's private environment. See the [API contract](overview.md).

Use three different files, never the site's database:

```dotenv
VITNA_SEARCH_INDEX=/var/lib/vitna-search/web/index.sqlite
VITNA_SEARCH_FRONTIER=/var/lib/vitna-search/web/frontier.sqlite
VITNA_SEARCH_QUOTA=/var/lib/vitna-search/web/quotas.sqlite
```

Setting `VITNA_SEARCH_QUOTA` enables persistent per-client minute/day limits
across API restarts. Failed accounting refuses the search. Authenticated
`GET /v1/status` reports index coverage, queue/usage summaries, and
`quotaStorage`. Receipts omit queries and credentials and remain best-effort
logs, not an invoice ledger. Signup, customer billing, and self-service keys
still need implementation before a commercial public launch.

For a URL returned by search, `POST /v1/extract` returns the cleaned text
already stored in the index, with source metadata and observed crawl times.
It uses the same bearer key and search quotas. Optional `maxCharacters`
defaults to 20,000 and is capped at 100,000. It performs no live fetch or
JavaScript execution and does not promise Markdown structure. An absent or
currently policy-unavailable page returns `page_not_indexed` with HTTP 404.
See [indexed text extraction](overview.md#indexed-text-extraction) for the
request and response contract.

`POST /v1/research` provides Source brief: cited verbatim passages from selected
indexed pages, with actual page titles, source metadata, search results, and
bounded evidence counts. It defaults to three sources and 8,000 Unicode code
points of distinct quoted passages, capped at five sources and 16,000 code
points. Repeated overview quotes are not counted twice. It accepts native
filters and bounded expansion choices, with no paid provider fields. Coverage
is limited to matched indexed evidence and is never exhaustive; a brief is
not a generated explanation or a complete review. See
[Source brief research](overview.md#source-brief-research) for its contract.

Local searches accept `domains`, `excludeDomains` (1–10 public hostnames each),
and `maxAgeDays` (1–3,650). Domain filters include subdomains; crawl-age filters
use observed crawl times, not publication dates. `startPublishedDate` and
`endPublishedDate` filter on a page's [declared publication date](overview.md#declared-publication-dates)
and match only pages that declare one. Filtered searches require
the Vitna index with paid fallback disabled. For example:

```json
{"query":"solar system","mode":"index","maxResults":5,"domains":["nasa.gov"],"maxAgeDays":14}
```

Keyword retrieval weights titles, drops common query words when other words
remain, ranks title phrases alongside complete matches in any word order, and suppresses
exact duplicate extracted content. It falls back to partial term matches only
when there are no complete matches within the selected filters. That fallback
matches any of the query's terms, rarest first, while their page counts sum to
at most 200,000; the rarest term always counts. On an index of crawled pages
every term counts, as before; on millions of imported articles the commonest
terms drop out, because an any-term match over them ranked most of the index.
Empty results mean there was no match in this index. Search content is
untrusted source data.

Every result list, keyword, semantic or hybrid, spreads its hosts. Among the
first 20 candidates, at most two pages from one host lead (`www.` aside), and
that host's further candidates follow the other sites' within those 20.
Candidates below the first 20 keep their order. Nothing is dropped, so a
search confined to one site still fills its page. On the production index
(49,063 pages), the spread raised expected-domain hits in the top five from
131 to 135 of the 200 probes for hybrid retrieval, from 134 to 140 for
semantic and from 108 to 117 for keyword. NQ-open answer recall stayed within
noise in every mode; hybrid answered 227 of 1,000 questions in its top ten,
against 225. On a staging index taking in Wikipedia, where `en.wikipedia.org`
held 7.6 of an average keyword top ten, keyword hits rose from 96 to 109, and
6 top-ten answers were lost for 1 gained. Spreading every row a search reads
(200 for a ten-result page) reached 113 hits there, but it lost 27 top-ten
answers for 5 gained, by promoting weak matches from deep in the list.

In web mode, `VITNA_SEARCH_QUERY_WORKERS=2` serves SQLite retrieval on separate
worker threads. The request queue is bounded at 32; its default query deadline
is 10 seconds, including queue time. A cancelled request is answered at once,
but its worker finishes the query and then serves the next one, since a
replacement would reload every vector. A worker still busy at the deadline,
cancelled or not, is terminated and replaced. `VITNA_SEARCH_TIMEOUT_MS` sets
that deadline.
At the API, the same setting is one whole-request search deadline. Discovery
and expansion run separately within their job budgets; waiting in the crawl
queue can take longer than either work deadline.
The HTTP process remains able to serve health checks while retrieval runs.
Worker status failures return service unavailable.

`VITNA_SEARCH_CACHE_MS=30000` caches up to 256 result sets in memory for 30
seconds. Domain filters form part of the cache key. Crawl-age queries bypass
it. Database/WAL metadata invalidates cached content, with up to 250 ms of
metadata staleness. No customer query is written to a cache file or receipt.
Set the cache duration to zero for uncached measurement.

## Measure coverage before expanding access

```bash
npm run search:evaluate
```

This runs 200 authored English queries across 20 topics and saves
`.search-data/reports/coverage.json`. `VITNA_SEARCH_INDEX` selects the database;
`VITNA_SEARCH_EVALUATION_REPORT` selects a different report path. The report
lists URLs, latency, nonempty results, and expected-domain hits in the top five.
Expected domains are coverage probes, not human relevance judgments. Partial
keyword matches can produce a result for an unrelated intent, so a high
nonempty rate alone says little about usefulness. Manually judge the returned
URLs and snippets before opening customer access.

The first 24-page baseline returned a result for 182/200 queries, but only
69/200 had an expected-domain hit. This small corpus is inadequate for a
general search launch. The later 65-page baseline returned 193 nonempty results
and 77 expected-domain hits. Neither result establishes relevance. The first
expansion milestone is 100,000 real documents
with balanced coverage across these topics, measured alongside failed queries,
freshness, and an HTTP test on the actual OVH instance. The profile's document
cap is a target limit, not indexed coverage.

With [semantic retrieval](overview.md#semantic-retrieval) configured, compare
the modes side by side:

```bash
npm run search:evaluate -- --retrieval keyword,semantic,hybrid
npm run search:evaluate -- --fixtures
```

Compare full-workflow accuracy, citation validity, information density, latency, failure rate, and workflow cost across Vitna and external providers under shared result and context budgets:

```bash
npm run search:provider-benchmark -- --offline-demo
```

Pass `--providers vitna,tavily,exa,parallel,perplexity,brave` with API keys in the environment (or `VITNA_SEARCH_URL` and `VITNA_SEARCH_KEY`) to measure live providers against `platform/search/provider-benchmark-fixture.json`.

The first runs the same 200 probes once per mode through the query pool the
API uses, against the index and its vectors, and saves
`.search-data/reports/retrieval-comparison.json`. It reports nonempty results,
expected-domain hits at one and five, expected-domain reciprocal rank, latency,
and every query that fell back to keyword retrieval; any fallback fails the
run. The second needs no index: it embeds the three authored judged corpora
into temporary vectors and reports nDCG@5, recall, reciprocal rank and
negative-query accuracy per mode, in `.search-data/reports/retrieval-fixtures.json`.

On 2026-10-06 these ran against a local 816-page index from a focused,
robots-respecting crawl of 21 sites, 15 of them evaluation publishers and 6
distractors (Wikipedia, Our World in Data and others). NOAA, the Smithsonian,
the Met, the Education and Transportation departments and the USDA refused or
failed the crawl, so the weather and art topics had no expected-domain page.
Pages were
embedded with `bge-small-en-v1.5` on the vitna-anchor engine (three threads,
one page of 816 retried shorter) and queries carried the BGE query prefix:

| Mode | Hits at 5 | Hits at 1 | Reciprocal rank | Expected-domain results | Median / 95th percentile |
|---|---:|---:|---:|---:|---:|
| `keyword` | 135 | 108 | 0.598 | 440 | 3.6 / 14.3 ms |
| `semantic` | 153 | 112 | 0.636 | 549 | 12.1 / 15.4 ms |
| `hybrid` | 155 | 122 | 0.669 | 537 | 15.0 / 25.8 ms |

Latency is per search through the query pool, query embedding included, at
816 pages. All 200 queries returned results in every mode and none fell back.
Without the query prefix, `semantic` scored 155, 112 and 0.641 and `hybrid` 157, 121 and
0.676. The authored judged suites gave nDCG@5 / negative-query accuracy:

| Suite | `keyword` | `semantic` | `hybrid` |
|---|---:|---:|---:|
| Challenge suite, 33 pages, 24 queries | 0.796 / 0.33 | 0.904 / 0 | 0.918 / 0 |
| Held-out relevance, 30 pages, 30 queries | 0.779 / 0.08 | 0.979 / 0 | 0.979 / 0 |
| Semantic held-out, 32 pages, 20 queries | 0.628 / 0.13 | 0.925 / 0 | 0.925 / 0 |

Hybrid lifts both rank measures on the coverage probes, and semantic and
hybrid lead on the judged suites, but neither ever returns an empty result, so
every negative query is answered with the nearest pages. These are small local
corpora, a domain proxy and authored judgments, measured once; they do not
establish production relevance. Hybrid's tie order, embedding first, was
chosen from these runs: keyword first scored 154, 117 and 0.657 on the
probes and 0.779 on the held-out relevance suite.

### Answer recall on real Google questions

The probes above measure domains, not answers. NQ-open's development set
holds 3,610 real Google search questions with accepted answer strings, from
Google's [Natural Questions](https://github.com/google-research-datasets/natural-questions)
(CC BY-SA 3.0). The file is fetched once, checked against its pinned SHA-256,
and never committed:

```bash
npm run search:answers -- --fetch --retrieval keyword,semantic,hybrid
```

For each question it records the first of the top ten results whose stored
text contains an accepted answer as consecutive words (top-k answer recall,
as in open-domain QA), and whether any searchable page contains one at all,
which tells a ranking miss from a coverage gap. A short answer such as "one"
also matches unrelated pages; the metric is standard and noisy in that way.
The report, `.search-data/reports/answers.json`, keeps every question's
results.

The baseline ran on 2026-10-06 against a copy of the production index, 49,063
pages and their vectors; two questions whose answers hold no word were left
out:

| Mode | Top 1 | Top 5 | Top 10 | Median / 95th percentile |
|---|---:|---:|---:|---:|
| `keyword` | 7.7% | 13.6% | 16.1% | 115 / 276 ms |
| `semantic` | 7.3% | 15.3% | 20.1% | 41 / 47 ms |
| `hybrid` | 7.4% | 16.9% | 20.9% | 195 / 389 ms |

Some page held an answer for 55.3% of questions. On the 2,980 questions whose
answers are all distinctive (two or more words, or one non-numeric word of five
or more letters), no page held one for 53.9%, and hybrid missed an indexed
answer for another 30.8%: the gap is coverage first. The index's sources are
not the reference sites these questions need; `en.wikipedia.org` answered 16
questions in hybrid's top ten, held to 2,000 pages by the per-origin cap (an
[import of all of it](#wikipedia-at-full-scale) is measured below). For
scale, the [DPR paper](https://arxiv.org/abs/2004.04906) (retrieved 2026-10-06)
reports 59.1% (BM25) and 78.4% (dense) top-20 answer recall on these 3,610
questions over all of Wikipedia, split into 21 million 100-word passages:
a reference point, not a like-for-like comparison, since those count passages
and these count pages.

Keyword retrieval now builds excerpts only for the results it returns. It
used to run FTS5's `snippet()` on every candidate row, up to twenty times the
results kept, and on long pages that was most of a search: over 900 keyword
searches from these questions against a 70,000-page staging index that was
taking in Wikipedia, median and 95th-percentile time fell from 98 and 788 ms to
32 and 430 ms, with identical results, excerpts included. Most of what remains
at the 95th percentile is ranking the fallback match of any query word across
tens of thousands of pages.

### Wikipedia at full scale

On 2026-10-17 the [importer](#import-english-wikipedia) loaded snapshot
20261001 into a staging index on a laptop (Snapdragon X Elite, 12 cores,
32 GB, NVMe), not on the OVH host. The staging index holds Wikipedia alone:
no copy of the production index was at hand, so these figures describe the
imported corpus, not the production index with it, where the
[host spread](#api-and-production-accounting) lets at most two Wikipedia
pages lead each result list.

| Import, snapshot 20261001 | |
|---|---:|
| Dump parts / compressed size | 71 / 26.9 GB |
| Pages read | 25,932,219 |
| Redirects / other namespaces skipped | 12,065,403 / 6,617,717 |
| Disambiguation pages / under 40 words skipped | 375,002 / 843,093 |
| Articles imported | 6,030,988 |
| Index on disk | 39.9 GB, 6.6 KB per article |
| Wall time, 8 decode threads, parts on disk | 110 minutes |
| Longest batch commit | 9.8 s |

The parts downloaded from dumps.wikimedia.org at 5 to 20 MB/s on one
connection, about 25 to 90 minutes for all 71. The wall time includes a
10-minute pass that, on a first import, found nothing to remove (it is now
skipped when the source held no pages), and it ran beside other work on the
laptop. The JavaScript bzip2 decoder produced the same bytes as the system
`bzip2` for a whole 1.13 GB part, at 14 MB/s per thread. In 4,999 sampled
articles of this index no template, reference or bold markup was left, and
stray link brackets or tags in at most 3; 54 (1.1%) kept the rows of a table
that a template had opened (`{{Awards table}}`), which the converter has
dropped since this import, as it drops other tables. An earlier converter also
leaked tables indented as `: {|` and erased about 0.2% of articles at an
unclosed `{{`; both were fixed before this import. The
longest commit is an FTS5 segment merge inside one 1,000-page batch: a crawler
writing beside the import waits behind it, and a write that waits past its
2-second busy timeout fails and is retried later as a failed fetch. Pausing
the crawl timer for the import avoids that.

Answer recall on NQ-open's 3,608 questions (keyword retrieval through the
query pool, two workers, cache off), against the production baseline above:

| Index | Top 1 | Top 5 | Top 10 | Some page holds an answer | Median / 95th percentile |
|---|---:|---:|---:|---:|---:|
| Production, 49,063 pages, `keyword` (2026-10-06) | 7.7% | 13.6% | 16.1% | 55.3% | 115 / 276 ms |
| Production, `hybrid` (2026-10-06) | 7.4% | 16.9% | 20.9% | 55.3% | 195 / 389 ms |
| Staging, 6,030,988 Wikipedia articles, `keyword` | 37.2% | 60.8% | 67.7% | 98.1% | 294 / 719 ms |

On the 2,980 questions whose answers are all distinctive, no page held one for
2.3% (53.9% on production), and keyword retrieval missed an indexed answer in
its top ten for 34.5%: with Wikipedia in, the gap is ranking, which is what
semantic and hybrid retrieval address. Every answer in a top ten came from
`en.wikipedia.org`. The title weighting can lead with a narrower page than the
article a query names: `solar system` led with "List of largest craters in the
Solar System". The 200 coverage probes all returned results (median 121 ms,
95th percentile 404 ms), with no expected-domain hit, since the staging index
holds none of the probes' publishers.

Keyword retrieval needed three changes at this size, each kept identical in
its results on small indexes. Status read every row: with the code on `main`
it took 167 s on 2.8 million imported pages and 13.9 minutes on 6.03 million,
long past any readiness check, and it reads in 0.9 ms now that a bulk source
reports its importer's committed count. Ranking read each match's
crawl time, hash and provenance, which SQLite reaches through the page text's
overflow pages; it now ranks without them, by FTS5 alone when a search has no
domain or age filter, and joins only the best rows. On 2.8 million pages that
cut "history" (853,675 matches) from 5,652 to 512 ms, and the 189 of 200
questions with a complete match returned identical results and excerpts. The
partial-term fallback matched any query word across millions of pages and
took 3 to 52 s on 2.8 million (one question took 7.8 minutes on 6.03
million with `main`'s code, while a vector-index build ran beside it); it now
keeps the rarest words within 200,000 pages, which kept those searches under
1 s and changed their results, the 11 of 200 questions that used it. At 6.03 million pages, single-threaded on a cold
cache, 300 questions took 428 ms median, 1.03 s at the 95th percentile and
1.7 s at most.

Semantic and hybrid answer recall were not measured at this size: they need a
vector for every article, about 1.6 billion tokens of embedding (see
[Vectors for imported Wikipedia](#vectors-for-imported-wikipedia)), which is
the owner's decision. The [approximate index](overview.md#semantic-retrieval)
that serves them was measured instead, against exact brute force, with real
`bge-small-en-v1.5` vectors (title and first 600 characters) of 50,000
articles sampled from the staging index and the 3,610 NQ-open questions as
queries (with the BGE query prefix). Its 256 lists hold about 200 vectors
each:

| Lists scanned / rescored | Recall of the exact top 10 | Median / 95th percentile |
|---|---:|---:|
| 16 / 100 | 86.7% | 2.1 / 4.0 ms |
| 32 / 200 | 93.8% | 3.8 / 6.9 ms |
| 64 / 400 (the defaults) | 97.8% | 6.9 / 12.1 ms |
| 128 / 800 | 99.6% | 14.9 / 21.9 ms |

Exact brute force over the same 50,000 took 24.4 ms median.

At the full corpus's size there were no real vectors to test, so the 50,000
were expanded to 6,030,988 by copying each about 120 times with Gaussian noise
and normalizing, and 200 of the questions were scored against exact neighbours
found by a full pass. The build (4,096 lists) took 7.7 minutes in one thread
and 1.19 GB, and wrote a 392 MB index beside a 12.5 GB vectors file; each query
worker holds about 372 MB of it (exact search would need 9.3 GB per worker and
about 3 s a query). Recall depends on how alike the copies are. With noise of
0.045 per dimension (a copy about 0.75 in cosine from its source):

| Lists scanned / rescored | Recall of the exact top 10 | Median / 95th percentile |
|---|---:|---:|
| 32 / 200 | 74.2% | 18.7 / 29.4 ms |
| 64 / 400 (the defaults) | 83.9% | 55.5 / 84.8 ms |
| 128 / 400 | 85.6% | 95.5 / 141.4 ms |
| 128 / 800 | 90.0% | 107.7 / 157.4 ms |

With noise of 0.02 (copies about 0.93 from their source, so the exact top ten
are near-duplicates that one-bit codes cannot order), the defaults recalled
74.8% at 31.7 ms median and 128 / 800 recalled 85.3%. The copies stand in for
the density of 6 million vectors, not for their structure, so recall on real
Wikipedia vectors at this size remains unmeasured until they exist; scanning
more lists and rescoring more candidates raise it, at the latency shown,
through `VITNA_SEARCH_VECTORS_PROBES` and `VITNA_SEARCH_VECTORS_RESCORE`. Rescoring
reads stored floats from the vectors file, so on the search host it depends on
how much of that file stays in memory beside the index.

## Domain authority from the web graph

Common Crawl's [web graph](https://commoncrawl.org/web-graphs) ranks every
registered domain by harmonic centrality, and by PageRank, over three months of
crawled hyperlinks; release `cc-main-2026-jul-aug-sep` covers 133.2 million
domains and 2.1 billion links. Its domain ranks file is sorted by harmonic
centrality, so the top domains are a prefix of one gzip file:

```bash
npm run search:authority -- --release cc-main-2026-jul-aug-sep --limit 1000000
```

This streams the first lines of the release's `domain-ranks.txt.gz`, stops the
download at the limit, and replaces the contents of its own SQLite file,
`VITNA_SEARCH_AUTHORITY` (default `.search-data/web/authority.sqlite`), in one
transaction, so a failed run leaves the previous release in place. The top
million domains took 26.6 seconds and 49 MB on 2026-10-06. A host takes the
rank of its longest stored suffix (`science.nasa.gov` takes `nasa.gov`'s), so
no public suffix list is needed: the release applied one when it grouped
hosts. The file is an input for ranking and for choosing what to import; the
top of the list is infrastructure (`googleapis.com`, `gstatic.com`,
`cloudflare.com`), which serves no searchable pages. Graph data is free to use
under Common Crawl's [terms](https://commoncrawl.org/terms-of-use).

## Selective archive bootstrap

Common Crawl's [CDXJ index](https://commoncrawl.org/cdxj-index) identifies archived
URLs and the byte ranges of their WARC records. It can bootstrap reviewed
sources; it does not answer keyword searches. The importer accepts a maximum
of 100 captures per plan and only sources from `web-seeds.json`. It never
downloads whole crawl archives.

```bash
npm run search:archive -- --plan --source web-nasa --collection CC-MAIN-2026-39 --limit 20
```

This makes one bounded, metered CDX metadata request and saves a new plan under
`.search-data/archives/`. Inspect its source scope, URLs, capture dates, and
content permissions before running:

```bash
npm run search:archive -- --import .search-data/archives/web-nasa-CC-MAIN-2026-39.json
```

Import checks current publisher robots/search policy, verifies the HTTP range,
gzip size, WARC target/date, and payload digest, and honors captured noindex.
Compressed records are limited to 2 MiB and decompressed records to 3 MiB.
Unsupported encodings and chunked captures fail closed. Every network request
uses the same durable day/month crawl meter; import shares the crawler lease
and document/disk caps. Newer indexed content is retained, repeated captures
are skipped, and each imported URL is queued for a live refresh. Current page
metadata changes are observed during that live refresh.

Archived results carry `contentOrigin: "common_crawl"`, `capturedAt`,
`importedAt`, and archive collection/digest. Their `crawledAt` is null. Age
filters use the original capture time, and status counts live and archived
documents separately. Schema 1 indexes migrate transactionally to schema 2
when opened for writing; existing read-only connections detect that migration.
Keep a verified snapshot before upgrading a production database.

The parser, import policies, metering, and migration passed fixture tests. A
live CDX planning request timed out; no real archive content has been imported
in this workspace. Live crawling remains the demonstrated ingestion path.
Common Crawl access is free, but content retains its owners' rights; review
applicable [terms](https://commoncrawl.org/terms-of-use) and publisher permissions.

## Import English Wikipedia

Live crawling adds pages slowly: at the profile's 600,000 requests a month
and the first OVH run's yield of 104 pages per 1,000 requests, about 60,000 a
month. A bulk import adds all of English Wikipedia, about 7 million articles,
from Wikimedia's official dump:

```bash
npm run search:wikipedia -- --snapshot 20261001
```

The snapshot is a `YYYYMMDD` run listed at
[dumps.wikimedia.org/enwiki](https://dumps.wikimedia.org/enwiki/) whose
`articlesmultistreamdump` job is done. The importer reads that run's
`dumpstatus.json` and fetches its `pages-articles-multistream` parts (71 for
20261001, 27 GB in all) one at a time from the fixed `dumps.wikimedia.org`
URLs it names, resuming a partial part with a range request. Each part is
checked against the size and SHA-1 Wikimedia publishes for it before any row
is read from it, and a mismatch deletes the part and stops the import. A part
is deleted once imported, so the disk holds at most one (about 650 MB);
`--keep-parts` keeps them, `--dir` places them (by default
`wikipedia/enwiki-<snapshot>/` beside the index), `--parts N` stops after N
parts, and `--workers N` sets the decode threads (by default the CPU count
less two, at most 8). Nothing needs installing: the bzip2 decoder is plain
JavaScript and checks every block's CRC.

Worker threads decode the dump's bzip2 streams in parallel and render each
page's wikitext to the plain text the crawler's extractor produces for a page
(`extract.mjs`): the article's prose, with links reduced to their words and
the inline templates that carry prose (dates, unit conversions, language and
formatting wrappers) rendered to their visible text. References, infoboxes,
tables, files, categories, comments and the trailing reference, link and
"See also" sections are dropped. Only main-namespace articles are imported:
redirects, disambiguation pages, other namespaces and pages under the
profile's 40-word minimum are skipped and counted. A page's URL is the one
Wikipedia itself links to (`https://en.wikipedia.org/wiki/Ender%27s_Game`).

Imported articles live in their own source, `wikipedia-enwiki-dump`, with the
CC BY-SA 4.0 reuse policy, and each result carries
`contentOrigin: "wikipedia_dump"`, the snapshot's capture time (when its
stub dump finished; every revision in the dump was current then), the import
time, and `archive: {collection, revision}`; its `contentChangedAt` is the
revision's edit time. See the [result fields](overview.md#response).
Wikipedia's text is reusable under CC BY-SA 4.0 with credit to Wikipedia's
contributors and a link to the article; Wikimedia's
[reuse guidance](https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content)
describes the attribution.

The index writes each batch of pages with its resume state in one transaction,
so an interrupted import (a signal, a crash, a full disk) runs again with the
same command and continues after the last committed batch. Before every batch
the importer checks the profile's free-disk floor (`minFreeDiskBytes`, 10 GiB)
and the source's cap (`maxDocumentsPerBulkSource`, 8,000,000), and stops
cleanly at either. Its downloads are not crawling and are not charged to the
crawl's daily or monthly request meters. It holds its own lease, renewed
every minute, so the crawler and demand jobs keep running beside it; SQLite
serializes their writes, a batch at a time. A batch can hold the write lock
for seconds while the full-text index merges (9.8 s at most in the staging
import below), longer than a crawler write waits, so stop the crawl timer
for the import if failed fetches during it matter. Searches are unaffected.

The crawler leaves imported pages to the import: it neither queues nor fetches
a URL the import holds, and its caps count crawled pages only (see
[Crawl scope and limits](#crawl-scope-and-limits)). A page the crawler fetched
after the snapshot's capture keeps its crawled copy; an older crawled copy
gives way to the dump's. Importing a newer snapshot rewrites the articles
whose text changed, gives unchanged ones the new provenance without touching
the full-text index, adds new ones, and, once every part is in, removes the
articles the snapshot no longer holds (deleted, now a redirect or a
disambiguation page, or under the word floor). An older snapshot than one
already imported is refused. The JSON summary reports pages read, articles,
each skip reason, inserted, changed, unchanged and removed pages, and
`stopReason` (null once the snapshot is complete).

Status reports the source's article count from the import's own committed
count, so status never reads millions of pages; `importedDocuments` totals
them. Vector maintenance passes imported pages over unless
`VITNA_SEARCH_EMBED_BULK=true`; see
[Semantic retrieval deployment](#semantic-retrieval-deployment).

Optional upstream adapters need appropriate commercial rights before resale.
Brave's standard terms prohibit redistribution/resale of results and using its
API to replace the service; an ordinary subscription does not authorize a
public search resale product. See [Brave's terms](https://api-dashboard.search.brave.com/documentation/resources/terms-of-service).

## Hosted beta

The general web beta is deployed at [search.vitna.ai](https://search.vitna.ai),
with an [API reference](https://search.vitna.ai/docs). Search and status require
a separate Vitna Search key. The public page contains no embedded credential.
Customer signup and billing are not enabled; coverage is still limited.

The first scheduled OVH crawl completed 1,000 requests in about 5.4 minutes,
recording 90.3 MB of payload and adding 104 pages for 175 searchable pages.
Its 88 publisher failures were recorded for retries while the bounded run
completed successfully. The next crawl was scheduled five minutes later.
Registered origins and queued URLs are not searchable coverage.

Daily private R2 backups are enabled. A real backup download passed SHA-256
and SQLite integrity checks for the index, frontier and quota snapshots.
Backups expire after 14 days. HTTPS and API-key protection were verified, and
a simulated certificate renewal succeeded through Cloudflare. The crawler
and backup timers run on the server while the operator's PC is asleep.

## Virginia cloud deployment

The Terraform module under `deploy/search-web` prepares one Ubuntu VM in
`US-EAST-VA-1`, OVH's [Virginia region](https://support.us.ovhcloud.com/hc/en-us/articles/360002245164-How-to-create-a-Public-Cloud-instance-and-connect-to-it).
Its default is `b3-32` (8 vCPU / 32 GB), with SSH restricted to the supplied
administrator IPv4 CIDR. Only ports 80/443 are public; the API listens on
loopback. The module installs Node, nginx, and rclone, and stops nginx until
configuration is complete. It does not upload application code or credentials.

The pilot allocation defaults to $600/month and cannot exceed $800, preserving
at least $200 of the supplied $1,000/month OVH budget. An explicit current
all-in `quoted_monthly_usd` must fit that allocation before planning succeeds.
This is a configuration guard, not a cloud billing enforcement service.

At OVH's US hourly rates checked 2026-10-03, `b3-32` compute is approximately
$179.68 for 744 hours. An illustrative 200 GB of allocated local NVMe adds
$25.61. IPv4, taxes, any extra storage, and other services must be added to
the actual project quote. Storage and IPv4 became separate charges on
2026-10-01. Do not use the page's annual commitment price for this hourly
pilot. Source: [OVH US pricing](https://us.ovhcloud.com/public-cloud/prices/).

For dashboard setup, sign in to the credited OVH project, select Virginia,
and inspect its current hourly instance quote and credit eligibility. Use the
generated `cloud-init.yaml` below as the instance's startup configuration.
Both accounts were accessed through their signed-in dashboards on 2026-10-03.
The current beta reuses an existing instance; the Terraform instructions below
are an alternative for a future dedicated VM, not the deployed resource state.

For Terraform setup, connect the credited OVH project using its private
OpenStack RC or cloud configuration through the local environment, rather
than chat or Terraform variables. Confirm the project, available image/flavor, SSH public key and
administrator CIDR, actual allocated disk/IP charges, and credit eligibility.
The [OVH service account guide](https://support.us.ovhcloud.com/hc/en-us/articles/43234629764755-Using-service-accounts-to-connect-to-OpenStack)
describes project authentication. Keep Terraform state private as well.

Copy `terraform.tfvars.example` to a private `terraform.tfvars`, fill its
public-key/CIDR/quote values, then run:

```bash
terraform -chdir=deploy/search-web init
terraform -chdir=deploy/search-web validate
terraform -chdir=deploy/search-web plan -out=beta.tfplan
terraform -chdir=deploy/search-web apply beta.tfplan
```

Only apply an inspected plan for that project within the approved allocation.
The offline Terraform tests verify one Virginia instance, reject overspending,
and reject unrestricted SSH. They do not verify a live account or provision VM
resources. Local instance storage can be lost with the instance; the beta has
no high-availability replica. R2 backups are part of its recovery plan.

Build the deployment package with:

```bash
npm run search:release
```

It writes `.search-data/releases/vitna-search-web.tgz`, a SHA-256 checksum,
and `cloud-init.yaml` for dashboard-based instance creation. The startup
configuration prepares the same runtime/user as Terraform and keeps nginx
stopped until application configuration is complete. The Linux Node runtime,
SQLite FTS5 support, service installation, API authentication, live crawl, and
R2 upload/download restore check have been verified on the actual reused VM.

The local release archive `.search-data/releases/vitna-search-web.tgz` contains
search code, deployment files, and these docs. It excludes data, environment
files, Terraform state, provider caches, and credentials. Transfer it to the
prepared VM through SSH, extract into `/opt/vitna-search`, and run:

```bash
cd /opt/vitna-search
sudo bash deploy/search-web/install-release.sh
sudo /opt/vitna-search-node/bin/node platform/search/mint-key.mjs pilot
```

Fill `/etc/vitna-search/web.env` with the client hash and reviewed settings.
This environment file is readable only by root and the service group. The
install helper preserves existing private settings and does not start services.
Initialize and inspect a small live crawl before enabling its timer:

```bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/web-run.mjs --max-requests 100
sudo systemctl enable --now vitna-search-web.service
curl --fail http://127.0.0.1:8789/readyz
```

Choose the owned search hostname, configure DNS and a valid TLS certificate,
adapt `deploy/vitna-search.nginx.conf.example`, and verify nginx's configuration
before starting it. The hostname in that template is an example, not a created
DNS record. Configure the R2 backup below and perform one successful upload
and restore check before enabling the recurring work:

```bash
sudo systemctl enable --now vitna-search-web-crawl.timer vitna-search-web-backup.timer
```

The crawl timer advances the durable queue five minutes after each run ends;
it does not recrawl every page every five minutes. The backup timer runs daily.
These are VM timers; they do not keep the user's PC crawling while it is asleep.

## Semantic retrieval deployment

Optional; [semantic retrieval](overview.md#semantic-retrieval) describes the
contract. It uses Vitna's own embedding engine, vitna-anchor, which runs on the
search host as two units serving `bge-small-en-v1.5`: `vitna-embed.service`
at `127.0.0.1:8795` embeds queries, and `vitna-embed-batch.service` at
`127.0.0.1:8797` embeds pages for vector maintenance. The engine serves one
request at a time, so with two units a query never waits behind a batch:
during a backfill at 16 pages a batch, a query's embedding took 17 to 21 ms
from the query unit against 0.8 to 1.9 s when it shared the one engine. The
weights are mapped from the same file, so the second unit costs little
memory. Nothing here makes a request to any third party.

The engine binary is built from convexityos/vitna-anchor, gate A14 or later,
for Linux x86-64 (for example `zig cc -target x86_64-linux-gnu.2.35 -O3
-Iinclude src/*.c -o vitna-anchor -lm -lpthread` in its `engine/`), and the
model fetched with that repository's `node scripts/fetch-model.mjs
bge-small-en-v1.5`. The release's installer checks the binary against the
digest given and every model file against the pin, installs both units, and
starts nothing:

```bash
sudo bash deploy/search-web/install-embed.sh /path/to/vitna-anchor <sha256> /path/to/bge-small-en-v1.5
sudo systemctl enable --now vitna-embed.service vitna-embed-batch.service
curl -s http://127.0.0.1:8795/health; curl -s http://127.0.0.1:8797/health
```

It should answer with `"embeddings":true`, `"dimensions":384` and
`"max_tokens":512`. Use the model identifier it reports. Add the settings to
`/etc/vitna-search/web.env`; commented examples are in
`deploy/vitna-search-web.env.example`, and the vectors file defaults to
`/var/lib/vitna-search/web/vectors.sqlite`:

```dotenv
VITNA_SEARCH_EMBED_URL=http://127.0.0.1:8795/v1/embeddings
VITNA_SEARCH_EMBED_BATCH_URL=http://127.0.0.1:8797/v1/embeddings
VITNA_SEARCH_EMBED_BATCH_SIZE=16
VITNA_SEARCH_EMBED_MODEL=bge-small-en-v1.5
VITNA_SEARCH_EMBED_QUERY_PREFIX="Represent this sentence for searching relevant passages: "
VITNA_SEARCH_EMBED_DIMENSIONS=384
```

The query prefix is the one BGE documents for short queries; in the
measurements below, omitting it changed the results by a few queries either
way.

Fill the vectors file before switching the API over. Each run stops at its
budget and the next continues; repeat until the summary reports
`"complete":true`. The first pass embeds every page, later passes only new and
changed pages:

```bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/embed-run.mjs --budget-ms 3600000
```

On a desktop Ryzen 7 3700X with the engine at the production unit's three
threads, embedding averaged about 2,100 tokens per second over pages of about
285 tokens each. At that rate 49,063 pages need roughly two hours of engine
time: about two such runs by hand, or four to five hours through the timer
alone. The search host's rate will differ.

The release installer places `vitna-search-web-embed.service` and `.timer`
without enabling them. Enable the timer, which runs the job ten minutes after
each run ends within an eight-minute budget, then restart the API so it reads
the settings:

```bash
sudo systemctl enable --now vitna-search-web-embed.timer
sudo systemctl restart vitna-search-web.service
```

Authenticated `GET /v1/status` should then report
`retrieval.semantic.state: "ready"` with a `vectors` count near
`index.documents`, and a search should report
`meta.index.retrieval.used: "hybrid"`. Receipts in
`journalctl -u vitna-search-web` show any `fallback` codes.

Each query worker holds its own copy of the vectors inside the API unit's 4 GB
limit, about 160 MB for two workers at 49,063 pages and 384 dimensions. The
engine serves one request at a time, so without `VITNA_SEARCH_EMBED_BATCH_URL`
a search's query embedding waits for any maintenance batch in progress on the
one engine. At three threads, a query arriving during
32-page batches waited 4.0 s median, and 24 of 26 exceeded the 2 s
`VITNA_SEARCH_EMBED_TIMEOUT_MS` default and would have fallen back to keyword
retrieval. During the default 4-page batches the wait was 0.35 s median and
0.51 s at the 95th percentile, none over 2 s, with no loss of throughput
(2,125 against 2,223 tokens per second). On one engine, raise
`VITNA_SEARCH_EMBED_BATCH_SIZE` only if searches can tolerate that wait; with
the batch unit, queries never see it, and 16 pages a batch keeps the batch
engine busy. To turn the feature off, comment out the
settings and restart the API; the vectors file is then ignored. To change the
model, point `VITNA_SEARCH_VECTORS` at a new file, fill it, and restart the
API. The vectors file is part of the R2 bundle, as `vectors.sqlite.gz`. It is
a snapshot of its own, so it can trail or lead the index snapshot slightly;
after a restore the job removes vectors whose page changed identity and embeds
what is missing. Rebuilding it from the index instead took about 1.2 hours at
4 engine threads for 49,063 pages. The backup reads only `web-backup.env`, so
a `VITNA_SEARCH_VECTORS` set for the API must be set there too.

### Vectors for imported Wikipedia

Vector maintenance passes [imported](#import-english-wikipedia) articles over
unless `VITNA_SEARCH_EMBED_BULK=true`, because embedding them is the expensive
part of semantic retrieval over Wikipedia. Each article embeds its title and
leading text up to the 1,500-character recipe, 260 tokens on average over
3,000 sampled articles, so the 6.03 million articles of snapshot 20261001 are
about 1.6 billion tokens: some 8.6 days at the 2,100 tokens per second
measured above for three engine threads, and still days on all eight cores of
the search host, during which the batch engine is fully busy.
Whether and where to spend that is the owner's decision. With the setting on,
crawled pages still go first, the job keeps its memory bounded by comparing
pages and vectors in ranges of 100,000 ids, and each run starts with a pass
over the whole index and vectors file, so a backfill runs best by hand with a
one-hour budget rather than the timer's eight minutes.

Seven million 384-dimension vectors are about 11 GB in `vectors.sqlite`,
far past what a query worker can hold, so build the approximate index once
the backfill is in, with the API's environment, and give each worker room for
it:

```bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/vector-index-run.mjs
```

```dotenv
VITNA_SEARCH_VECTORS_MAX_MB=1024
```

The API reads the bound at start, so restart it after raising the bound; it
picks up a new or rebuilt index on its own. Vectors embedded after a build are
searched exactly beside the index until the next build.

## Private R2 backups and recovery

Create a dedicated private R2 Standard bucket and a bucket-scoped Object Read
& Write credential. Put it in `/etc/vitna-search/rclone.conf`, using
`deploy/vitna-search-rclone.conf.example`. Restrict it to root and the service
group, mode 0640. Keep the existing `vitna-search-r2` remote name or update
`/etc/vitna-search/web-backup.env` to match it and the actual bucket.

Configure a 14-day R2 expiration rule for the `web-backups/` prefix, plus
incomplete multipart cleanup. Remote lifecycle settings require cloud account
access and are not created by this package. The helper keeps two local bundles
after successful upload; failed uploads retain their local files and require
attention before they consume the disk.

Backups need temporary space for a raw SQLite snapshot plus the compressed
file. Since the 2026-10-17 Wikipedia import the index is about 40 GB, so the
snapshot is that large and a bundle should be about 16 GB. A run plans for
its peak (each raw snapshot beside its compressed file and the compressed files
before it, at the compression the last local manifest measured, or one half
without one) and leaves `VITNA_SEARCH_BACKUP_MIN_FREE_GB` free at every step, 20 GiB by
default, twice the crawler's own stop:

- To make room it removes local bundles that R2 already holds, oldest first and
  only as many as it needs. A bundle is marked as held by `uploaded.json`,
  written once its remote manifest lands. A bundle that never finished
  uploading may be the only copy and is never removed for space.
- If the room is still short, the run fails before any snapshot and says how
  much it needed.
- It checks again before each snapshot and while compressing, so a file that
  compresses worse than planned stops the run at the floor, not at a full disk.
- A run that stops before its local manifest removes its own partial bundle.
  One killed outright (the unit's `TimeoutStartSec` is 6 hours) leaves one,
  and the next run removes it once it is 12 hours old.

Each upload may take five minutes plus one second for every 2 MiB, and
`manifest.json` records each file's raw size as `sourceBytes`, which is the
space a restore needs to decompress it.

```bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web-backup.env platform/search/web-backup-run.mjs
npm run search:web:backup
```

The npm alias reads the same settings from its environment when used locally.
Each bundle contains compressed index, frontier, and API quota databases, the
concept catalog when it holds titles, and the semantic vectors file when one
holds vectors (beside the index unless `VITNA_SEARCH_VECTORS` names another),
plus SHA-256 hashes in `manifest.json`. A file at the vectors path that belongs
to another database stops the backup before any snapshot. The run's JSON line
lists the snapshots taken. SQLite's snapshot operation and integrity
check run before upload. The remote manifest uploads last as a completion
marker. A prefix without a manifest is an incomplete remote backup.
Discovery contexts are encrypted inside the frontier snapshot. The separate
discovery key is not included in the database bundle; keep a private recovery
copy if pending jobs must survive restoration, and restore the same key to
all three service processes before resuming them.

For recovery, stop the API, crawler service/timer, embed timer, and backup
service/timer. Download one complete manifest and every file it lists into a
new private staging directory. Verify compressed byte counts and SHA-256
hashes, decompress them, and run SQLite integrity checks before replacing any
data. Preserve the old files for rollback and set restored ownership/modes for
`vitna-search`. A bundle without `vectors.sqlite.gz` predates the vectors
backup: leave the vectors file absent and the embed job rebuilds it.

The snapshots are individually consistent, not one transaction across
databases. Queue leases expire and interrupted work may be retried; reconcile
the queue against the index after restoration. Restoring older quota or crawl
meters can lose usage after the snapshot. Reconcile those counters from saved
status/receipts before resuming; if usage cannot be recovered, keep affected
work paused until its day/month limit resets. Verify `/readyz`, an authenticated
search/status, and the new usage values before restarting timers.

R2 Standard storage is $0.015/GB-month, with a 10 GB-month free storage tier and
free Internet egress; operation charges also apply. Source: [R2 pricing](https://developers.cloudflare.com/r2/pricing/).
The supplied $10,000 Cloudflare credit award needs its actual eligible products
and expiry checked in the account. This implementation assumes no credit
balance, award duration, or free unlimited storage.
