Get started

Run the general web beta

Source

Vitna Search can grow its own index across public websites, keep a durable crawl queue, and serve authenticated searches without a paid search provider. This is a bounded beta with limited coverage. The API is deployed at https://search.vitna.ai on a reused Virginia OVH server, with private R2 backups. No additional OVH instance was purchased.

Index is the browser's default search mode. Searches return the current corpus immediately. Empty name/topic queries and topic queries with only partial-term matches can automatically queue discovery of a few candidate sites, then crawl selected pages into the same index. Domain input is optional. This uses Vitna's own crawler and makes no paid search-engine call. It does not promise whole-web coverage or immediate access to pages that have not been indexed. Source brief can assemble cited, verbatim passages from the matched indexed pages; it does not generate an explanation.

Current evidence

Three local live runs through 2026-10-03 used 460 HTTP requests and recorded about 31.3 MB of response payload. They indexed 65 real pages, registered 250 origins, and left 27,066 URLs pending. Registered origins and queued URLs are discoveries, not searchable coverage. Some publishers refused requests, timed out, or had unavailable robots files; those failures remain recorded for bounded retries.

A staging import of all of English Wikipedia (6,030,988 articles, snapshot 20261001) answered 67.7% of NQ-open's questions in keyword retrieval's top ten, against 16.1% on the production index; it is not deployed. See Wikipedia at full scale.

A separate synthetic test inserted 100,000 documents in 16.2 seconds and measured 79 ms median / 91 ms 95th-percentile retrieval on this PC over 100 queries. Its database was about 454 MB. These measurements concern the keyword index, not web relevance, real page sizes, or OVH performance.

A later local HTTP test against 100,000 synthetic pages completed 100/100 requests with four simultaneous callers, two query workers, and caching off. Median search latency was 314 ms; the 95th percentile was 1,382 ms. Public health checks had a 1.1 ms 95th percentile. This is a short local smoke test, not sustained throughput or a production capacity guarantee.

bash
npm run search:benchmark -- --documents 100000
npm run search:benchmark -- --documents 100000 --http --concurrency 4

--vectors 384 adds one synthetic 384-dimension vector per page. On 2026-10-06, --documents 49063 --vectors 384 --http --concurrency 2 on a desktop Ryzen 7 3700X measured a query worker's full load of the 49,063 vectors at 434 ms and 79.5 MB, the top-40 scan at 24.7 ms median and 25.4 ms at the 95th percentile, and 500 changed vectors applied in 5.3 ms. Over HTTP with two workers, hybrid searches took 119 ms median against 72 ms for keyword, with 95th percentiles of 650 and 564 ms. The benchmark makes its query vectors in process, so those hybrid times exclude the engine call, about 10 ms on an idle engine.

Crawl and preview locally

Node 24 is recommended. The service uses built-in modules and needs neither an npm install nor a website build. Leave at least 10 GiB free on the data disk.

Initialize local databases and a private pilot credential with:

bash
npm run search:setup

This creates .env.search and .search-data/private/pilot.key without printing the token. It preserves an existing environment file and key. Both are excluded from Git and release archives. The new environment selects the Vitna index and loopback port 8791. Start the authenticated local API with:

bash
node --env-file=.env.search platform/search/start.mjs

There is no crawl or cloud deployment during setup. An empty index stays not-ready until a successful crawl. Windows does not enforce POSIX file modes; store these private files in an account-restricted workspace. For the browser preview with its own temporary credential, use:

bash
npm run search:web -- --max-requests 80
npm run search:web:preview

The first command makes real public HTTP requests. Its override can only lower the configured run limit. The second opens a loopback-only service at http://127.0.0.1:8790, with a temporary key usable only by that preview. It searches the real local index and performs no crawl. Its quota database is in memory; production uses the file specified below.

Live runs can exit with a failure status while preserving successfully indexed pages. Read the summary's failed, errors, and stopReason. Requests stopped by a run limit resume from the queue on the next invocation. The preview never creates provider calls or reads the site's database.

Crawl scope and limits

The starter platform/search/web-seeds.json contains 28 public sites spanning science, health, books, parks, government, art, economics, and software. Links can add other public HTTPS origins within the configured cap. Coverage and permission are evaluated as each origin is visited; a seed is not a promise that its site will be indexed. Pages in languages other than English can enter through discovered links.

platform/search/web-profile.json sets these defaults:

ControlDefault
Total searchable documents100,000
Documents per origin2,000
Documents per bulk-imported source (a Wikipedia dump)8,000,000
Registered origins1,000
Queue entries250,000
Concurrent origin jobs4
HTTP requests per run1,000
Jobs per run / maximum duration4,000 / 15 minutes
Requests per UTC day / month25,000 / 600,000
Metered payload per UTC day / month4 GiB / 120 GiB
Daily/monthly request and payload capacity reserved for demand work20%
Raw and decompressed page size1 MiB each
Raw and decompressed discovery file size2 MiB each
Sitemap files per origin / nesting depth6 / 2
Advertised feeds per page / feed page refresh2 / 24 hours
Minimum origin spacing1 second, increased for robots rules
Regular page refresh / robots cache7 days / 24 hours
Discovered link depth4
Failed attempts before manual retry5
Minimum extracted words40
Minimum free disk space10 GiB

The queue, origin delays, robots cache, retry state, and day/month meters are stored in SQLite. Every HTTP request, including robots, discovery, and redirects, reserves budget before starting. Successful reads refund unused payload reservation; crashes leave the full reservation charged. These meters bound crawl work and payload, not transport headers, compute charges, or an account-wide cloud bill. Changing the profile does not prune existing queue entries or documents to a smaller cap. Validate a smaller profile against existing counts before activating it.

The first two caps count crawled pages only (Common Crawl imports included). A bulk import fills its own source, which has its own cap, maxDocumentsPerBulkSource, so millions of imported articles neither stop the crawler at its 100,000 nor count against en.wikipedia.org's crawl allocation, and the crawler's caps are no looser than before. The crawler's source sync never trims or removes a bulk source.

Background work stops when shared daily or monthly usage reaches 80% of its request or payload limit, reserving the remainder for user-requested jobs. Both lanes charge the same durable totals; the reserve does not create an additional budget or guarantee immediate crawl capacity.

At a full frontier, background discovery updates known entries without registering new sites that cannot receive queue space. Each run may reclaim up to 64 automatically registered origins that have never been visited and have no queued URL, robots cache, lease, delay, or indexed document. Configured seeds and current demand targets remain protected.

Newly reviewed starting sources also receive bounded queue admission when the frontier is full. Each pass admits at most eight missing seed pages across sources, replacing only untouched discoveries; indexed pages, crawl history, active leases and configured seeds stay protected. The catalog now includes 28 starting sources spanning science, government, health, general reference, travel, programming, reporting and textbooks. These are starting points for the crawl, rather than a claim of complete coverage on those sites.

Query-guided coverage expansion

In configured web mode, an empty native name/topic search or a topic query with only partial-term matches can queue site discovery automatically when no domains or URLs are supplied and external fallback is disabled. A plain request works without domain input or crawl flags:

json
{"query":"examplebrand","mode":"index"}

The browser enables this option by default. Untouched requests omit the expand and discover flags; disabling the option sends both as false. Either explicit expand: false or discover: false prevents automatic discovery. Automatic admissions include meta.expansion.automatic: true. Production meta.index.match can report complete, partial, or none; partial describes partial-term fallback. Discovery can address that weak coverage while leaving the current results intact. The field is optional, and complete term matches do not guarantee that a result answers the query. Name detection is heuristic. Common topic words and multiword topic queries do not generate guessed hostnames. Broad queries instead use publisher description matches from reviewed or observed publisher descriptors. A catalog domain is a discovery candidate, not an indexed document or a promise of access. A query without a suitable directory/name, publisher, or configured concept-title match can return no_candidates.

Automatic name lookup uses Vitna's own directory of names observed on fetched pages; only explicit expansion also matches known URLs or caller-supplied sites. A narrow distinctive single ASCII brand can additionally suggest up to three hosts formed with .com, .ai, and .org. These are unverified candidates, not a claim that any of them is the official site or matches the intended entity.

Topic discovery matches query words against publisher descriptions, then examines at most three selected publishers' homepages and sitemaps before choosing up to eight target pages. It shares the same request, time, policy, and ownership limits as name discovery. The catalog does not contribute search results until the crawler successfully indexes pages.

When configured, a separate local English Wikipedia title catalog can match concepts to candidate article URLs. It contains title metadata, including redirect titles, rather than article bodies or canonical redirect mappings. Each candidate must still pass crawler policy and be fetched and indexed before its text is searchable or available to Source brief. Matching a title does not establish current page availability or whole-web coverage.

Automatic name lookup uses exact directory names and bounded candidates; incidental matches in article addresses do not establish a site identity. Before a candidate enters the index, its HTTPS page must respect robots and noindex, contain useful readable text, and identify the requested name. Parking pages, challenges and unrelated redirects are rejected. A successful check establishes a self-identified content match, rather than verified ownership.

The directory learns public names from indexed homepages, retains observations for 180 days, and stores up to 10,000 sites with eight aliases each. Failed probes are retained as private HMAC digests for at most 24 hours, preventing repeated failed requests without saving guessed names. Both tables live in the frontier database and are covered by the existing backup.

To learn from homepages already in an initialized index, run this optional local command with the same index and frontier environment settings:

sh
npm run search:sites -- --backfill

It inspects at most 1,000 live homepages, respects enabled source scope and cached publisher rules, makes no network requests, and prints aggregate counts.

Callers can also request work explicitly. Current indexed results return first with a separate meta.expansion job. Send expand: true, discover: false for target-only refresh. The service selects up to eight relevant known URLs using words in their addresses, plus supplied urls. domains can supply domain roots when no relevant known URL is available. This job's scope is known_urls_and_supplied_targets.

Add discover: true to examine selected sites' homepages and sitemaps before choosing target pages. It requires expand: true and a configured private discovery key. Optional domains guides site selection; without it, sites come from matching known URLs, supplied URL origins, observed names, narrow brand-name candidates, topic-catalog matches, or configured concept-title matches. Automatic discovery skips incidental known-URL matching. With discovery, callers can provide at most three domains or three distinct supplied URL origins; ordinary native domain filters retain their ten-host limit.

Site-discovery jobs have scope site_discovery. Each examines at most three origins, including robots files, homepages, and up to two sitemap files per site, and considers at most 2,000 sitemap candidates. It selects at most eight target pages within 40 total HTTP requests and 60 seconds of crawler work. Discovery requests and target fetches share those budgets. URL matches can be weak, publishers can refuse crawling, and selected candidates may never be indexed. This is bounded site discovery, not whole-web search.

Jobs are durable in the frontier database, private to the authenticated client, and expire after 24 hours. Admission allows 64 pending jobs in total and four per client. Repeating the same normalized query, filters, discovery choice, and supplied targets returns the same job until expiry. Deduplication uses a private HMAC digest. Bounded lexical and site hints, including names, candidate origins and their provenance, are encrypted in private expiring discovery jobs. The raw original query and full request are not stored or logged. Plaintext terms are not persisted in jobs or receipts. Target URLs remain in the job and normal crawl state.

GET /v1/search/jobs/<id> returns progress with the same bearer key. A completed job means its bounded pass finished; inspect summary.indexed, summary.failed, and summary.demand.unmet for actual coverage. summary.discovery adds integer counters origins, candidates, selected, and unavailable; provisional name jobs also report probed, accepted, and cached. Candidate and selection counts do not mean searchable pages; indexed/unchanged counts and unmet targets describe the crawl outcome. Search again to retrieve newly indexed pages; the browser's completed-job rerun sets expand: false to avoid repeating the work. Polling allows at most 60 checks per minute and does not consume the ordinary search allowance. An unknown, expired, or other client's job returns 404.

Each job uses at most eight target attempts, 40 HTTP requests, and 60 seconds of crawl work. It follows the crawler's existing public-address validation, robots, noindex, origin spacing, free-disk guard, and shared daily/monthly budgets. It does not recursively crawl links or feeds. Ordinary target-only jobs do not fetch sitemaps; site-discovery jobs add only the bounded pass above. The same global index lease protects background and demand work. Never-fetched pending queue records may be replaced to admit explicit targets at the frontier cap; fetched history and active leases are preserved.

The background runner releases its lease between 100-request or one-minute batches and checks demand jobs between batches. The demand timer checks every 30 seconds while the background pass is inactive. A busy crawler, publisher delay, or exhausted budget can postpone work; queued jobs report retry timing.

bash
curl https://search.vitna.ai/v1/search \
  -H "Authorization: Bearer $VITNA_SEARCH_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"query":"lunar science","mode":"index","expand":true,"discover":true,"domains":["nasa.gov"]}'

Set expand: true, discover: false to crawl only known and supplied targets. For name discovery without domains, send expand: true, discover: true. Supplied urls remain public HTTPS page addresses without credentials or query strings and must satisfy domain restrictions.

Run one bounded demand pass manually:

bash
npm run search:demand

The installer includes vitna-search-demand.service and .timer; enable the timer alongside the API and background crawl units. Jobs are included in existing frontier backups. This deployment uses Vitna's own crawler and index.

Initialize the topic catalog

Run these explicit local commands with the existing VITNA_SEARCH_FRONTIER and VITNA_SEARCH_INDEX settings loaded into the process environment:

bash
npm run search:topics -- --init
npm run search:topics -- --review-seeds
npm run search:topics -- --backfill

Without those settings, the defaults are .search-data/web/frontier.sqlite and .search-data/web/index.sqlite. Initialization creates the frontier/catalog schema. Seed review records 28 curated publisher descriptions. Backfill reads at most 1,000 eligible already indexed public homepages, respecting current source scope and cached publisher rules. These commands make no network requests and neither register crawl origins nor start crawling. Startup does not automatically seed or download the catalog.

An optional explicit command downloads a bounded domain-candidate dataset:

bash
npm run search:topics -- --import-majestic --limit 10000

It uses the fixed Majestic Million CSV, retaining 10,000 candidates by default and at most 100,000. Reads are bounded to 100 MiB and 60 seconds. A durable 24-hour cooldown includes failed download attempts; no force flag or alternate URL is accepted. Failed refreshes preserve the previous candidate snapshot. Stored attribution identifies Majestic Million and its CC BY 3.0 Unported license, retrieval information, and normalization to a bounded subset of HTTPS roots. That dataset license does not establish permission to reuse publishers' pages or assets.

Imported candidates remain separate unverified metadata. Importing them does not create publisher descriptions, register origins, enqueue a crawl, or add searchable documents. Topic discovery matches reviewed or observed publisher descriptions; the candidate count does not measure its indexed coverage.

Import a concept-title snapshot

VITNA_SEARCH_CONCEPTS selects a separate SQLite catalog. Web mode defaults to .search-data/web/concepts.sqlite; the API and demand worker must use the same path, distinct from index, frontier, and quota files. Startup performs no title download. Import a chosen published English Wikipedia snapshot explicitly:

bash
npm run search:concepts -- --import-wikipedia --snapshot 20261003

The date is an example published snapshot. Select a valid YYYYMMDD snapshot from the official title dumps. The importer constructs that fixed Wikimedia source URL; it accepts no alternate download URL or prefix-sized title subset. It reads the complete gzip stream, validates title records and database integrity, and publishes a staged replacement only after successful completion. A failed import preserves the existing catalog.

Import boundLimit
Compressed / decoded payload160 MiB / 2 GiB
Title records / database size25 million / 8 GiB
Minimum available disk space12 GiB
Import deadline15 minutes

Original English main-namespace titles and redirect titles are retained with separate normalized lookup keys. There are no redirect-target mappings or article bodies. Status records the snapshot, import time, source URL, payload counts, and contributor/license attribution; consult that attribution and the publisher's policy when reusing material. This catalog cannot supply search results or Source brief passages by itself. Candidate pages must first be fetched and indexed within the crawler's policy and budgets.

The R2 backup described below includes the title catalog once it holds titles, when VITNA_SEARCH_CONCEPTS is set in web-backup.env. Without that setting, preserve it separately or repeat its recorded snapshot import when restoring concept discovery.

Private discovery jobs

Configure VITNA_SEARCH_DEMAND_SECRET_FILE in the shared /etc/vitna-search/web.env, for example:

dotenv
VITNA_SEARCH_DEMAND_SECRET_FILE=/etc/vitna-search/demand.key

Create that private file with a cryptographically random 32-byte key encoded as 64 lowercase hexadecimal characters, with an optional final newline. Use a regular file, not a symbolic link. On Linux, give it to vitna-search with mode 0600; group or public permissions are refused. Use private account ACLs on Windows. Keep the key out of chat, source control, release archives, and receipts. The API, background web crawler, and demand unit must read the same file through their shared environment. The key is needed only for new site discovery; ordinary retrieval and target-only expansion remain available without it.

Authenticated GET /v1/status exposes expansion.siteDiscovery with enabled, automaticNames, automaticTopics and automaticConcepts (true when configured), limits: {origins: 3, candidates: 2000}, and safe name-directory and publisher-catalog aggregate counts. topicCatalog reports source: "local_topic_descriptors", publishers (unexpired descriptors), observed, reviewed, candidates (separate unverified domain metadata), and imports (dataset attribution/retrieval records). These do not count searchable documents. conceptCatalog identifies source: "wikipedia_titles", language: "en", title count, snapshot/retrieval information, source URL, license attribution, and normalization changes. Its title count is also metadata, not indexed pages. Status also reports extraction availability, indexedOnly: true, and its default/maximum text limits, plus research availability and extractive limits. Check enabled before submitting discover: true. The key's contents and decrypted query terms are never returned through public job status. Unavailable discovery is reported in meta.expansion; current indexed results remain available. A job whose encrypted context cannot be opened fails with discovery_context_unavailable, rather than silently changing into a different crawl.

Crawler policies

The crawler checks public DNS addresses and pins the checked address into the TLS connection. It refuses IP literals, private addresses, credentials, custom ports, query strings, login/cart paths, and download links. It identifies as VitnaSearch, honors robots allow/disallow rules, Content-Signal: search=no, noindex, and nofollow. Newly forbidden indexed pages are removed when the publisher's robots policy is refreshed. Other-origin robots redirects fail closed. HTTP 429/503 Retry-After pauses that origin; exhausted jobs require:

bash
npm run search:web -- --retry-failed

Publisher sitemaps, RSS/Atom feeds, and llms.txt add scoped same-origin page URLs. Nested sitemap indexes and gzip files share the origin's six-file budget; cycles are skipped and nesting stops at two levels. XML entity declarations are rejected. HTML-advertised feeds are fetched only when following links is allowed. Feed pages, seed pages, and imported archive URLs receive a 24-hour refresh interval; the queue and crawl budget can delay that further. Optional feed failures cannot discard an already fetched page. Discovery URLs are enqueued in batches. The crawler extracts HTML, plain text, and Markdown. JavaScript rendering, PDFs, model reranking, spam scoring, image/news search, and generated answers are outside this beta. Semantic embeddings are an optional deployment on Vitna's own engine. Weekly refresh is a minimum interval, not a freshness guarantee: queue size, failures, and caps can delay it further.

API and production accounting

Run the API with VITNA_SEARCH_MODE=web, VITNA_SEARCH_PROVIDERS=vitna, and dedicated client hashes in VITNA_SEARCH_CLIENTS. search:key creates a token and its hash; deliver the token privately and place only the hash in the service's private environment. See the API contract.

Use three different files, never the site's database:

dotenv
VITNA_SEARCH_INDEX=/var/lib/vitna-search/web/index.sqlite
VITNA_SEARCH_FRONTIER=/var/lib/vitna-search/web/frontier.sqlite
VITNA_SEARCH_QUOTA=/var/lib/vitna-search/web/quotas.sqlite

Setting VITNA_SEARCH_QUOTA enables persistent per-client minute/day limits across API restarts. Failed accounting refuses the search. Authenticated GET /v1/status reports index coverage, queue/usage summaries, and quotaStorage. Receipts omit queries and credentials and remain best-effort logs, not an invoice ledger. Signup, customer billing, and self-service keys still need implementation before a commercial public launch.

For a URL returned by search, POST /v1/extract returns the cleaned text already stored in the index, with source metadata and observed crawl times. It uses the same bearer key and search quotas. Optional maxCharacters defaults to 20,000 and is capped at 100,000. It performs no live fetch or JavaScript execution and does not promise Markdown structure. An absent or currently policy-unavailable page returns page_not_indexed with HTTP 404. See indexed text extraction for the request and response contract.

POST /v1/research provides Source brief: cited verbatim passages from selected indexed pages, with actual page titles, source metadata, search results, and bounded evidence counts. It defaults to three sources and 8,000 Unicode code points of distinct quoted passages, capped at five sources and 16,000 code points. Repeated overview quotes are not counted twice. It accepts native filters and bounded expansion choices, with no paid provider fields. Coverage is limited to matched indexed evidence and is never exhaustive; a brief is not a generated explanation or a complete review. See Source brief research for its contract.

Local searches accept domains, excludeDomains (1–10 public hostnames each), and maxAgeDays (1–3,650). Domain filters include subdomains; crawl-age filters use observed crawl times, not publication dates. startPublishedDate and endPublishedDate filter on a page's declared publication date and match only pages that declare one. Filtered searches require the Vitna index with paid fallback disabled. For example:

json
{"query":"solar system","mode":"index","maxResults":5,"domains":["nasa.gov"],"maxAgeDays":14}

Keyword retrieval weights titles, drops common query words when other words remain, ranks title phrases alongside complete matches in any word order, and suppresses exact duplicate extracted content. It falls back to partial term matches only when there are no complete matches within the selected filters. That fallback matches any of the query's terms, rarest first, while their page counts sum to at most 200,000; the rarest term always counts. On an index of crawled pages every term counts, as before; on millions of imported articles the commonest terms drop out, because an any-term match over them ranked most of the index. Empty results mean there was no match in this index. Search content is untrusted source data.

Every result list, keyword, semantic or hybrid, spreads its hosts. Among the first 20 candidates, at most two pages from one host lead (www. aside), and that host's further candidates follow the other sites' within those 20. Candidates below the first 20 keep their order. Nothing is dropped, so a search confined to one site still fills its page. On the production index (49,063 pages), the spread raised expected-domain hits in the top five from 131 to 135 of the 200 probes for hybrid retrieval, from 134 to 140 for semantic and from 108 to 117 for keyword. NQ-open answer recall stayed within noise in every mode; hybrid answered 227 of 1,000 questions in its top ten, against 225. On a staging index taking in Wikipedia, where en.wikipedia.org held 7.6 of an average keyword top ten, keyword hits rose from 96 to 109, and 6 top-ten answers were lost for 1 gained. Spreading every row a search reads (200 for a ten-result page) reached 113 hits there, but it lost 27 top-ten answers for 5 gained, by promoting weak matches from deep in the list.

In web mode, VITNA_SEARCH_QUERY_WORKERS=2 serves SQLite retrieval on separate worker threads. The request queue is bounded at 32; its default query deadline is 10 seconds, including queue time. A cancelled request is answered at once, but its worker finishes the query and then serves the next one, since a replacement would reload every vector. A worker still busy at the deadline, cancelled or not, is terminated and replaced. VITNA_SEARCH_TIMEOUT_MS sets that deadline. At the API, the same setting is one whole-request search deadline. Discovery and expansion run separately within their job budgets; waiting in the crawl queue can take longer than either work deadline. The HTTP process remains able to serve health checks while retrieval runs. Worker status failures return service unavailable.

VITNA_SEARCH_CACHE_MS=30000 caches up to 256 result sets in memory for 30 seconds. Domain filters form part of the cache key. Crawl-age queries bypass it. Database/WAL metadata invalidates cached content, with up to 250 ms of metadata staleness. No customer query is written to a cache file or receipt. Set the cache duration to zero for uncached measurement.

Measure coverage before expanding access

bash
npm run search:evaluate

This runs 200 authored English queries across 20 topics and saves .search-data/reports/coverage.json. VITNA_SEARCH_INDEX selects the database; VITNA_SEARCH_EVALUATION_REPORT selects a different report path. The report lists URLs, latency, nonempty results, and expected-domain hits in the top five. Expected domains are coverage probes, not human relevance judgments. Partial keyword matches can produce a result for an unrelated intent, so a high nonempty rate alone says little about usefulness. Manually judge the returned URLs and snippets before opening customer access.

The first 24-page baseline returned a result for 182/200 queries, but only 69/200 had an expected-domain hit. This small corpus is inadequate for a general search launch. The later 65-page baseline returned 193 nonempty results and 77 expected-domain hits. Neither result establishes relevance. The first expansion milestone is 100,000 real documents with balanced coverage across these topics, measured alongside failed queries, freshness, and an HTTP test on the actual OVH instance. The profile's document cap is a target limit, not indexed coverage.

With semantic retrieval configured, compare the modes side by side:

bash
npm run search:evaluate -- --retrieval keyword,semantic,hybrid
npm run search:evaluate -- --fixtures

Compare full-workflow accuracy, citation validity, information density, latency, failure rate, and workflow cost across Vitna and external providers under shared result and context budgets:

bash
npm run search:provider-benchmark -- --offline-demo

Pass --providers vitna,tavily,exa,parallel,perplexity,brave with API keys in the environment (or VITNA_SEARCH_URL and VITNA_SEARCH_KEY) to measure live providers against platform/search/provider-benchmark-fixture.json.

The first runs the same 200 probes once per mode through the query pool the API uses, against the index and its vectors, and saves .search-data/reports/retrieval-comparison.json. It reports nonempty results, expected-domain hits at one and five, expected-domain reciprocal rank, latency, and every query that fell back to keyword retrieval; any fallback fails the run. The second needs no index: it embeds the three authored judged corpora into temporary vectors and reports nDCG@5, recall, reciprocal rank and negative-query accuracy per mode, in .search-data/reports/retrieval-fixtures.json.

On 2026-10-06 these ran against a local 816-page index from a focused, robots-respecting crawl of 21 sites, 15 of them evaluation publishers and 6 distractors (Wikipedia, Our World in Data and others). NOAA, the Smithsonian, the Met, the Education and Transportation departments and the USDA refused or failed the crawl, so the weather and art topics had no expected-domain page. Pages were embedded with bge-small-en-v1.5 on the vitna-anchor engine (three threads, one page of 816 retried shorter) and queries carried the BGE query prefix:

ModeHits at 5Hits at 1Reciprocal rankExpected-domain resultsMedian / 95th percentile
keyword1351080.5984403.6 / 14.3 ms
semantic1531120.63654912.1 / 15.4 ms
hybrid1551220.66953715.0 / 25.8 ms

Latency is per search through the query pool, query embedding included, at 816 pages. All 200 queries returned results in every mode and none fell back. Without the query prefix, semantic scored 155, 112 and 0.641 and hybrid 157, 121 and 0.676. The authored judged suites gave nDCG@5 / negative-query accuracy:

Suitekeywordsemantichybrid
Challenge suite, 33 pages, 24 queries0.796 / 0.330.904 / 00.918 / 0
Held-out relevance, 30 pages, 30 queries0.779 / 0.080.979 / 00.979 / 0
Semantic held-out, 32 pages, 20 queries0.628 / 0.130.925 / 00.925 / 0

Hybrid lifts both rank measures on the coverage probes, and semantic and hybrid lead on the judged suites, but neither ever returns an empty result, so every negative query is answered with the nearest pages. These are small local corpora, a domain proxy and authored judgments, measured once; they do not establish production relevance. Hybrid's tie order, embedding first, was chosen from these runs: keyword first scored 154, 117 and 0.657 on the probes and 0.779 on the held-out relevance suite.

Answer recall on real Google questions

The probes above measure domains, not answers. NQ-open's development set holds 3,610 real Google search questions with accepted answer strings, from Google's Natural Questions (CC BY-SA 3.0). The file is fetched once, checked against its pinned SHA-256, and never committed:

bash
npm run search:answers -- --fetch --retrieval keyword,semantic,hybrid

For each question it records the first of the top ten results whose stored text contains an accepted answer as consecutive words (top-k answer recall, as in open-domain QA), and whether any searchable page contains one at all, which tells a ranking miss from a coverage gap. A short answer such as "one" also matches unrelated pages; the metric is standard and noisy in that way. The report, .search-data/reports/answers.json, keeps every question's results.

The baseline ran on 2026-10-06 against a copy of the production index, 49,063 pages and their vectors; two questions whose answers hold no word were left out:

ModeTop 1Top 5Top 10Median / 95th percentile
keyword7.7%13.6%16.1%115 / 276 ms
semantic7.3%15.3%20.1%41 / 47 ms
hybrid7.4%16.9%20.9%195 / 389 ms

Some page held an answer for 55.3% of questions. On the 2,980 questions whose answers are all distinctive (two or more words, or one non-numeric word of five or more letters), no page held one for 53.9%, and hybrid missed an indexed answer for another 30.8%: the gap is coverage first. The index's sources are not the reference sites these questions need; en.wikipedia.org answered 16 questions in hybrid's top ten, held to 2,000 pages by the per-origin cap (an import of all of it is measured below). For scale, the DPR paper (retrieved 2026-10-06) reports 59.1% (BM25) and 78.4% (dense) top-20 answer recall on these 3,610 questions over all of Wikipedia, split into 21 million 100-word passages: a reference point, not a like-for-like comparison, since those count passages and these count pages.

Keyword retrieval now builds excerpts only for the results it returns. It used to run FTS5's snippet() on every candidate row, up to twenty times the results kept, and on long pages that was most of a search: over 900 keyword searches from these questions against a 70,000-page staging index that was taking in Wikipedia, median and 95th-percentile time fell from 98 and 788 ms to 32 and 430 ms, with identical results, excerpts included. Most of what remains at the 95th percentile is ranking the fallback match of any query word across tens of thousands of pages.

Wikipedia at full scale

On 2026-10-17 the importer loaded snapshot 20261001 into a staging index on a laptop (Snapdragon X Elite, 12 cores, 32 GB, NVMe), not on the OVH host. The staging index holds Wikipedia alone: no copy of the production index was at hand, so these figures describe the imported corpus, not the production index with it, where the host spread lets at most two Wikipedia pages lead each result list.

Import, snapshot 20261001
Dump parts / compressed size71 / 26.9 GB
Pages read25,932,219
Redirects / other namespaces skipped12,065,403 / 6,617,717
Disambiguation pages / under 40 words skipped375,002 / 843,093
Articles imported6,030,988
Index on disk39.9 GB, 6.6 KB per article
Wall time, 8 decode threads, parts on disk110 minutes
Longest batch commit9.8 s

The parts downloaded from dumps.wikimedia.org at 5 to 20 MB/s on one connection, about 25 to 90 minutes for all 71. The wall time includes a 10-minute pass that, on a first import, found nothing to remove (it is now skipped when the source held no pages), and it ran beside other work on the laptop. The JavaScript bzip2 decoder produced the same bytes as the system bzip2 for a whole 1.13 GB part, at 14 MB/s per thread. In 4,999 sampled articles of this index no template, reference or bold markup was left, and stray link brackets or tags in at most 3; 54 (1.1%) kept the rows of a table that a template had opened ({{Awards table}}), which the converter has dropped since this import, as it drops other tables. An earlier converter also leaked tables indented as : {| and erased about 0.2% of articles at an unclosed {{; both were fixed before this import. The longest commit is an FTS5 segment merge inside one 1,000-page batch: a crawler writing beside the import waits behind it, and a write that waits past its 2-second busy timeout fails and is retried later as a failed fetch. Pausing the crawl timer for the import avoids that.

Answer recall on NQ-open's 3,608 questions (keyword retrieval through the query pool, two workers, cache off), against the production baseline above:

IndexTop 1Top 5Top 10Some page holds an answerMedian / 95th percentile
Production, 49,063 pages, keyword (2026-10-06)7.7%13.6%16.1%55.3%115 / 276 ms
Production, hybrid (2026-10-06)7.4%16.9%20.9%55.3%195 / 389 ms
Staging, 6,030,988 Wikipedia articles, keyword37.2%60.8%67.7%98.1%294 / 719 ms

On the 2,980 questions whose answers are all distinctive, no page held one for 2.3% (53.9% on production), and keyword retrieval missed an indexed answer in its top ten for 34.5%: with Wikipedia in, the gap is ranking, which is what semantic and hybrid retrieval address. Every answer in a top ten came from en.wikipedia.org. The title weighting can lead with a narrower page than the article a query names: solar system led with "List of largest craters in the Solar System". The 200 coverage probes all returned results (median 121 ms, 95th percentile 404 ms), with no expected-domain hit, since the staging index holds none of the probes' publishers.

Keyword retrieval needed three changes at this size, each kept identical in its results on small indexes. Status read every row: with the code on main it took 167 s on 2.8 million imported pages and 13.9 minutes on 6.03 million, long past any readiness check, and it reads in 0.9 ms now that a bulk source reports its importer's committed count. Ranking read each match's crawl time, hash and provenance, which SQLite reaches through the page text's overflow pages; it now ranks without them, by FTS5 alone when a search has no domain or age filter, and joins only the best rows. On 2.8 million pages that cut "history" (853,675 matches) from 5,652 to 512 ms, and the 189 of 200 questions with a complete match returned identical results and excerpts. The partial-term fallback matched any query word across millions of pages and took 3 to 52 s on 2.8 million (one question took 7.8 minutes on 6.03 million with main's code, while a vector-index build ran beside it); it now keeps the rarest words within 200,000 pages, which kept those searches under 1 s and changed their results, the 11 of 200 questions that used it. At 6.03 million pages, single-threaded on a cold cache, 300 questions took 428 ms median, 1.03 s at the 95th percentile and 1.7 s at most.

Semantic and hybrid answer recall were not measured at this size: they need a vector for every article, about 1.6 billion tokens of embedding (see Vectors for imported Wikipedia), which is the owner's decision. The approximate index that serves them was measured instead, against exact brute force, with real bge-small-en-v1.5 vectors (title and first 600 characters) of 50,000 articles sampled from the staging index and the 3,610 NQ-open questions as queries (with the BGE query prefix). Its 256 lists hold about 200 vectors each:

Lists scanned / rescoredRecall of the exact top 10Median / 95th percentile
16 / 10086.7%2.1 / 4.0 ms
32 / 20093.8%3.8 / 6.9 ms
64 / 400 (the defaults)97.8%6.9 / 12.1 ms
128 / 80099.6%14.9 / 21.9 ms

Exact brute force over the same 50,000 took 24.4 ms median.

At the full corpus's size there were no real vectors to test, so the 50,000 were expanded to 6,030,988 by copying each about 120 times with Gaussian noise and normalizing, and 200 of the questions were scored against exact neighbours found by a full pass. The build (4,096 lists) took 7.7 minutes in one thread and 1.19 GB, and wrote a 392 MB index beside a 12.5 GB vectors file; each query worker holds about 372 MB of it (exact search would need 9.3 GB per worker and about 3 s a query). Recall depends on how alike the copies are. With noise of 0.045 per dimension (a copy about 0.75 in cosine from its source):

Lists scanned / rescoredRecall of the exact top 10Median / 95th percentile
32 / 20074.2%18.7 / 29.4 ms
64 / 400 (the defaults)83.9%55.5 / 84.8 ms
128 / 40085.6%95.5 / 141.4 ms
128 / 80090.0%107.7 / 157.4 ms

With noise of 0.02 (copies about 0.93 from their source, so the exact top ten are near-duplicates that one-bit codes cannot order), the defaults recalled 74.8% at 31.7 ms median and 128 / 800 recalled 85.3%. The copies stand in for the density of 6 million vectors, not for their structure, so recall on real Wikipedia vectors at this size remains unmeasured until they exist; scanning more lists and rescoring more candidates raise it, at the latency shown, through VITNA_SEARCH_VECTORS_PROBES and VITNA_SEARCH_VECTORS_RESCORE. Rescoring reads stored floats from the vectors file, so on the search host it depends on how much of that file stays in memory beside the index.

Domain authority from the web graph

Common Crawl's web graph ranks every registered domain by harmonic centrality, and by PageRank, over three months of crawled hyperlinks; release cc-main-2026-jul-aug-sep covers 133.2 million domains and 2.1 billion links. Its domain ranks file is sorted by harmonic centrality, so the top domains are a prefix of one gzip file:

bash
npm run search:authority -- --release cc-main-2026-jul-aug-sep --limit 1000000

This streams the first lines of the release's domain-ranks.txt.gz, stops the download at the limit, and replaces the contents of its own SQLite file, VITNA_SEARCH_AUTHORITY (default .search-data/web/authority.sqlite), in one transaction, so a failed run leaves the previous release in place. The top million domains took 26.6 seconds and 49 MB on 2026-10-06. A host takes the rank of its longest stored suffix (science.nasa.gov takes nasa.gov's), so no public suffix list is needed: the release applied one when it grouped hosts. The file is an input for ranking and for choosing what to import; the top of the list is infrastructure (googleapis.com, gstatic.com, cloudflare.com), which serves no searchable pages. Graph data is free to use under Common Crawl's terms.

Selective archive bootstrap

Common Crawl's CDXJ index identifies archived URLs and the byte ranges of their WARC records. It can bootstrap reviewed sources; it does not answer keyword searches. The importer accepts a maximum of 100 captures per plan and only sources from web-seeds.json. It never downloads whole crawl archives.

bash
npm run search:archive -- --plan --source web-nasa --collection CC-MAIN-2026-39 --limit 20

This makes one bounded, metered CDX metadata request and saves a new plan under .search-data/archives/. Inspect its source scope, URLs, capture dates, and content permissions before running:

bash
npm run search:archive -- --import .search-data/archives/web-nasa-CC-MAIN-2026-39.json

Import checks current publisher robots/search policy, verifies the HTTP range, gzip size, WARC target/date, and payload digest, and honors captured noindex. Compressed records are limited to 2 MiB and decompressed records to 3 MiB. Unsupported encodings and chunked captures fail closed. Every network request uses the same durable day/month crawl meter; import shares the crawler lease and document/disk caps. Newer indexed content is retained, repeated captures are skipped, and each imported URL is queued for a live refresh. Current page metadata changes are observed during that live refresh.

Archived results carry contentOrigin: "common_crawl", capturedAt, importedAt, and archive collection/digest. Their crawledAt is null. Age filters use the original capture time, and status counts live and archived documents separately. Schema 1 indexes migrate transactionally to schema 2 when opened for writing; existing read-only connections detect that migration. Keep a verified snapshot before upgrading a production database.

The parser, import policies, metering, and migration passed fixture tests. A live CDX planning request timed out; no real archive content has been imported in this workspace. Live crawling remains the demonstrated ingestion path. Common Crawl access is free, but content retains its owners' rights; review applicable terms and publisher permissions.

Import English Wikipedia

Live crawling adds pages slowly: at the profile's 600,000 requests a month and the first OVH run's yield of 104 pages per 1,000 requests, about 60,000 a month. A bulk import adds all of English Wikipedia, about 7 million articles, from Wikimedia's official dump:

bash
npm run search:wikipedia -- --snapshot 20261001

The snapshot is a YYYYMMDD run listed at dumps.wikimedia.org/enwiki whose articlesmultistreamdump job is done. The importer reads that run's dumpstatus.json and fetches its pages-articles-multistream parts (71 for 20261001, 27 GB in all) one at a time from the fixed dumps.wikimedia.org URLs it names, resuming a partial part with a range request. Each part is checked against the size and SHA-1 Wikimedia publishes for it before any row is read from it, and a mismatch deletes the part and stops the import. A part is deleted once imported, so the disk holds at most one (about 650 MB); --keep-parts keeps them, --dir places them (by default wikipedia/enwiki-<snapshot>/ beside the index), --parts N stops after N parts, and --workers N sets the decode threads (by default the CPU count less two, at most 8). Nothing needs installing: the bzip2 decoder is plain JavaScript and checks every block's CRC.

Worker threads decode the dump's bzip2 streams in parallel and render each page's wikitext to the plain text the crawler's extractor produces for a page (extract.mjs): the article's prose, with links reduced to their words and the inline templates that carry prose (dates, unit conversions, language and formatting wrappers) rendered to their visible text. References, infoboxes, tables, files, categories, comments and the trailing reference, link and "See also" sections are dropped. Only main-namespace articles are imported: redirects, disambiguation pages, other namespaces and pages under the profile's 40-word minimum are skipped and counted. A page's URL is the one Wikipedia itself links to (https://en.wikipedia.org/wiki/Ender%27s_Game).

Imported articles live in their own source, wikipedia-enwiki-dump, with the CC BY-SA 4.0 reuse policy, and each result carries contentOrigin: "wikipedia_dump", the snapshot's capture time (when its stub dump finished; every revision in the dump was current then), the import time, and archive: {collection, revision}; its contentChangedAt is the revision's edit time. See the result fields. Wikipedia's text is reusable under CC BY-SA 4.0 with credit to Wikipedia's contributors and a link to the article; Wikimedia's reuse guidance describes the attribution.

The index writes each batch of pages with its resume state in one transaction, so an interrupted import (a signal, a crash, a full disk) runs again with the same command and continues after the last committed batch. Before every batch the importer checks the profile's free-disk floor (minFreeDiskBytes, 10 GiB) and the source's cap (maxDocumentsPerBulkSource, 8,000,000), and stops cleanly at either. Its downloads are not crawling and are not charged to the crawl's daily or monthly request meters. It holds its own lease, renewed every minute, so the crawler and demand jobs keep running beside it; SQLite serializes their writes, a batch at a time. A batch can hold the write lock for seconds while the full-text index merges (9.8 s at most in the staging import below), longer than a crawler write waits, so stop the crawl timer for the import if failed fetches during it matter. Searches are unaffected.

The crawler leaves imported pages to the import: it neither queues nor fetches a URL the import holds, and its caps count crawled pages only (see Crawl scope and limits). A page the crawler fetched after the snapshot's capture keeps its crawled copy; an older crawled copy gives way to the dump's. Importing a newer snapshot rewrites the articles whose text changed, gives unchanged ones the new provenance without touching the full-text index, adds new ones, and, once every part is in, removes the articles the snapshot no longer holds (deleted, now a redirect or a disambiguation page, or under the word floor). An older snapshot than one already imported is refused. The JSON summary reports pages read, articles, each skip reason, inserted, changed, unchanged and removed pages, and stopReason (null once the snapshot is complete).

Status reports the source's article count from the import's own committed count, so status never reads millions of pages; importedDocuments totals them. Vector maintenance passes imported pages over unless VITNA_SEARCH_EMBED_BULK=true; see Semantic retrieval deployment.

Optional upstream adapters need appropriate commercial rights before resale. Brave's standard terms prohibit redistribution/resale of results and using its API to replace the service; an ordinary subscription does not authorize a public search resale product. See Brave's terms.

Hosted beta

The general web beta is deployed at search.vitna.ai, with an API reference. Search and status require a separate Vitna Search key. The public page contains no embedded credential. Customer signup and billing are not enabled; coverage is still limited.

The first scheduled OVH crawl completed 1,000 requests in about 5.4 minutes, recording 90.3 MB of payload and adding 104 pages for 175 searchable pages. Its 88 publisher failures were recorded for retries while the bounded run completed successfully. The next crawl was scheduled five minutes later. Registered origins and queued URLs are not searchable coverage.

Daily private R2 backups are enabled. A real backup download passed SHA-256 and SQLite integrity checks for the index, frontier and quota snapshots. Backups expire after 14 days. HTTPS and API-key protection were verified, and a simulated certificate renewal succeeded through Cloudflare. The crawler and backup timers run on the server while the operator's PC is asleep.

Virginia cloud deployment

The Terraform module under deploy/search-web prepares one Ubuntu VM in US-EAST-VA-1, OVH's Virginia region. Its default is b3-32 (8 vCPU / 32 GB), with SSH restricted to the supplied administrator IPv4 CIDR. Only ports 80/443 are public; the API listens on loopback. The module installs Node, nginx, and rclone, and stops nginx until configuration is complete. It does not upload application code or credentials.

The pilot allocation defaults to $600/month and cannot exceed $800, preserving at least $200 of the supplied $1,000/month OVH budget. An explicit current all-in quoted_monthly_usd must fit that allocation before planning succeeds. This is a configuration guard, not a cloud billing enforcement service.

At OVH's US hourly rates checked 2026-10-03, b3-32 compute is approximately $179.68 for 744 hours. An illustrative 200 GB of allocated local NVMe adds $25.61. IPv4, taxes, any extra storage, and other services must be added to the actual project quote. Storage and IPv4 became separate charges on 2026-10-01. Do not use the page's annual commitment price for this hourly pilot. Source: OVH US pricing.

For dashboard setup, sign in to the credited OVH project, select Virginia, and inspect its current hourly instance quote and credit eligibility. Use the generated cloud-init.yaml below as the instance's startup configuration. Both accounts were accessed through their signed-in dashboards on 2026-10-03. The current beta reuses an existing instance; the Terraform instructions below are an alternative for a future dedicated VM, not the deployed resource state.

For Terraform setup, connect the credited OVH project using its private OpenStack RC or cloud configuration through the local environment, rather than chat or Terraform variables. Confirm the project, available image/flavor, SSH public key and administrator CIDR, actual allocated disk/IP charges, and credit eligibility. The OVH service account guide describes project authentication. Keep Terraform state private as well.

Copy terraform.tfvars.example to a private terraform.tfvars, fill its public-key/CIDR/quote values, then run:

bash
terraform -chdir=deploy/search-web init
terraform -chdir=deploy/search-web validate
terraform -chdir=deploy/search-web plan -out=beta.tfplan
terraform -chdir=deploy/search-web apply beta.tfplan

Only apply an inspected plan for that project within the approved allocation. The offline Terraform tests verify one Virginia instance, reject overspending, and reject unrestricted SSH. They do not verify a live account or provision VM resources. Local instance storage can be lost with the instance; the beta has no high-availability replica. R2 backups are part of its recovery plan.

Build the deployment package with:

bash
npm run search:release

It writes .search-data/releases/vitna-search-web.tgz, a SHA-256 checksum, and cloud-init.yaml for dashboard-based instance creation. The startup configuration prepares the same runtime/user as Terraform and keeps nginx stopped until application configuration is complete. The Linux Node runtime, SQLite FTS5 support, service installation, API authentication, live crawl, and R2 upload/download restore check have been verified on the actual reused VM.

The local release archive .search-data/releases/vitna-search-web.tgz contains search code, deployment files, and these docs. It excludes data, environment files, Terraform state, provider caches, and credentials. Transfer it to the prepared VM through SSH, extract into /opt/vitna-search, and run:

bash
cd /opt/vitna-search
sudo bash deploy/search-web/install-release.sh
sudo /opt/vitna-search-node/bin/node platform/search/mint-key.mjs pilot

Fill /etc/vitna-search/web.env with the client hash and reviewed settings. This environment file is readable only by root and the service group. The install helper preserves existing private settings and does not start services. Initialize and inspect a small live crawl before enabling its timer:

bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/web-run.mjs --max-requests 100
sudo systemctl enable --now vitna-search-web.service
curl --fail http://127.0.0.1:8789/readyz

Choose the owned search hostname, configure DNS and a valid TLS certificate, adapt deploy/vitna-search.nginx.conf.example, and verify nginx's configuration before starting it. The hostname in that template is an example, not a created DNS record. Configure the R2 backup below and perform one successful upload and restore check before enabling the recurring work:

bash
sudo systemctl enable --now vitna-search-web-crawl.timer vitna-search-web-backup.timer

The crawl timer advances the durable queue five minutes after each run ends; it does not recrawl every page every five minutes. The backup timer runs daily. These are VM timers; they do not keep the user's PC crawling while it is asleep.

Semantic retrieval deployment

Optional; semantic retrieval describes the contract. It uses Vitna's own embedding engine, vitna-anchor, which runs on the search host as two units serving bge-small-en-v1.5: vitna-embed.service at 127.0.0.1:8795 embeds queries, and vitna-embed-batch.service at 127.0.0.1:8797 embeds pages for vector maintenance. The engine serves one request at a time, so with two units a query never waits behind a batch: during a backfill at 16 pages a batch, a query's embedding took 17 to 21 ms from the query unit against 0.8 to 1.9 s when it shared the one engine. The weights are mapped from the same file, so the second unit costs little memory. Nothing here makes a request to any third party.

The engine binary is built from convexityos/vitna-anchor, gate A14 or later, for Linux x86-64 (for example zig cc -target x86_64-linux-gnu.2.35 -O3 -Iinclude src/*.c -o vitna-anchor -lm -lpthread in its engine/), and the model fetched with that repository's node scripts/fetch-model.mjs bge-small-en-v1.5. The release's installer checks the binary against the digest given and every model file against the pin, installs both units, and starts nothing:

bash
sudo bash deploy/search-web/install-embed.sh /path/to/vitna-anchor <sha256> /path/to/bge-small-en-v1.5
sudo systemctl enable --now vitna-embed.service vitna-embed-batch.service
curl -s http://127.0.0.1:8795/health; curl -s http://127.0.0.1:8797/health

It should answer with "embeddings":true, "dimensions":384 and "max_tokens":512. Use the model identifier it reports. Add the settings to /etc/vitna-search/web.env; commented examples are in deploy/vitna-search-web.env.example, and the vectors file defaults to /var/lib/vitna-search/web/vectors.sqlite:

dotenv
VITNA_SEARCH_EMBED_URL=http://127.0.0.1:8795/v1/embeddings
VITNA_SEARCH_EMBED_BATCH_URL=http://127.0.0.1:8797/v1/embeddings
VITNA_SEARCH_EMBED_BATCH_SIZE=16
VITNA_SEARCH_EMBED_MODEL=bge-small-en-v1.5
VITNA_SEARCH_EMBED_QUERY_PREFIX="Represent this sentence for searching relevant passages: "
VITNA_SEARCH_EMBED_DIMENSIONS=384

The query prefix is the one BGE documents for short queries; in the measurements below, omitting it changed the results by a few queries either way.

Fill the vectors file before switching the API over. Each run stops at its budget and the next continues; repeat until the summary reports "complete":true. The first pass embeds every page, later passes only new and changed pages:

bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/embed-run.mjs --budget-ms 3600000

On a desktop Ryzen 7 3700X with the engine at the production unit's three threads, embedding averaged about 2,100 tokens per second over pages of about 285 tokens each. At that rate 49,063 pages need roughly two hours of engine time: about two such runs by hand, or four to five hours through the timer alone. The search host's rate will differ.

The release installer places vitna-search-web-embed.service and .timer without enabling them. Enable the timer, which runs the job ten minutes after each run ends within an eight-minute budget, then restart the API so it reads the settings:

bash
sudo systemctl enable --now vitna-search-web-embed.timer
sudo systemctl restart vitna-search-web.service

Authenticated GET /v1/status should then report retrieval.semantic.state: "ready" with a vectors count near index.documents, and a search should report meta.index.retrieval.used: "hybrid". Receipts in journalctl -u vitna-search-web show any fallback codes.

Each query worker holds its own copy of the vectors inside the API unit's 4 GB limit, about 160 MB for two workers at 49,063 pages and 384 dimensions. The engine serves one request at a time, so without VITNA_SEARCH_EMBED_BATCH_URL a search's query embedding waits for any maintenance batch in progress on the one engine. At three threads, a query arriving during 32-page batches waited 4.0 s median, and 24 of 26 exceeded the 2 s VITNA_SEARCH_EMBED_TIMEOUT_MS default and would have fallen back to keyword retrieval. During the default 4-page batches the wait was 0.35 s median and 0.51 s at the 95th percentile, none over 2 s, with no loss of throughput (2,125 against 2,223 tokens per second). On one engine, raise VITNA_SEARCH_EMBED_BATCH_SIZE only if searches can tolerate that wait; with the batch unit, queries never see it, and 16 pages a batch keeps the batch engine busy. To turn the feature off, comment out the settings and restart the API; the vectors file is then ignored. To change the model, point VITNA_SEARCH_VECTORS at a new file, fill it, and restart the API. The vectors file is part of the R2 bundle, as vectors.sqlite.gz. It is a snapshot of its own, so it can trail or lead the index snapshot slightly; after a restore the job removes vectors whose page changed identity and embeds what is missing. Rebuilding it from the index instead took about 1.2 hours at 4 engine threads for 49,063 pages. The backup reads only web-backup.env, so a VITNA_SEARCH_VECTORS set for the API must be set there too.

Vectors for imported Wikipedia

Vector maintenance passes imported articles over unless VITNA_SEARCH_EMBED_BULK=true, because embedding them is the expensive part of semantic retrieval over Wikipedia. Each article embeds its title and leading text up to the 1,500-character recipe, 260 tokens on average over 3,000 sampled articles, so the 6.03 million articles of snapshot 20261001 are about 1.6 billion tokens: some 8.6 days at the 2,100 tokens per second measured above for three engine threads, and still days on all eight cores of the search host, during which the batch engine is fully busy. Whether and where to spend that is the owner's decision. With the setting on, crawled pages still go first, the job keeps its memory bounded by comparing pages and vectors in ranges of 100,000 ids, and each run starts with a pass over the whole index and vectors file, so a backfill runs best by hand with a one-hour budget rather than the timer's eight minutes.

Seven million 384-dimension vectors are about 11 GB in vectors.sqlite, far past what a query worker can hold, so build the approximate index once the backfill is in, with the API's environment, and give each worker room for it:

bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web.env platform/search/vector-index-run.mjs
dotenv
VITNA_SEARCH_VECTORS_MAX_MB=1024

The API reads the bound at start, so restart it after raising the bound; it picks up a new or rebuilt index on its own. Vectors embedded after a build are searched exactly beside the index until the next build.

Private R2 backups and recovery

Create a dedicated private R2 Standard bucket and a bucket-scoped Object Read & Write credential. Put it in /etc/vitna-search/rclone.conf, using deploy/vitna-search-rclone.conf.example. Restrict it to root and the service group, mode 0640. Keep the existing vitna-search-r2 remote name or update /etc/vitna-search/web-backup.env to match it and the actual bucket.

Configure a 14-day R2 expiration rule for the web-backups/ prefix, plus incomplete multipart cleanup. Remote lifecycle settings require cloud account access and are not created by this package. The helper keeps two local bundles after successful upload; failed uploads retain their local files and require attention before they consume the disk.

Backups need temporary space for a raw SQLite snapshot plus the compressed file. Since the 2026-10-17 Wikipedia import the index is about 40 GB, so the snapshot is that large and a bundle should be about 16 GB. A run plans for its peak (each raw snapshot beside its compressed file and the compressed files before it, at the compression the last local manifest measured, or one half without one) and leaves VITNA_SEARCH_BACKUP_MIN_FREE_GB free at every step, 20 GiB by default, twice the crawler's own stop:

  • To make room it removes local bundles that R2 already holds, oldest first and only as many as it needs. A bundle is marked as held by uploaded.json, written once its remote manifest lands. A bundle that never finished uploading may be the only copy and is never removed for space.
  • If the room is still short, the run fails before any snapshot and says how much it needed.
  • It checks again before each snapshot and while compressing, so a file that compresses worse than planned stops the run at the floor, not at a full disk.
  • A run that stops before its local manifest removes its own partial bundle. One killed outright (the unit's TimeoutStartSec is 6 hours) leaves one, and the next run removes it once it is 12 hours old.

Each upload may take five minutes plus one second for every 2 MiB, and manifest.json records each file's raw size as sourceBytes, which is the space a restore needs to decompress it.

bash
sudo -u vitna-search /opt/vitna-search-node/bin/node --env-file=/etc/vitna-search/web-backup.env platform/search/web-backup-run.mjs
npm run search:web:backup

The npm alias reads the same settings from its environment when used locally. Each bundle contains compressed index, frontier, and API quota databases, the concept catalog when it holds titles, and the semantic vectors file when one holds vectors (beside the index unless VITNA_SEARCH_VECTORS names another), plus SHA-256 hashes in manifest.json. A file at the vectors path that belongs to another database stops the backup before any snapshot. The run's JSON line lists the snapshots taken. SQLite's snapshot operation and integrity check run before upload. The remote manifest uploads last as a completion marker. A prefix without a manifest is an incomplete remote backup. Discovery contexts are encrypted inside the frontier snapshot. The separate discovery key is not included in the database bundle; keep a private recovery copy if pending jobs must survive restoration, and restore the same key to all three service processes before resuming them.

For recovery, stop the API, crawler service/timer, embed timer, and backup service/timer. Download one complete manifest and every file it lists into a new private staging directory. Verify compressed byte counts and SHA-256 hashes, decompress them, and run SQLite integrity checks before replacing any data. Preserve the old files for rollback and set restored ownership/modes for vitna-search. A bundle without vectors.sqlite.gz predates the vectors backup: leave the vectors file absent and the embed job rebuilds it.

The snapshots are individually consistent, not one transaction across databases. Queue leases expire and interrupted work may be retried; reconcile the queue against the index after restoration. Restoring older quota or crawl meters can lose usage after the snapshot. Reconcile those counters from saved status/receipts before resuming; if usage cannot be recovered, keep affected work paused until its day/month limit resets. Verify /readyz, an authenticated search/status, and the new usage values before restarting timers.

R2 Standard storage is $0.015/GB-month, with a 10 GB-month free storage tier and free Internet egress; operation charges also apply. Source: R2 pricing. The supplied $10,000 Cloudflare credit award needs its actual eligible products and expiry checked in the account. This implementation assumes no credit balance, award duration, or free unlimited storage.