Get started

Run a focused search index

Source

Vitna can crawl selected public AI documentation, keep an independent keyword index, and serve it through the search API. An OVHcloud Public Cloud instance can run the service and crawler; Cloudflare R2 can hold standalone snapshots. The deployment files are prepared locally. No cloud resources, DNS records, R2 uploads, or credit spending have been created by this implementation. For cross-site discovery, persistent crawl budgets, API quotas, and the Virginia deployment package, use the general web beta.

Try it locally

bash
node platform/search/demo.mjs

Open http://127.0.0.1:8789. The page connects automatically using a temporary demo-only key. The four illustrative pages pass through the real crawler, SQLite index, and HTTP API using an in-memory fetch fixture. Nothing is downloaded, no provider credential is required, and no price claim is implied. The demo binds to loopback and loses its index and key when stopped.

The repository also provides these script aliases. The crawl, snapshot, and backup commands read the settings from their process environment:

bash
npm run search:demo
npm run search:crawl -- --source exa
npm run search:snapshot
npm run search:backup

For a command-line check that exits after verifying a search:

bash
node platform/search/demo.mjs --check

Crawl and serve real pages

Review platform/search/sources.json or create a private catalog and set VITNA_SEARCH_CATALOG to its path. The starter catalog contains the official Cloudflare Workers AI/R2, Exa, and Tavily documentation. It is a crawl scope, not a downloaded corpus or a promise that those sites permit this crawler.

Generate a key with node platform/search/mint-key.mjs pilot. Keep the printed token for the API client and put only its hash into a private .env.search:

dotenv
VITNA_SEARCH_INDEX=.search-data/index.sqlite
VITNA_SEARCH_PROVIDERS=vitna
VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"}
bash
node --env-file=.env.search platform/search/crawl-run.mjs
node --env-file=.env.search platform/search/start.mjs

The first command performs a live, bounded crawl. The second starts the browser interface and authenticated API on 127.0.0.1:8789. Paste the token into the interface, connect, and search. No paid search-engine API account is needed. New clients should send mode: "index" explicitly; the browser selects Index by default. A valid empty Index search does not call an external engine. Cloud credits cover eligible infrastructure according to the account's offer.

This focused catalog controls its own crawl scope. For broader query-guided coverage, the general web beta can automatically queue discovery after an empty name/topic query or weak partial-term topic matches, while preserving current search results. A plain {"query":"examplebrand","mode":"index"} needs no domain input; domains are optional guidance. Automatic discovery uses observed names; explicit expansion can also use matching known URLs. Narrow ASCII brand candidates can use .com, .ai, and .org. Candidate hosts are not verified official identities. Name detection is heuristic; common topic words and multiword topic queries generate no guessed hosts. Broad queries can instead select known publisher domains by matching catalog descriptions. Those domains are discovery candidates, not indexed coverage. An optional separate local English Wikipedia title catalog can suggest concept article URLs; titles, including redirect titles, are metadata without canonical-target mappings. Article text becomes usable only after a permitted page is fetched and indexed. Queries without a suitable name, topic, or concept match can return no_candidates. Explicit expand: false or discover: false disables automatic discovery; expand: true, discover: false refreshes known targets only. Explicit expand: true, discover: true requests site discovery. It examines up to three relevant sites and 2,000 sitemap candidates, then attempts at most eight target pages within 40 HTTP requests and 60 seconds of crawler work. It requires a private discovery key and does not promise whole-web coverage. Current results return first; poll the private job, then search again after completion to retrieve any newly indexed pages.

Use POST /v1/extract with a returned URL to read its already indexed cleaned text. It shares the search key and quotas, returns source/crawl metadata, and performs no live fetch or JavaScript execution. The default text limit is 20,000 characters, up to 100,000; Markdown structure is not guaranteed. An absent or policy-unavailable page returns HTTP 404. See indexed text extraction.

POST /v1/research assembles Source brief from cited verbatim passages in matched indexed pages. It defaults to three sources and an 8,000-code-point quoted passage budget, capped at five sources and 16,000 code points. It uses actual page titles and reports bounded evidence coverage; it does not generate an explanation or claim an exhaustive review. See Source brief research.

For the general web deployment, initialize the topic catalog explicitly with the existing web VITNA_SEARCH_FRONTIER and VITNA_SEARCH_INDEX settings in the process environment:

bash
npm run search:topics -- --init
npm run search:topics -- --review-seeds
npm run search:topics -- --backfill

These commands make no network requests. They initialize storage, record 28 curated publisher descriptors, and inspect at most 1,000 eligible already indexed homepages. The optional dataset download is a separate command:

bash
npm run search:topics -- --import-majestic --limit 10000

Imported domain candidates remain unverified metadata, separate from publisher descriptions and indexed pages. None of these commands register crawl origins, enqueue pages, or start crawling. See topic catalog setup for default files, download bounds, the 24-hour cooldown, and source attribution.

For optional concept discovery, point VITNA_SEARCH_CONCEPTS to a separate SQLite file shared by the API and demand worker. Web mode defaults to .search-data/web/concepts.sqlite. Import a selected published title snapshot explicitly; startup does not download it:

bash
npm run search:concepts -- --import-wikipedia --snapshot 20261003

This is title metadata, including redirect names without target mappings, not article content. See concept snapshot import for source selection, disk and download bounds, and attribution.

/healthz checks the HTTP process. /readyz returns 200 only when the default local provider has indexed content. /v1/status requires a search key and reports source counts, crawl freshness, and the last run. See the API contract for request limits, authentication, and fallback.

Catalog and crawl boundaries

The operator controls the catalog. There is no public crawl/import endpoint. Each source specifies a unique id, display name, HTTPS origin, directory pathPrefixes ending in /, and seeds inside those prefixes. Sources can be disabled with enabled: false; dropping, disabling, or narrowing a source removes its excluded documents the next time the crawler synchronizes it.

Source settingDefaultLimit
maxPages50Page attempts per source per run; 1–10,000
maxDocuments5,000Indexed documents per source; 1–100,000
maxDepth2Discovered-link depth, 0–5
delayMs1,000Minimum time between request starts to an origin, 1,000–60,000
refreshHours24Minimum interval before downloading a known page again, 1–8,760

Run only one source with search:crawl -- --source exa, or add --force to refresh known pages before their interval. Force does not override robots, scope, byte limits, document caps, or the global request budget.

Crawler environmentDefaultMeaning
VITNA_SEARCH_INDEX.search-data/index.sqliteWritable standalone search file
VITNA_SEARCH_CATALOGBundled sources.jsonReviewed operator catalog
VITNA_SEARCH_CRAWL_REQUESTS200All HTTP attempts, including robots and redirects
VITNA_SEARCH_CRAWL_DURATION_MS900,000Maximum run duration; up to one hour
VITNA_SEARCH_CRAWL_TIMEOUT_MS10,000Timeout per request; up to 60 seconds

The crawler identifies itself as VitnaSearch, checks robots.txt each run, merges matching agent groups, honors allow/disallow wildcard and end rules, and uses the larger of the source delay and robots Crawl-delay. A delay above 60 seconds skips that source. A missing robots file (404/410) permits the configured scope; failed, forbidden, oversized, or redirected-to-other-origin robots responses skip the source. Newly disallowed indexed pages are removed even if they are beyond the page budget. SERP page contents are read by a separate agent, `VitnaSearch-User`, only when a request asks, and nothing it reads enters the index.

Connections require public HTTPS hostnames on the standard port. DNS answers must all be public addresses; the validated address is pinned into the actual TLS connection. Redirects are checked again before a connection. Page redirects remain inside the source's origin and path prefixes. Credentials, IP literals, internal destinations, query strings, encoded path separators, and non-text/download links are refused. Pages are capped at 2 MiB and robots.txt at 512 KiB. Excessively nested HTML is refused.

An instance that must send no Google traffic from its own addresses sets VITNA_SEARCH_REFUSE_GOOGLE=true (default false) for every search process: the API, the web crawl and the demand runs. Each public fetch then refuses Google's own hosts by name before any lookup (crawl_google_host), and any name whose DNS answer falls in Google's own serving ranges before any connection (crawl_google_address), which also covers a site on Google's front ends under its own domain. The ranges are Google's published goog.json less the Google Cloud customer ranges of cloud.json, so a third-party server on Google Cloud is still read; refresh them with node scripts/google-ranges.mjs on a workstation, never on the instance. A crawl source on Google then fails its jobs with those codes, and pages already indexed from it stay until removed. Any other value refuses to start.

HTML, plain text, and Markdown are supported. Extraction prefers main/article content, removes scripts/navigation/hidden content, and indexes bounded plain text. Noindex pages and observed 404/410 pages are removed; nofollow suppresses link discovery. Canonical tags cannot redirect the crawler. ETag and Last-Modified validators support conditional refreshes. A 304 or unchanged extracted text advances crawl freshness without inventing a content-change date. A failed refresh retains the previous content and its old timestamp.

The index ranks lexical matches with SQLite FTS5 BM25, weighting titles more than body text. It returns excerpts, source identity, crawledAt, and contentChangedAt, and a publication date only where the page declares one (declared publication dates); it never infers one. It does not execute JavaScript, parse PDFs, ingest arbitrary sitemaps, rerank with a model, or generate answers. Embedding pages is optional: with Vitna's own embedding engine configured, semantic retrieval adds semantic and hybrid ranking from a separate vectors file. In this single-process mode the vector scan runs on the API's own thread rather than in query workers. Raw HTML is not archived by this version; the index stores extracted text, validators, and discovered in-scope links.

One index lease prevents overlapping crawler runs, with expiry to recover after an interruption. Writes and FTS updates are atomic; WAL permits the read-only API connection to coexist with a crawler. Recent run summaries are kept in the index, with source counts and error codes. No customer search queries, client tokens, signing keys, or site database data enter this file.

OVHcloud deployment package

Use a single Linux Public Cloud instance with Node 24, persistent local/block storage, nginx, and rclone if R2 backups are enabled. A 2-vCPU instance with 4–8 GB RAM is a starting configuration to benchmark, not a throughput guarantee. Credit coverage and expiry come from the account's actual offer; no specific OVH instance price or credit balance is assumed here.

Keep the checkout under /opt/vitna-search, create an unprivileged vitna-search account, and give that account ownership of /var/lib/vitna-search. Code stays owned by the deployment administrator. There is no web build or npm install needed for these standalone commands.

Copy and fill the three environment templates into /etc/vitna-search:

  • deploy/vitna-search.env.example → api.env: hashed clients and local-index mode.
  • deploy/vitna-search-crawl.env.example → crawl.env: reviewed scope and request budgets.
  • deploy/vitna-search-backup.env.example → backup.env: snapshot directory and dedicated R2 remote.

Use root ownership, the vitna-search group, and mode 0640 for those files. The API receives client hashes; the crawler receives no API keys; the backup process alone receives the path to its R2 config. None needs the website's database environment or any signing credential.

Install deploy/vitna-search.service, vitna-search-crawl.service, vitna-search-crawl.timer, vitna-search-backup.service, and vitna-search-backup.timer under /etc/systemd/system. After reviewing the scope and activating credit-covered resources, the operator can run:

bash
sudo systemctl daemon-reload
sudo systemctl start vitna-search-crawl.service
sudo systemctl enable --now vitna-search.service vitna-search-crawl.timer

Check readiness and the first crawl summary before exposing the service:

bash
curl --fail http://127.0.0.1:8789/readyz
sudo journalctl -u vitna-search-crawl -u vitna-search --since today

Configure DNS, a valid TLS certificate, and nginx using deploy/vitna-search.nginx.conf.example. The example host is search.vitna.ai; it is not a provisioned hostname. The API remains bound to loopback. The hourly timer checks freshness; the catalog's default refresh interval is daily. Units bound CPU/memory and leave only the search state directory writable.

An alternative standalone image is available with:

bash
docker build -f deploy/vitna-search.Dockerfile -t vitna-search .
docker volume create vitna-search-data
docker run --rm -v vitna-search-data:/data vitna-search node platform/search/crawl-run.mjs
docker run --rm --env-file .env.search -p 127.0.0.1:8789:8789 -v vitna-search-data:/data vitna-search

For the container environment file, use VITNA_SEARCH_INDEX=/data/index.sqlite and VITNA_SEARCH_PROVIDERS=vitna. API and crawler share the writable volume for SQLite WAL coordination; the API opens the database read-only. The image contains the API, crawler, and local snapshot command. Rclone uploads use the host backup unit, or a separate backup image with rclone installed. Linux systemd/nginx/Docker execution has not been verified by the local Windows run.

R2 snapshots and restore

Create a dedicated private vitna-search bucket and a bucket-scoped Object Read & Write credential. Fill deploy/vitna-search-rclone.conf.example into /etc/vitna-search/rclone.conf, readable only by root and vitna-search. Use the account endpoint without appending the bucket name. Actual keys stay outside the checkout and outside the API environment.

search:snapshot creates a consistent standalone SQLite file using VACUUM INTO. It works while the service is up and refuses to overwrite an existing destination. Copying the live file directly can miss WAL changes.

bash
node --env-file=.env.search platform/search/snapshot-run.mjs
sudo systemctl start vitna-search-backup.service
sudo systemctl enable --now vitna-search-backup.timer

The R2 command writes to a unique .partial name, finalizes the snapshot, then uses rclone copyto. A failed upload keeps the complete local snapshot. After successful uploads, 14 completed snapshots are retained locally by default; only generated snapshot names inside the configured directory are eligible for pruning. Remote objects are never deleted by the command. Set an R2 lifecycle policy for the dedicated backup prefix to bound remote storage as the index grows. Failed uploads require monitoring because local retention does not run until an upload succeeds.

For restore, stop the API and crawl timer, download a selected snapshot with rclone copyto to a new filename, and check it with openSearchIndex in read-only mode. Point the API and crawler to that file, restore the search account's ownership, and restart. Keep the old index for rollback. Do not replace a live SQLite file or mix a restored file with old WAL/SHM sidecars. If a snapshot contains an active crawl lease, it expires within the configured run duration plus timeout and one minute before a new crawl can acquire it.

Validation and remaining scope

The offline tests cover persistence, ranking, freshness, source removal, robots behavior, scope/DNS/redirect restrictions, crawl caps, conditional GET, deletion/noindex, cancellation, local authentication, and snapshot restore. The backup test uses an injected upload operation; it does not contact R2. Live source accessibility, account credit eligibility, real uploads, Linux service installation, corpus size, and query throughput require validation on the configured cloud instance. In-memory client quotas remain an alpha limit; public signup, durable billing metering, and multi-instance serving are future work described in the API overview.

Protocol references: Node SQLite, SQLite FTS5, robots exclusion, and Cloudflare R2 with rclone.