vitna.ai/docs/search/self-hosted Machine view
Raw .md llms.txt
> Source: https://vitna.ai/docs/search/self-hosted
> Raw Markdown: https://vitna.ai/docs/search/self-hosted.md
> Corpus index: https://vitna.ai/llms.txt

# Run a focused search index

Vitna can crawl selected public AI documentation, keep an independent keyword
index, and serve it through the search API. An OVHcloud Public Cloud instance
can run the service and crawler; Cloudflare R2 can hold standalone snapshots.
The deployment files are prepared locally. No cloud resources, DNS records,
R2 uploads, or credit spending have been created by this implementation.
For cross-site discovery, persistent crawl budgets, API quotas, and the Virginia
deployment package, use the [general web beta](general-web.md).

## Try it locally

```bash
node platform/search/demo.mjs
```

Open `http://127.0.0.1:8789`. The page connects automatically using a temporary
demo-only key. The four illustrative pages pass through the real crawler,
SQLite index, and HTTP API using an in-memory fetch fixture. Nothing is
downloaded, no provider credential is required, and no price claim is implied.
The demo binds to loopback and loses its index and key when stopped.

The repository also provides these script aliases. The crawl, snapshot, and
backup commands read the settings from their process environment:

```bash
npm run search:demo
npm run search:crawl -- --source exa
npm run search:snapshot
npm run search:backup
```

For a command-line check that exits after verifying a search:

```bash
node platform/search/demo.mjs --check
```

## Crawl and serve real pages

Review `platform/search/sources.json` or create a private catalog and set
`VITNA_SEARCH_CATALOG` to its path. The starter catalog contains the official
Cloudflare Workers AI/R2, Exa, and Tavily documentation. It is a crawl scope,
not a downloaded corpus or a promise that those sites permit this crawler.

Generate a key with `node platform/search/mint-key.mjs pilot`. Keep the printed
token for the API client and put only its hash into a private `.env.search`:

```dotenv
VITNA_SEARCH_INDEX=.search-data/index.sqlite
VITNA_SEARCH_PROVIDERS=vitna
VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_THE_PRINTED_SHA256_HASH"}
```

```bash
node --env-file=.env.search platform/search/crawl-run.mjs
node --env-file=.env.search platform/search/start.mjs
```

The first command performs a live, bounded crawl. The second starts the browser
interface and authenticated API on `127.0.0.1:8789`. Paste the token into the
interface, connect, and search. No paid search-engine API account is needed.
New clients should send `mode: "index"` explicitly; the browser selects Index
by default. A valid empty Index search does not call an external engine.
Cloud credits cover eligible infrastructure according to the account's offer.

This focused catalog controls its own crawl scope. For broader query-guided
coverage, the [general web beta](general-web.md#query-guided-coverage-expansion)
can automatically queue discovery after an empty name/topic query or weak
partial-term topic matches, while preserving current search results.
A plain `{"query":"examplebrand","mode":"index"}` needs no domain input;
domains are optional guidance. Automatic discovery uses observed names;
explicit expansion can also use matching known URLs. Narrow ASCII brand
candidates can use `.com`, `.ai`, and `.org`. Candidate hosts are not verified
official identities. Name detection is heuristic; common topic words and
multiword topic queries generate no guessed hosts. Broad queries can instead
select known publisher domains by matching catalog descriptions. Those
domains are discovery candidates, not indexed coverage. An optional separate
local English Wikipedia title catalog can suggest concept article URLs;
titles, including redirect titles, are metadata without canonical-target
mappings. Article text becomes usable only after a permitted page is fetched
and indexed. Queries without a suitable name, topic, or concept match can
return `no_candidates`.
Explicit `expand: false` or `discover: false` disables automatic discovery;
`expand: true, discover: false` refreshes known targets only. Explicit
`expand: true, discover: true` requests site discovery. It examines up
to three relevant sites and 2,000 sitemap candidates, then attempts at most
eight target pages within 40 HTTP requests and 60 seconds of crawler work.
It requires a private discovery key and does not promise whole-web coverage.
Current results return first; poll the private job, then search again after
completion to retrieve any newly indexed pages.

Use `POST /v1/extract` with a returned URL to read its already indexed cleaned
text. It shares the search key and quotas, returns source/crawl metadata,
and performs no live fetch or JavaScript execution. The default text limit
is 20,000 characters, up to 100,000; Markdown structure is not guaranteed.
An absent or policy-unavailable page returns HTTP 404. See
[indexed text extraction](overview.md#indexed-text-extraction).

`POST /v1/research` assembles Source brief from cited verbatim passages in
matched indexed pages. It defaults to three sources and an 8,000-code-point
quoted passage budget, capped at five sources and 16,000 code points. It uses
actual page titles and reports bounded evidence coverage; it does not generate
an explanation or claim an exhaustive review. See
[Source brief research](overview.md#source-brief-research).

For the general web deployment, initialize the topic catalog explicitly with
the existing web `VITNA_SEARCH_FRONTIER` and `VITNA_SEARCH_INDEX` settings in
the process environment:

```bash
npm run search:topics -- --init
npm run search:topics -- --review-seeds
npm run search:topics -- --backfill
```

These commands make no network requests. They initialize storage, record
28 curated publisher descriptors, and inspect at most 1,000 eligible already
indexed homepages. The optional dataset download is a separate command:

```bash
npm run search:topics -- --import-majestic --limit 10000
```

Imported domain candidates remain unverified metadata, separate from publisher
descriptions and indexed pages. None of these commands register crawl origins,
enqueue pages, or start crawling. See [topic catalog setup](general-web.md#initialize-the-topic-catalog)
for default files, download bounds, the 24-hour cooldown, and source attribution.

For optional concept discovery, point `VITNA_SEARCH_CONCEPTS` to a separate
SQLite file shared by the API and demand worker. Web mode defaults to
`.search-data/web/concepts.sqlite`. Import a selected published title snapshot
explicitly; startup does not download it:

```bash
npm run search:concepts -- --import-wikipedia --snapshot 20261003
```

This is title metadata, including redirect names without target mappings,
not article content. See [concept snapshot import](general-web.md#import-a-concept-title-snapshot)
for source selection, disk and download bounds, and attribution.

`/healthz` checks the HTTP process. `/readyz` returns 200 only when the default
local provider has indexed content. `/v1/status` requires a search key and
reports source counts, crawl freshness, and the last run. See the
[API contract](overview.md) for request limits, authentication, and fallback.

## Catalog and crawl boundaries

The operator controls the catalog. There is no public crawl/import endpoint.
Each source specifies a unique `id`, display `name`, HTTPS `origin`, directory
`pathPrefixes` ending in `/`, and `seeds` inside those prefixes. Sources can be
disabled with `enabled: false`; dropping, disabling, or narrowing a source
removes its excluded documents the next time the crawler synchronizes it.

| Source setting | Default | Limit |
|---|---:|---|
| `maxPages` | 50 | Page attempts per source per run; 1–10,000 |
| `maxDocuments` | 5,000 | Indexed documents per source; 1–100,000 |
| `maxDepth` | 2 | Discovered-link depth, 0–5 |
| `delayMs` | 1,000 | Minimum time between request starts to an origin, 1,000–60,000 |
| `refreshHours` | 24 | Minimum interval before downloading a known page again, 1–8,760 |

Run only one source with `search:crawl -- --source exa`, or add `--force` to
refresh known pages before their interval. Force does not override robots,
scope, byte limits, document caps, or the global request budget.

| Crawler environment | Default | Meaning |
|---|---:|---|
| `VITNA_SEARCH_INDEX` | `.search-data/index.sqlite` | Writable standalone search file |
| `VITNA_SEARCH_CATALOG` | Bundled `sources.json` | Reviewed operator catalog |
| `VITNA_SEARCH_CRAWL_REQUESTS` | 200 | All HTTP attempts, including robots and redirects |
| `VITNA_SEARCH_CRAWL_DURATION_MS` | 900,000 | Maximum run duration; up to one hour |
| `VITNA_SEARCH_CRAWL_TIMEOUT_MS` | 10,000 | Timeout per request; up to 60 seconds |

The crawler identifies itself as `VitnaSearch`, checks robots.txt each run,
merges matching agent groups, honors allow/disallow wildcard and end rules,
and uses the larger of the source delay and robots `Crawl-delay`. A delay above
60 seconds skips that source. A missing robots file (404/410) permits the
configured scope; failed, forbidden, oversized, or redirected-to-other-origin
robots responses skip the source. Newly disallowed indexed pages are removed
even if they are beyond the page budget. SERP page contents are read by a
separate agent, [`VitnaSearch-User`](serp.md#vitnasearch-user), only when a
request asks, and nothing it reads enters the index.

Connections require public HTTPS hostnames on the standard port. DNS answers
must all be public addresses; the validated address is pinned into the actual
TLS connection. Redirects are checked again before a connection. Page
redirects remain inside the source's origin and path prefixes. Credentials,
IP literals, internal destinations, query strings, encoded path separators,
and non-text/download links are refused. Pages are capped at 2 MiB and
robots.txt at 512 KiB. Excessively nested HTML is refused.

An instance that must send no Google traffic from its own addresses sets
`VITNA_SEARCH_REFUSE_GOOGLE=true` (default `false`) for every search process:
the API, the web crawl and the demand runs. Each public fetch then refuses
Google's own hosts by name before any lookup (`crawl_google_host`), and any
name whose DNS answer falls in Google's own serving ranges before any
connection (`crawl_google_address`), which also covers a site on Google's
front ends under its own domain. The ranges are Google's published `goog.json`
less the Google Cloud customer ranges of `cloud.json`, so a third-party server
on Google Cloud is still read; refresh them with `node scripts/google-ranges.mjs`
on a workstation, never on the instance. A crawl source on Google then fails its
jobs with those codes, and pages already indexed from it stay until removed.
Any other value refuses to start.

HTML, plain text, and Markdown are supported. Extraction prefers main/article
content, removes scripts/navigation/hidden content, and indexes bounded plain
text. Noindex pages and observed 404/410 pages are removed; nofollow suppresses
link discovery. Canonical tags cannot redirect the crawler. ETag and
Last-Modified validators support conditional refreshes. A 304 or unchanged
extracted text advances crawl freshness without inventing a content-change
date. A failed refresh retains the previous content and its old timestamp.

The index ranks lexical matches with SQLite FTS5 BM25, weighting titles more
than body text. It returns excerpts, source identity, `crawledAt`, and
`contentChangedAt`, and a publication date only where the page declares one
([declared publication dates](overview.md#declared-publication-dates)); it
never infers one. It does not execute
JavaScript, parse PDFs, ingest arbitrary sitemaps, rerank with a model, or
generate answers. Embedding pages is optional: with Vitna's own embedding
engine configured, [semantic retrieval](overview.md#semantic-retrieval) adds
semantic and hybrid ranking from a separate vectors file. In this
single-process mode the vector scan runs on the API's own thread rather than
in query workers. Raw HTML is not archived by this version; the
index stores extracted text, validators, and discovered in-scope links.

One index lease prevents overlapping crawler runs, with expiry to recover
after an interruption. Writes and FTS updates are atomic; WAL permits the
read-only API connection to coexist with a crawler. Recent run summaries are
kept in the index, with source counts and error codes. No customer search
queries, client tokens, signing keys, or site database data enter this file.

## OVHcloud deployment package

Use a single Linux Public Cloud instance with Node 24, persistent local/block
storage, nginx, and rclone if R2 backups are enabled. A 2-vCPU instance with
4–8 GB RAM is a starting configuration to benchmark, not a throughput guarantee.
Credit coverage and expiry come from the account's actual offer; no specific
OVH instance price or credit balance is assumed here.

Keep the checkout under `/opt/vitna-search`, create an unprivileged
`vitna-search` account, and give that account ownership of
`/var/lib/vitna-search`. Code stays owned by the deployment administrator.
There is no web build or npm install needed for these standalone commands.

Copy and fill the three environment templates into `/etc/vitna-search`:

- `deploy/vitna-search.env.example` → `api.env`: hashed clients and local-index mode.
- `deploy/vitna-search-crawl.env.example` → `crawl.env`: reviewed scope and request budgets.
- `deploy/vitna-search-backup.env.example` → `backup.env`: snapshot directory and dedicated R2 remote.

Use root ownership, the `vitna-search` group, and mode 0640 for those files.
The API receives client hashes; the crawler receives no API keys; the backup
process alone receives the path to its R2 config. None needs the website's
database environment or any signing credential.

Install `deploy/vitna-search.service`, `vitna-search-crawl.service`,
`vitna-search-crawl.timer`, `vitna-search-backup.service`, and
`vitna-search-backup.timer` under `/etc/systemd/system`. After reviewing the
scope and activating credit-covered resources, the operator can run:

```bash
sudo systemctl daemon-reload
sudo systemctl start vitna-search-crawl.service
sudo systemctl enable --now vitna-search.service vitna-search-crawl.timer
```

Check readiness and the first crawl summary before exposing the service:

```bash
curl --fail http://127.0.0.1:8789/readyz
sudo journalctl -u vitna-search-crawl -u vitna-search --since today
```

Configure DNS, a valid TLS certificate, and nginx using
`deploy/vitna-search.nginx.conf.example`. The example host is `search.vitna.ai`;
it is not a provisioned hostname. The API remains bound to loopback. The
hourly timer checks freshness; the catalog's default refresh interval is daily.
Units bound CPU/memory and leave only the search state directory writable.

An alternative standalone image is available with:

```bash
docker build -f deploy/vitna-search.Dockerfile -t vitna-search .
docker volume create vitna-search-data
docker run --rm -v vitna-search-data:/data vitna-search node platform/search/crawl-run.mjs
docker run --rm --env-file .env.search -p 127.0.0.1:8789:8789 -v vitna-search-data:/data vitna-search
```

For the container environment file, use `VITNA_SEARCH_INDEX=/data/index.sqlite`
and `VITNA_SEARCH_PROVIDERS=vitna`. API and crawler share the writable volume
for SQLite WAL coordination; the API opens the database read-only. The image
contains the API, crawler, and local snapshot command. Rclone uploads use the
host backup unit, or a separate backup image with rclone installed. Linux
systemd/nginx/Docker execution has not been verified by the local Windows run.

## R2 snapshots and restore

Create a dedicated private `vitna-search` bucket and a bucket-scoped Object
Read & Write credential. Fill `deploy/vitna-search-rclone.conf.example` into
`/etc/vitna-search/rclone.conf`, readable only by root and `vitna-search`.
Use the account endpoint without appending the bucket name. Actual keys stay
outside the checkout and outside the API environment.

`search:snapshot` creates a consistent standalone SQLite file using
`VACUUM INTO`. It works while the service is up and refuses to overwrite an
existing destination. Copying the live file directly can miss WAL changes.

```bash
node --env-file=.env.search platform/search/snapshot-run.mjs
sudo systemctl start vitna-search-backup.service
sudo systemctl enable --now vitna-search-backup.timer
```

The R2 command writes to a unique `.partial` name, finalizes the snapshot,
then uses `rclone copyto`. A failed upload keeps the complete local snapshot.
After successful uploads, 14 completed snapshots are retained locally by
default; only generated snapshot names inside the configured directory are
eligible for pruning. Remote objects are never deleted by the command. Set
an R2 lifecycle policy for the dedicated backup prefix to bound remote storage
as the index grows. Failed uploads require monitoring because local retention
does not run until an upload succeeds.

For restore, stop the API and crawl timer, download a selected snapshot with
`rclone copyto` to a new filename, and check it with `openSearchIndex` in
read-only mode. Point the API and crawler to that file, restore the search
account's ownership, and restart. Keep the old index for rollback. Do not
replace a live SQLite file or mix a restored file with old WAL/SHM sidecars.
If a snapshot contains an active crawl lease, it expires within the configured
run duration plus timeout and one minute before a new crawl can acquire it.

## Validation and remaining scope

The offline tests cover persistence, ranking, freshness, source removal,
robots behavior, scope/DNS/redirect restrictions, crawl caps, conditional GET,
deletion/noindex, cancellation, local authentication, and snapshot restore.
The backup test uses an injected upload operation; it does not contact R2.
Live source accessibility, account credit eligibility, real uploads, Linux
service installation, corpus size, and query throughput require validation on
the configured cloud instance. In-memory client quotas remain an alpha limit;
public signup, durable billing metering, and multi-instance serving are future
work described in the [API overview](overview.md).

Protocol references: [Node SQLite](https://nodejs.org/api/sqlite.html),
[SQLite FTS5](https://www.sqlite.org/fts5.html),
[robots exclusion](https://www.rfc-editor.org/rfc/rfc9309.html), and
[Cloudflare R2 with rclone](https://developers.cloudflare.com/r2/examples/rclone/).
Human Machine Shift+M