> Source: https://vitna.ai/docs/search/indexes
> Raw Markdown: https://vitna.ai/docs/search/indexes.md
> Corpus index: https://vitna.ai/llms.txt

# Vitna Search Vertical Indexes

Vitna Search exposes specialized vertical corpora alongside the default `web`
index through a single `index` field on `/v1/search`, `/v1/similar`, and
`/v1/extract`:

- `web`: Vitna's general web index (default when `index` is omitted).
- `research`: Scholarly paper abstracts and metadata (PubMed / MEDLINE, arXiv,
  bioRxiv, medRxiv, and CC0 OpenAlex non-overlapping works).
- `government`: US federal and public-sector records (eCFR, Federal Register,
  CourtListener US case law, and Common Crawl `.gov` / `.mil` extracts).
- `developer`: Open-source package registries, OpenAPI specifications, and
  public GitHub issues, merged pull requests, and releases (npm, PyPI,
  crates.io, APIs.guru, GH Archive).
- `filings`: US public company filings and XBRL financial statements (SEC EDGAR
  submissions, company facts, 10-K, 10-Q, 8-K, DEF 14A, S-1, and 20-F filings).
- `security`: Vulnerability and threat intelligence (CVE v5, OSV across 48
  package/OS ecosystems, CISA KEV, and MITRE ATT&CK).
- `clinical`: Clinical research and regulatory drug/device records
  (ClinicalTrials.gov v2 studies, openFDA drug labels, 510(k) and PMA device
  clearances, Orange Book, and drug/device recalls).
- `patents`: US utility and design patents (USPTO bulk text corpus).
- `knowledge`: Clean encyclopedic reference articles (English Wikipedia).
- `finance`: Financial filings, insider/institutional holdings, macroeconomic series, and corporate entity registries (Wave 3 scaffold: SEC Form 4/13F/Form D, FRED, IAPD, Companies House, GLEIF LEI, CFPB).
- `public-records`: Federal contracts, grants, nonprofit 990s, campaign finance, lobbying, trademarks, and regulatory registries (Wave 3 scaffold).
- `health`: Open-access biomedical full text, DailyMed SPL inserts, MedlinePlus, StatPearls, CMS provider data, ICD classifications, and FAERS (Wave 3 scaffold).
- `ai`: AI model cards, open datasets, demo spaces, peer-reviewed ML papers, and open AI tool registries (Wave 3 scaffold).
- `media`: News wire extracts, books, music releases, podcasts, and media catalog records with 365-day Quickwit retention (Wave 3 scaffold).
- `gaming`: Steam catalog/patch notes, Modrinth mods, IGDB/Wikidata game catalogs, and CC-BY-SA gaming wikis (Wave 3 scaffold).
- `sports`: Historical play-by-play/box scores and schedules across Retrosheet, nflverse, hoopR, hockeyR, fastf1, and OpenFootball (Wave 3 scaffold).
- `listings`: Local places/POIs, public job openings, events, and public property records with 90-day Quickwit retention (Wave 3 scaffold).
- `ecommerce`: Schema.org Product/Offer web extracts and Open Food/Beauty/Products Facts with 90-day Quickwit retention (Wave 3 scaffold).
- `social`: Public discussions and open federated streams across Bluesky Jetstream, Mastodon, Hacker News, Stack Exchange, and Lemmy with 180-day Quickwit retention (Wave 3 scaffold).
- `real-estate`: Zillow Research ZHVI/ZORI/market metrics, Realtor.com Core Inventory, FHFA House Price Index, NYC Property Sales, and UK HM Land Registry Price Paid sales (Wave 4 scaffold).
- `jobs`: O*NET occupational profiles and live public ATS job boards across Greenhouse, Lever, and Ashby with 90-day Quickwit retention (Wave 4 scaffold).
- `legal`: CourtListener bulk court directories, judicial biographies, opinion clusters, and federal/state dockets (Wave 4 scaffold).
- `travel`: OurAirports global directory, OpenFlights airlines/routes, and English Wikivoyage destination guides (Wave 4 scaffold).
- `energy-climate`: WRI Global Power Plant Database, Our World in Data energy transition profiles, NOAA Storm Events, and USGS Earthquakes (Wave 4 scaffold).

## API Contract

Pass `index: "auto" | "web" | "research" | "developer" | "government" | "filings" | "security" | "clinical" | "patents" | "knowledge" | "finance" | "public-records" | "health" | "ai" | "media" | "gaming" | "sports" | "listings" | "ecommerce" | "social" | "real-estate" | "jobs" | "legal" | "travel" | "energy-climate"` on `POST /v1/search` (`"auto"` classifies the query, fans out to `"web"` plus up to 2 matching configured verticals, and fuses ranked hits with Reciprocal Rank Fusion; omit or pass `"web"` for general search). `/v1/similar` and `/v1/extract` accept `"web"` or a single named vertical index:

- `POST /v1/search`: Hybrid BM25 (Quickwit) + dense vector ANN (`vitna-vec`)
  retrieval fused with Reciprocal Rank Fusion. Supports `domains`,
  `excludeDomains`, `startPublishedDate`, `endPublishedDate`, `maxResults`,
  `retrieval` (`keyword`, `semantic`, `hybrid`), and inline `contents`. Query-guided
  web expansion (`expand: true`) and third-party fallback (`fallback: true`)
  are rejected on vertical indexes.
- `POST /v1/similar`: Resolves the seed document from Quickwit by URL, queries
  `vitna-vec` `/v1/similar` by `doc_id`, and hydrates ranked hits from Quickwit.
- `POST /v1/extract`: Fetches stored full text directly from Quickwit / R2
  without touching the origin website.
- `GET /v1/indexes`: Lists every registered vertical corpus with `wave`,
  `status` (`"serving"`, `"backfilling"`, or `"scaffold"`), `documents`,
  `vectors`, and `snapshotDate`.
- `GET /v1/status`: Reports `indexes.<name>` with `wave`, `status`
  (`"serving"`, `"backfilling"`, or `"scaffold"`), `ready`, `documents`,
  `vectors`, `snapshotDate`, and `semantic` for each configured vertical.

When a requested vertical is unconfigured (`"scaffold"`), unavailable, or still
`"backfilling"` (`0` published documents) on the serving node, `/v1/search`,
`/v1/similar`, and `/v1/extract` return HTTP `503` with
`error.code = "index_unavailable"` before charging client quota.

## Serving Architecture

Vertical indexes separate storage from compute so all 140M+ documents serve
from a single OVH `b3-16` (4 vCPU, 15 GiB RAM, 100 GB NVMe) behind the private
vRack (`10.1.0.0/16`):

1. **Cloudflare R2 (`vitna-index`)**: Source of truth for canonical `.jsonl.gz`
   source shards (`sources/<index>/<snapshot>/`), Quickwit BM25 + zstd doc-store
   splits (`quickwit/<index>/`), and versioned `vitna-vec` binary artifacts
   (`vectors/<index>/<snapshot>/`).
2. **Quickwit (`10.1.2.86:7280`)**: Stateless Rust/Tantivy lexical search and
   document hydration reading split metadata and posting lists directly from R2
   with bounded local RAM/NVMe split caches.
3. **`vitna-vec` (`10.1.2.86:8798`)**: Compact Rust ANN server (`vec/`) using
   shared-mmap 1-bit binary IVF lists for RAM-efficient candidate generation,
   slot-ordered `int8` or `sq4` NVMe vector rescoring, and a binary-searchable
   `idmap.bin` for `/v1/similar` lookups.
4. **Vitna Search Web (`40.160.7.54` / `10.1.0.94`)**: Terminates public TLS,
   embeds queries with local `vitna-embed` (`bge-small-en-v1.5`), fans out over
   private vRack (`<1 ms` RTT) to `10.1.2.86:7280` and `10.1.2.86:8798`, fuses
   results, and applies Vitna's standard response envelope.

Firewall rules on `10.1.2.86` restrict TCP `7280` and `8798` exclusively to
`10.1.0.94` on the private vRack interface (`ens4`).

## Deduplication and Retention

1. **Unique-Upload Guard (`DedupStore`)**: Both Node (`dedup-store.mjs`) and
   Python (`dedup_store.py`) shard writers, Quickwit R2 watchers, and
   `embed_shards.py` back every corpus with a SQLite `WITHOUT ROWID` `seen_docs`
   index (plus optional 128-bit `content_hash` deduplication). Candidate
   records are checked read-only first and committed atomically only after R2
   upload, Quickwit ingestion, or atomic `.ids`/`.f16` writes succeed.
2. **Wave-3/4 Quickwit Retention**: High-churn indexes apply automatic Quickwit
   retention policies in `deploy/vitna-indexes/index-*.yaml` (`90 days` on
   `listings`, `ecommerce`, and `jobs`, `180 days` on `social`, and `365 days` on
   `media`).

## Operator Commands

Credentials are split by role: `/etc/vitna-indexes/reader.env` (read-only R2
token for serving/sync) and `/etc/vitna-indexes/writer.env` (write R2 token for
ingestion/GPU build). Never place write credentials on public-facing web nodes.

### Backfills

```bash
# Research (PubMed baseline + updates + arXiv + bioRxiv + medRxiv, plus optional OpenAlex top-up)
npm run index:research:backfill -- --snapshot 20261120 --concurrency 3
npm run index:research:backfill -- --snapshot 20261120 --stages openalex --openalex-limit 15000000
npm run index:research:ingest -- --snapshot 20261120 --source pubmed-updates

# Government (eCFR + Federal Register)
npm run index:gov:backfill -- --snapshot 20261120 --stages ecfr,federal-register

# Developer (APIs.guru OpenAPI, bulk PyPI/npm/crates.io registries, and GH Archive issues/PRs/releases)
npm run index:dev:backfill -- --snapshot 20261120 --stages openapi,gh-archive --gh-from 2025-01-01
npm run index:dev:packages -- --snapshot 20261120 --stages crates,pypi,npm

# Resumable R2-to-Quickwit watch ingest for AWS bulk shards
npm run index:quickwit:ingest -- --index vitna-research --prefix sources/research/20261120/openalex/ --watch

# Build the native Rust vitna-vec binary (release mode)
npm run index:vec:build
```

### AWS `us-east-1` Bulk Processing & GPU Vector Builds

In-region AWS Spot instances read `s3://commoncrawl`, `s3://openalex`, and
`s3://com-courtlistener-storage` at zero egress cost and build `vitna-vec`
artifacts from R2 canonical shards:

```bash
# 1. Verify FP16 GPU embedding parity against vitna-embed (cosine >= 0.999)
python3 platform/search/indexes/gpu/verify_parity.py --shard /mnt/nvme/sample.jsonl.gz --anchor-url http://127.0.0.1:8795/v1/embeddings

# 2. Embed canonical shards on GPU, build vitna-vec (int8 or sq4), and upload to R2
bash platform/search/indexes/gpu/aws_spot_run.sh research 20261120 int8

# 3. Pull verified vectors onto 10.1.2.86 and hot-reload vitna-vec
npm run index:vectors:sync -- pull --corpus research --snapshot 20261120 --dir /var/lib/vitna-indexes/vec/research --reload-url http://10.1.2.86:8798
```

### Enabling Verticals on `vitna-search-web`

Configure these environment variables on `vitna-search-web`:

```bash
VITNA_SEARCH_QUICKWIT_URL=http://10.1.2.86:7280
VITNA_SEARCH_VEC_URL=http://10.1.2.86:8798
VITNA_SEARCH_VERTICAL_INDEXES=research,government,developer
```
