Get started

Vitna Search Vertical Indexes

Source

Vitna Search exposes specialized vertical corpora alongside the default web index through a single index field on /v1/search, /v1/similar, and /v1/extract:

  • web: Vitna's general web index (default when index is omitted).
  • research: Scholarly paper abstracts and metadata (PubMed / MEDLINE, arXiv, bioRxiv, medRxiv, and CC0 OpenAlex non-overlapping works).
  • government: US federal and public-sector records (eCFR, Federal Register, CourtListener US case law, and Common Crawl .gov / .mil extracts).
  • developer: Open-source package registries, OpenAPI specifications, and public GitHub issues, merged pull requests, and releases (npm, PyPI, crates.io, APIs.guru, GH Archive).
  • filings: US public company filings and XBRL financial statements (SEC EDGAR submissions, company facts, 10-K, 10-Q, 8-K, DEF 14A, S-1, and 20-F filings).
  • security: Vulnerability and threat intelligence (CVE v5, OSV across 48 package/OS ecosystems, CISA KEV, and MITRE ATT&CK).
  • clinical: Clinical research and regulatory drug/device records (ClinicalTrials.gov v2 studies, openFDA drug labels, 510(k) and PMA device clearances, Orange Book, and drug/device recalls).
  • patents: US utility and design patents (USPTO bulk text corpus).
  • knowledge: Clean encyclopedic reference articles (English Wikipedia).
  • finance: Financial filings, insider/institutional holdings, macroeconomic series, and corporate entity registries (Wave 3 scaffold: SEC Form 4/13F/Form D, FRED, IAPD, Companies House, GLEIF LEI, CFPB).
  • public-records: Federal contracts, grants, nonprofit 990s, campaign finance, lobbying, trademarks, and regulatory registries (Wave 3 scaffold).
  • health: Open-access biomedical full text, DailyMed SPL inserts, MedlinePlus, StatPearls, CMS provider data, ICD classifications, and FAERS (Wave 3 scaffold).
  • ai: AI model cards, open datasets, demo spaces, peer-reviewed ML papers, and open AI tool registries (Wave 3 scaffold).
  • media: News wire extracts, books, music releases, podcasts, and media catalog records with 365-day Quickwit retention (Wave 3 scaffold).
  • gaming: Steam catalog/patch notes, Modrinth mods, IGDB/Wikidata game catalogs, and CC-BY-SA gaming wikis (Wave 3 scaffold).
  • sports: Historical play-by-play/box scores and schedules across Retrosheet, nflverse, hoopR, hockeyR, fastf1, and OpenFootball (Wave 3 scaffold).
  • listings: Local places/POIs, public job openings, events, and public property records with 90-day Quickwit retention (Wave 3 scaffold).
  • ecommerce: Schema.org Product/Offer web extracts and Open Food/Beauty/Products Facts with 90-day Quickwit retention (Wave 3 scaffold).
  • social: Public discussions and open federated streams across Bluesky Jetstream, Mastodon, Hacker News, Stack Exchange, and Lemmy with 180-day Quickwit retention (Wave 3 scaffold).
  • real-estate: Zillow Research ZHVI/ZORI/market metrics, Realtor.com Core Inventory, FHFA House Price Index, NYC Property Sales, and UK HM Land Registry Price Paid sales (Wave 4 scaffold).
  • jobs: O*NET occupational profiles and live public ATS job boards across Greenhouse, Lever, and Ashby with 90-day Quickwit retention (Wave 4 scaffold).
  • legal: CourtListener bulk court directories, judicial biographies, opinion clusters, and federal/state dockets (Wave 4 scaffold).
  • travel: OurAirports global directory, OpenFlights airlines/routes, and English Wikivoyage destination guides (Wave 4 scaffold).
  • energy-climate: WRI Global Power Plant Database, Our World in Data energy transition profiles, NOAA Storm Events, and USGS Earthquakes (Wave 4 scaffold).

API Contract

Pass index: "auto" | "web" | "research" | "developer" | "government" | "filings" | "security" | "clinical" | "patents" | "knowledge" | "finance" | "public-records" | "health" | "ai" | "media" | "gaming" | "sports" | "listings" | "ecommerce" | "social" | "real-estate" | "jobs" | "legal" | "travel" | "energy-climate" on POST /v1/search ("auto" classifies the query, fans out to "web" plus up to 2 matching configured verticals, and fuses ranked hits with Reciprocal Rank Fusion; omit or pass "web" for general search). /v1/similar and /v1/extract accept "web" or a single named vertical index:

  • POST /v1/search: Hybrid BM25 (Quickwit) + dense vector ANN (vitna-vec) retrieval fused with Reciprocal Rank Fusion. Supports domains, excludeDomains, startPublishedDate, endPublishedDate, maxResults, retrieval (keyword, semantic, hybrid), and inline contents. Query-guided web expansion (expand: true) and third-party fallback (fallback: true) are rejected on vertical indexes.
  • POST /v1/similar: Resolves the seed document from Quickwit by URL, queries vitna-vec /v1/similar by doc_id, and hydrates ranked hits from Quickwit.
  • POST /v1/extract: Fetches stored full text directly from Quickwit / R2 without touching the origin website.
  • GET /v1/indexes: Lists every registered vertical corpus with wave, status ("serving", "backfilling", or "scaffold"), documents, vectors, and snapshotDate.
  • GET /v1/status: Reports indexes.<name> with wave, status ("serving", "backfilling", or "scaffold"), ready, documents, vectors, snapshotDate, and semantic for each configured vertical.

When a requested vertical is unconfigured ("scaffold"), unavailable, or still "backfilling" (0 published documents) on the serving node, /v1/search, /v1/similar, and /v1/extract return HTTP 503 with error.code = "index_unavailable" before charging client quota.

Serving Architecture

Vertical indexes separate storage from compute so all 140M+ documents serve from a single OVH b3-16 (4 vCPU, 15 GiB RAM, 100 GB NVMe) behind the private vRack (10.1.0.0/16):

  1. Cloudflare R2 (`vitna-index`): Source of truth for canonical .jsonl.gz source shards (sources/<index>/<snapshot>/), Quickwit BM25 + zstd doc-store splits (quickwit/<index>/), and versioned vitna-vec binary artifacts (vectors/<index>/<snapshot>/).
  2. Quickwit (`10.1.2.86:7280`): Stateless Rust/Tantivy lexical search and document hydration reading split metadata and posting lists directly from R2 with bounded local RAM/NVMe split caches.
  3. `vitna-vec` (`10.1.2.86:8798`): Compact Rust ANN server (vec/) using shared-mmap 1-bit binary IVF lists for RAM-efficient candidate generation, slot-ordered int8 or sq4 NVMe vector rescoring, and a binary-searchable idmap.bin for /v1/similar lookups.
  4. Vitna Search Web (`40.160.7.54` / `10.1.0.94`): Terminates public TLS, embeds queries with local vitna-embed (bge-small-en-v1.5), fans out over private vRack (<1 ms RTT) to 10.1.2.86:7280 and 10.1.2.86:8798, fuses results, and applies Vitna's standard response envelope.

Firewall rules on 10.1.2.86 restrict TCP 7280 and 8798 exclusively to 10.1.0.94 on the private vRack interface (ens4).

Deduplication and Retention

  1. Unique-Upload Guard (`DedupStore`): Both Node (dedup-store.mjs) and Python (dedup_store.py) shard writers, Quickwit R2 watchers, and embed_shards.py back every corpus with a SQLite WITHOUT ROWID seen_docs index (plus optional 128-bit content_hash deduplication). Candidate records are checked read-only first and committed atomically only after R2 upload, Quickwit ingestion, or atomic .ids/.f16 writes succeed.
  2. Wave-3/4 Quickwit Retention: High-churn indexes apply automatic Quickwit retention policies in deploy/vitna-indexes/index-*.yaml (90 days on listings, ecommerce, and jobs, 180 days on social, and 365 days on media).

Operator Commands

Credentials are split by role: /etc/vitna-indexes/reader.env (read-only R2 token for serving/sync) and /etc/vitna-indexes/writer.env (write R2 token for ingestion/GPU build). Never place write credentials on public-facing web nodes.

Backfills

bash
# Research (PubMed baseline + updates + arXiv + bioRxiv + medRxiv, plus optional OpenAlex top-up)
npm run index:research:backfill -- --snapshot 20261120 --concurrency 3
npm run index:research:backfill -- --snapshot 20261120 --stages openalex --openalex-limit 15000000
npm run index:research:ingest -- --snapshot 20261120 --source pubmed-updates

# Government (eCFR + Federal Register)
npm run index:gov:backfill -- --snapshot 20261120 --stages ecfr,federal-register

# Developer (APIs.guru OpenAPI, bulk PyPI/npm/crates.io registries, and GH Archive issues/PRs/releases)
npm run index:dev:backfill -- --snapshot 20261120 --stages openapi,gh-archive --gh-from 2025-01-01
npm run index:dev:packages -- --snapshot 20261120 --stages crates,pypi,npm

# Resumable R2-to-Quickwit watch ingest for AWS bulk shards
npm run index:quickwit:ingest -- --index vitna-research --prefix sources/research/20261120/openalex/ --watch

# Build the native Rust vitna-vec binary (release mode)
npm run index:vec:build

AWS us-east-1 Bulk Processing & GPU Vector Builds

In-region AWS Spot instances read s3://commoncrawl, s3://openalex, and s3://com-courtlistener-storage at zero egress cost and build vitna-vec artifacts from R2 canonical shards:

bash
# 1. Verify FP16 GPU embedding parity against vitna-embed (cosine >= 0.999)
python3 platform/search/indexes/gpu/verify_parity.py --shard /mnt/nvme/sample.jsonl.gz --anchor-url http://127.0.0.1:8795/v1/embeddings

# 2. Embed canonical shards on GPU, build vitna-vec (int8 or sq4), and upload to R2
bash platform/search/indexes/gpu/aws_spot_run.sh research 20261120 int8

# 3. Pull verified vectors onto 10.1.2.86 and hot-reload vitna-vec
npm run index:vectors:sync -- pull --corpus research --snapshot 20261120 --dir /var/lib/vitna-indexes/vec/research --reload-url http://10.1.2.86:8798

Enabling Verticals on vitna-search-web

Configure these environment variables on vitna-search-web:

bash
VITNA_SEARCH_QUICKWIT_URL=http://10.1.2.86:7280
VITNA_SEARCH_VEC_URL=http://10.1.2.86:8798
VITNA_SEARCH_VERTICAL_INDEXES=research,government,developer