Vitna Search Vertical Indexes
Vitna Search exposes specialized vertical corpora alongside the default web index through a single index field on /v1/search, /v1/similar, and /v1/extract:
web: Vitna's general web index (default whenindexis omitted).research: Scholarly paper abstracts and metadata (PubMed / MEDLINE, arXiv, bioRxiv, medRxiv, and CC0 OpenAlex non-overlapping works).government: US federal and public-sector records (eCFR, Federal Register, CourtListener US case law, and Common Crawl.gov/.milextracts).developer: Open-source package registries, OpenAPI specifications, and public GitHub issues, merged pull requests, and releases (npm, PyPI, crates.io, APIs.guru, GH Archive).filings: US public company filings and XBRL financial statements (SEC EDGAR submissions, company facts, 10-K, 10-Q, 8-K, DEF 14A, S-1, and 20-F filings).security: Vulnerability and threat intelligence (CVE v5, OSV across 48 package/OS ecosystems, CISA KEV, and MITRE ATT&CK).clinical: Clinical research and regulatory drug/device records (ClinicalTrials.gov v2 studies, openFDA drug labels, 510(k) and PMA device clearances, Orange Book, and drug/device recalls).patents: US utility and design patents (USPTO bulk text corpus).knowledge: Clean encyclopedic reference articles (English Wikipedia).finance: Financial filings, insider/institutional holdings, macroeconomic series, and corporate entity registries (Wave 3 scaffold: SEC Form 4/13F/Form D, FRED, IAPD, Companies House, GLEIF LEI, CFPB).public-records: Federal contracts, grants, nonprofit 990s, campaign finance, lobbying, trademarks, and regulatory registries (Wave 3 scaffold).health: Open-access biomedical full text, DailyMed SPL inserts, MedlinePlus, StatPearls, CMS provider data, ICD classifications, and FAERS (Wave 3 scaffold).ai: AI model cards, open datasets, demo spaces, peer-reviewed ML papers, and open AI tool registries (Wave 3 scaffold).media: News wire extracts, books, music releases, podcasts, and media catalog records with 365-day Quickwit retention (Wave 3 scaffold).gaming: Steam catalog/patch notes, Modrinth mods, IGDB/Wikidata game catalogs, and CC-BY-SA gaming wikis (Wave 3 scaffold).sports: Historical play-by-play/box scores and schedules across Retrosheet, nflverse, hoopR, hockeyR, fastf1, and OpenFootball (Wave 3 scaffold).listings: Local places/POIs, public job openings, events, and public property records with 90-day Quickwit retention (Wave 3 scaffold).ecommerce: Schema.org Product/Offer web extracts and Open Food/Beauty/Products Facts with 90-day Quickwit retention (Wave 3 scaffold).social: Public discussions and open federated streams across Bluesky Jetstream, Mastodon, Hacker News, Stack Exchange, and Lemmy with 180-day Quickwit retention (Wave 3 scaffold).real-estate: Zillow Research ZHVI/ZORI/market metrics, Realtor.com Core Inventory, FHFA House Price Index, NYC Property Sales, and UK HM Land Registry Price Paid sales (Wave 4 scaffold).jobs: O*NET occupational profiles and live public ATS job boards across Greenhouse, Lever, and Ashby with 90-day Quickwit retention (Wave 4 scaffold).legal: CourtListener bulk court directories, judicial biographies, opinion clusters, and federal/state dockets (Wave 4 scaffold).travel: OurAirports global directory, OpenFlights airlines/routes, and English Wikivoyage destination guides (Wave 4 scaffold).energy-climate: WRI Global Power Plant Database, Our World in Data energy transition profiles, NOAA Storm Events, and USGS Earthquakes (Wave 4 scaffold).
API Contract
Pass index: "auto" | "web" | "research" | "developer" | "government" | "filings" | "security" | "clinical" | "patents" | "knowledge" | "finance" | "public-records" | "health" | "ai" | "media" | "gaming" | "sports" | "listings" | "ecommerce" | "social" | "real-estate" | "jobs" | "legal" | "travel" | "energy-climate" on POST /v1/search ("auto" classifies the query, fans out to "web" plus up to 2 matching configured verticals, and fuses ranked hits with Reciprocal Rank Fusion; omit or pass "web" for general search). /v1/similar and /v1/extract accept "web" or a single named vertical index:
POST /v1/search: Hybrid BM25 (Quickwit) + dense vector ANN (vitna-vec) retrieval fused with Reciprocal Rank Fusion. Supportsdomains,excludeDomains,startPublishedDate,endPublishedDate,maxResults,retrieval(keyword,semantic,hybrid), and inlinecontents. Query-guided web expansion (expand: true) and third-party fallback (fallback: true) are rejected on vertical indexes.POST /v1/similar: Resolves the seed document from Quickwit by URL, queriesvitna-vec/v1/similarbydoc_id, and hydrates ranked hits from Quickwit.POST /v1/extract: Fetches stored full text directly from Quickwit / R2 without touching the origin website.GET /v1/indexes: Lists every registered vertical corpus withwave,status("serving","backfilling", or"scaffold"),documents,vectors, andsnapshotDate.GET /v1/status: Reportsindexes.<name>withwave,status("serving","backfilling", or"scaffold"),ready,documents,vectors,snapshotDate, andsemanticfor each configured vertical.
When a requested vertical is unconfigured ("scaffold"), unavailable, or still "backfilling" (0 published documents) on the serving node, /v1/search, /v1/similar, and /v1/extract return HTTP 503 with error.code = "index_unavailable" before charging client quota.
Serving Architecture
Vertical indexes separate storage from compute so all 140M+ documents serve from a single OVH b3-16 (4 vCPU, 15 GiB RAM, 100 GB NVMe) behind the private vRack (10.1.0.0/16):
- Cloudflare R2 (`vitna-index`): Source of truth for canonical
.jsonl.gzsource shards (sources/<index>/<snapshot>/), Quickwit BM25 + zstd doc-store splits (quickwit/<index>/), and versionedvitna-vecbinary artifacts (vectors/<index>/<snapshot>/). - Quickwit (`10.1.2.86:7280`): Stateless Rust/Tantivy lexical search and document hydration reading split metadata and posting lists directly from R2 with bounded local RAM/NVMe split caches.
- `vitna-vec` (`10.1.2.86:8798`): Compact Rust ANN server (
vec/) using shared-mmap 1-bit binary IVF lists for RAM-efficient candidate generation, slot-orderedint8orsq4NVMe vector rescoring, and a binary-searchableidmap.binfor/v1/similarlookups. - Vitna Search Web (`40.160.7.54` / `10.1.0.94`): Terminates public TLS, embeds queries with local
vitna-embed(bge-small-en-v1.5), fans out over private vRack (<1 msRTT) to10.1.2.86:7280and10.1.2.86:8798, fuses results, and applies Vitna's standard response envelope.
Firewall rules on 10.1.2.86 restrict TCP 7280 and 8798 exclusively to 10.1.0.94 on the private vRack interface (ens4).
Deduplication and Retention
- Unique-Upload Guard (`DedupStore`): Both Node (
dedup-store.mjs) and Python (dedup_store.py) shard writers, Quickwit R2 watchers, andembed_shards.pyback every corpus with a SQLiteWITHOUT ROWIDseen_docsindex (plus optional 128-bitcontent_hashdeduplication). Candidate records are checked read-only first and committed atomically only after R2 upload, Quickwit ingestion, or atomic.ids/.f16writes succeed. - Wave-3/4 Quickwit Retention: High-churn indexes apply automatic Quickwit retention policies in
deploy/vitna-indexes/index-*.yaml(90 daysonlistings,ecommerce, andjobs,180 daysonsocial, and365 daysonmedia).
Operator Commands
Credentials are split by role: /etc/vitna-indexes/reader.env (read-only R2 token for serving/sync) and /etc/vitna-indexes/writer.env (write R2 token for ingestion/GPU build). Never place write credentials on public-facing web nodes.
Backfills
# Research (PubMed baseline + updates + arXiv + bioRxiv + medRxiv, plus optional OpenAlex top-up)
npm run index:research:backfill -- --snapshot 20261120 --concurrency 3
npm run index:research:backfill -- --snapshot 20261120 --stages openalex --openalex-limit 15000000
npm run index:research:ingest -- --snapshot 20261120 --source pubmed-updates
# Government (eCFR + Federal Register)
npm run index:gov:backfill -- --snapshot 20261120 --stages ecfr,federal-register
# Developer (APIs.guru OpenAPI, bulk PyPI/npm/crates.io registries, and GH Archive issues/PRs/releases)
npm run index:dev:backfill -- --snapshot 20261120 --stages openapi,gh-archive --gh-from 2025-01-01
npm run index:dev:packages -- --snapshot 20261120 --stages crates,pypi,npm
# Resumable R2-to-Quickwit watch ingest for AWS bulk shards
npm run index:quickwit:ingest -- --index vitna-research --prefix sources/research/20261120/openalex/ --watch
# Build the native Rust vitna-vec binary (release mode)
npm run index:vec:buildAWS us-east-1 Bulk Processing & GPU Vector Builds
In-region AWS Spot instances read s3://commoncrawl, s3://openalex, and s3://com-courtlistener-storage at zero egress cost and build vitna-vec artifacts from R2 canonical shards:
# 1. Verify FP16 GPU embedding parity against vitna-embed (cosine >= 0.999)
python3 platform/search/indexes/gpu/verify_parity.py --shard /mnt/nvme/sample.jsonl.gz --anchor-url http://127.0.0.1:8795/v1/embeddings
# 2. Embed canonical shards on GPU, build vitna-vec (int8 or sq4), and upload to R2
bash platform/search/indexes/gpu/aws_spot_run.sh research 20261120 int8
# 3. Pull verified vectors onto 10.1.2.86 and hot-reload vitna-vec
npm run index:vectors:sync -- pull --corpus research --snapshot 20261120 --dir /var/lib/vitna-indexes/vec/research --reload-url http://10.1.2.86:8798Enabling Verticals on vitna-search-web
Configure these environment variables on vitna-search-web:
VITNA_SEARCH_QUICKWIT_URL=http://10.1.2.86:7280
VITNA_SEARCH_VEC_URL=http://10.1.2.86:8798
VITNA_SEARCH_VERTICAL_INDEXES=research,government,developer