> Source: https://vitna.ai/docs/search/indexes > Raw Markdown: https://vitna.ai/docs/search/indexes.md > Corpus index: https://vitna.ai/llms.txt # Vitna Search Vertical Indexes Vitna Search exposes specialized vertical corpora alongside the default `web` index through a single `index` field on `/v1/search`, `/v1/similar`, and `/v1/extract`: - `web`: Vitna's general web index (default when `index` is omitted). - `research`: Scholarly paper abstracts and metadata (PubMed / MEDLINE, arXiv, bioRxiv, medRxiv, and CC0 OpenAlex non-overlapping works). - `government`: US federal and public-sector records (eCFR, Federal Register, CourtListener US case law, and Common Crawl `.gov` / `.mil` extracts). - `developer`: Open-source package registries, OpenAPI specifications, and public GitHub issues, merged pull requests, and releases (npm, PyPI, crates.io, APIs.guru, GH Archive). - `filings`: US public company filings and XBRL financial statements (SEC EDGAR submissions, company facts, 10-K, 10-Q, 8-K, DEF 14A, S-1, and 20-F filings). - `security`: Vulnerability and threat intelligence (CVE v5, OSV across 48 package/OS ecosystems, CISA KEV, and MITRE ATT&CK). - `clinical`: Clinical research and regulatory drug/device records (ClinicalTrials.gov v2 studies, openFDA drug labels, 510(k) and PMA device clearances, Orange Book, and drug/device recalls). - `patents`: US utility and design patents (USPTO bulk text corpus). - `knowledge`: Clean encyclopedic reference articles (English Wikipedia). - `finance`: Financial filings, insider/institutional holdings, macroeconomic series, and corporate entity registries (Wave 3 scaffold: SEC Form 4/13F/Form D, FRED, IAPD, Companies House, GLEIF LEI, CFPB). - `public-records`: Federal contracts, grants, nonprofit 990s, campaign finance, lobbying, trademarks, and regulatory registries (Wave 3 scaffold). - `health`: Open-access biomedical full text, DailyMed SPL inserts, MedlinePlus, StatPearls, CMS provider data, ICD classifications, and FAERS (Wave 3 scaffold). - `ai`: AI model cards, open datasets, demo spaces, peer-reviewed ML papers, and open AI tool registries (Wave 3 scaffold). - `media`: News wire extracts, books, music releases, podcasts, and media catalog records with 365-day Quickwit retention (Wave 3 scaffold). - `gaming`: Steam catalog/patch notes, Modrinth mods, IGDB/Wikidata game catalogs, and CC-BY-SA gaming wikis (Wave 3 scaffold). - `sports`: Historical play-by-play/box scores and schedules across Retrosheet, nflverse, hoopR, hockeyR, fastf1, and OpenFootball (Wave 3 scaffold). - `listings`: Local places/POIs, public job openings, events, and public property records with 90-day Quickwit retention (Wave 3 scaffold). - `ecommerce`: Schema.org Product/Offer web extracts and Open Food/Beauty/Products Facts with 90-day Quickwit retention (Wave 3 scaffold). - `social`: Public discussions and open federated streams across Bluesky Jetstream, Mastodon, Hacker News, Stack Exchange, and Lemmy with 180-day Quickwit retention (Wave 3 scaffold). - `real-estate`: Zillow Research ZHVI/ZORI/market metrics, Realtor.com Core Inventory, FHFA House Price Index, NYC Property Sales, and UK HM Land Registry Price Paid sales (Wave 4 scaffold). - `jobs`: O*NET occupational profiles and live public ATS job boards across Greenhouse, Lever, and Ashby with 90-day Quickwit retention (Wave 4 scaffold). - `legal`: CourtListener bulk court directories, judicial biographies, opinion clusters, and federal/state dockets (Wave 4 scaffold). - `travel`: OurAirports global directory, OpenFlights airlines/routes, and English Wikivoyage destination guides (Wave 4 scaffold). - `energy-climate`: WRI Global Power Plant Database, Our World in Data energy transition profiles, NOAA Storm Events, and USGS Earthquakes (Wave 4 scaffold). ## API Contract Pass `index: "auto" | "web" | "research" | "developer" | "government" | "filings" | "security" | "clinical" | "patents" | "knowledge" | "finance" | "public-records" | "health" | "ai" | "media" | "gaming" | "sports" | "listings" | "ecommerce" | "social" | "real-estate" | "jobs" | "legal" | "travel" | "energy-climate"` on `POST /v1/search` (`"auto"` classifies the query, fans out to `"web"` plus up to 2 matching configured verticals, and fuses ranked hits with Reciprocal Rank Fusion; omit or pass `"web"` for general search). `/v1/similar` and `/v1/extract` accept `"web"` or a single named vertical index: - `POST /v1/search`: Hybrid BM25 (Quickwit) + dense vector ANN (`vitna-vec`) retrieval fused with Reciprocal Rank Fusion. Supports `domains`, `excludeDomains`, `startPublishedDate`, `endPublishedDate`, `maxResults`, `retrieval` (`keyword`, `semantic`, `hybrid`), and inline `contents`. Query-guided web expansion (`expand: true`) and third-party fallback (`fallback: true`) are rejected on vertical indexes. - `POST /v1/similar`: Resolves the seed document from Quickwit by URL, queries `vitna-vec` `/v1/similar` by `doc_id`, and hydrates ranked hits from Quickwit. - `POST /v1/extract`: Fetches stored full text directly from Quickwit / R2 without touching the origin website. - `GET /v1/indexes`: Lists every registered vertical corpus with `wave`, `status` (`"serving"`, `"backfilling"`, or `"scaffold"`), `documents`, `vectors`, and `snapshotDate`. - `GET /v1/status`: Reports `indexes.<name>` with `wave`, `status` (`"serving"`, `"backfilling"`, or `"scaffold"`), `ready`, `documents`, `vectors`, `snapshotDate`, and `semantic` for each configured vertical. When a requested vertical is unconfigured (`"scaffold"`), unavailable, or still `"backfilling"` (`0` published documents) on the serving node, `/v1/search`, `/v1/similar`, and `/v1/extract` return HTTP `503` with `error.code = "index_unavailable"` before charging client quota. ## Serving Architecture Vertical indexes separate storage from compute so all 140M+ documents serve from a single OVH `b3-16` (4 vCPU, 15 GiB RAM, 100 GB NVMe) behind the private vRack (`10.1.0.0/16`): 1. **Cloudflare R2 (`vitna-index`)**: Source of truth for canonical `.jsonl.gz` source shards (`sources/<index>/<snapshot>/`), Quickwit BM25 + zstd doc-store splits (`quickwit/<index>/`), and versioned `vitna-vec` binary artifacts (`vectors/<index>/<snapshot>/`). 2. **Quickwit (`10.1.2.86:7280`)**: Stateless Rust/Tantivy lexical search and document hydration reading split metadata and posting lists directly from R2 with bounded local RAM/NVMe split caches. 3. **`vitna-vec` (`10.1.2.86:8798`)**: Compact Rust ANN server (`vec/`) using shared-mmap 1-bit binary IVF lists for RAM-efficient candidate generation, slot-ordered `int8` or `sq4` NVMe vector rescoring, and a binary-searchable `idmap.bin` for `/v1/similar` lookups. 4. **Vitna Search Web (`40.160.7.54` / `10.1.0.94`)**: Terminates public TLS, embeds queries with local `vitna-embed` (`bge-small-en-v1.5`), fans out over private vRack (`<1 ms` RTT) to `10.1.2.86:7280` and `10.1.2.86:8798`, fuses results, and applies Vitna's standard response envelope. Firewall rules on `10.1.2.86` restrict TCP `7280` and `8798` exclusively to `10.1.0.94` on the private vRack interface (`ens4`). ## Deduplication and Retention 1. **Unique-Upload Guard (`DedupStore`)**: Both Node (`dedup-store.mjs`) and Python (`dedup_store.py`) shard writers, Quickwit R2 watchers, and `embed_shards.py` back every corpus with a SQLite `WITHOUT ROWID` `seen_docs` index (plus optional 128-bit `content_hash` deduplication). Candidate records are checked read-only first and committed atomically only after R2 upload, Quickwit ingestion, or atomic `.ids`/`.f16` writes succeed. 2. **Wave-3/4 Quickwit Retention**: High-churn indexes apply automatic Quickwit retention policies in `deploy/vitna-indexes/index-*.yaml` (`90 days` on `listings`, `ecommerce`, and `jobs`, `180 days` on `social`, and `365 days` on `media`). ## Operator Commands Credentials are split by role: `/etc/vitna-indexes/reader.env` (read-only R2 token for serving/sync) and `/etc/vitna-indexes/writer.env` (write R2 token for ingestion/GPU build). Never place write credentials on public-facing web nodes. ### Backfills ```bash # Research (PubMed baseline + updates + arXiv + bioRxiv + medRxiv, plus optional OpenAlex top-up) npm run index:research:backfill -- --snapshot 20261120 --concurrency 3 npm run index:research:backfill -- --snapshot 20261120 --stages openalex --openalex-limit 15000000 npm run index:research:ingest -- --snapshot 20261120 --source pubmed-updates # Government (eCFR + Federal Register) npm run index:gov:backfill -- --snapshot 20261120 --stages ecfr,federal-register # Developer (APIs.guru OpenAPI, bulk PyPI/npm/crates.io registries, and GH Archive issues/PRs/releases) npm run index:dev:backfill -- --snapshot 20261120 --stages openapi,gh-archive --gh-from 2025-01-01 npm run index:dev:packages -- --snapshot 20261120 --stages crates,pypi,npm # Resumable R2-to-Quickwit watch ingest for AWS bulk shards npm run index:quickwit:ingest -- --index vitna-research --prefix sources/research/20261120/openalex/ --watch # Build the native Rust vitna-vec binary (release mode) npm run index:vec:build ``` ### AWS `us-east-1` Bulk Processing & GPU Vector Builds In-region AWS Spot instances read `s3://commoncrawl`, `s3://openalex`, and `s3://com-courtlistener-storage` at zero egress cost and build `vitna-vec` artifacts from R2 canonical shards: ```bash # 1. Verify FP16 GPU embedding parity against vitna-embed (cosine >= 0.999) python3 platform/search/indexes/gpu/verify_parity.py --shard /mnt/nvme/sample.jsonl.gz --anchor-url http://127.0.0.1:8795/v1/embeddings # 2. Embed canonical shards on GPU, build vitna-vec (int8 or sq4), and upload to R2 bash platform/search/indexes/gpu/aws_spot_run.sh research 20261120 int8 # 3. Pull verified vectors onto 10.1.2.86 and hot-reload vitna-vec npm run index:vectors:sync -- pull --corpus research --snapshot 20261120 --dir /var/lib/vitna-indexes/vec/research --reload-url http://10.1.2.86:8798 ``` ### Enabling Verticals on `vitna-search-web` Configure these environment variables on `vitna-search-web`: ```bash VITNA_SEARCH_QUICKWIT_URL=http://10.1.2.86:7280 VITNA_SEARCH_VEC_URL=http://10.1.2.86:8798 VITNA_SEARCH_VERTICAL_INDEXES=research,government,developer ```