Get started

Grounding check

Source

platform/search/grounding.mjs decides, segment by segment, whether an answer a model wrote from Vitna Search evidence is supported by the verbatim passages it cites. It is deterministic and lexical: it uses no model and no network, and it checks that what a sentence states occurs in the cited passages, not what the sentence means.

Where it sits

Search supplies the evidence. Source brief research returns verbatim passages, each with a citation id, a source, and code-point offsets into the stored text version read for that source. The answer is written outside Search, by a model the caller runs, from those passages only, citing them with inline markers such as [c1]. The verifier then reads the evidence and the answer together and says which segments the evidence supports, and returns the supported text alone.

It runs where the answer is written, in the caller's process, reads the two values it is handed, and sends nothing anywhere. The file imports nothing and uses no regular-expression lookbehind, so it can be copied whole into code for Node, Deno, Bun, Workers or a browser; it is not yet published in the JavaScript or Python SDK.

js
import { verifyAnswer } from "./grounding.mjs";

const evidence = await response.json(); // the /v1/research response, as returned
const result = verifyAnswer(evidence, answer, { minOverlap: 0.45 });

minOverlap is optional and defaults to GROUNDING_MIN_OVERLAP, the measured 0.45. The module also exports parseCitedAnswer(text), which returns the answer's segments as {text, citationIds, kind} with markers removed, and the INSUFFICIENT_EVIDENCE sentinel.

supported means the cited passages hold the segment's figures, quotations and names and most of its words. It does not mean the segment is true: passages are publisher text, and the checks below cannot see a reversed meaning.

Evidence

The verifier reads sources, citations, and the passage texts quoted in overview and sections[].passages. A top-level passages list, or a text on the citation itself, is read too. Several research responses merged into one pack work the same way, provided their source and citation ids are renumbered to stay unique. A citation is usable only when its id is unique, offsetUnit is unicode_code_points, start and end are integers with start below end, sourceId names a listed source, exactly one passage text is quoted for it, and that text is end - start code points long. When a source carries its text, the passage must also be that slice of it.

A citation that fails is listed in evidence.invalid with its reason: duplicate_id, offset_unit, offsets, unknown_source, missing_text, conflicting_text, too_long, length_mismatch or source_text_mismatch. A segment citing it is refused with invalid_citation instead of being checked against it.

Answer

The answer is Markdown text with inline markers, or a list of segments, [{ text, citationIds }].

markdown
## Opening

It opened to pedestrians on May 27, 1937 [c1]. Cars crossed on May 29, 1937 [c1].

Text is parsed into segments: each heading (a # line, or a line that is only bold text), and each sentence of a paragraph or list item. Sentences end at ., ! or ? followed by a capital, a digit or a quotation mark, except after common abbreviations such as Dr., e.g. and U.S., and after a single initial. Markers just after a sentence's full stop belong to that sentence, and a line holding only markers belongs to the sentence before it. A bracket is a marker when each comma-separated part is an id in the evidence or has the shape of one, letters then digits or digits alone, so [1] is refused as an unknown id rather than read as the figure 1. A link's bracketed text is not a marker. Fences and horizontal rules are markup; the text inside a fence is read as a paragraph.

A segment list is used as given, with any inline markers in a segment's text added to its ids. A segment whose text opens with #, or whose kind is heading, is a heading; one opening with a list marker is a list item.

When the passages do not answer the question, the model should reply with INSUFFICIENT_EVIDENCE alone. An answer that is only that word, allowing surrounding whitespace, quotes, backticks or a full stop, has no segments: coverage.status is insufficient and coverage.abstained is true. Beside other text the word is an uncited sentence and is refused.

Output

json
{
  "segments": [
    {"index": 0, "kind": "heading", "text": "Opening", "citationIds": [], "claim": false, "supported": true, "reasons": [], "details": [], "overlap": null},
    {"index": 1, "kind": "sentence", "text": "It opened to pedestrians on May 27, 1937.", "citationIds": ["c1"], "claim": true, "supported": true, "reasons": [], "details": [], "overlap": 1},
    {"index": 2, "kind": "sentence", "text": "Cars crossed on May 29, 1937.", "citationIds": ["c1"], "claim": true, "supported": false,
     "reasons": ["unsupported_number", "low_overlap"], "details": [{"reason": "unsupported_number", "text": "May 29, 1937"}, {"reason": "low_overlap", "text": "cars crossed"}], "overlap": 0}
  ],
  "supportedText": "## Opening\n\nIt opened to pedestrians on May 27, 1937.",
  "coverage": {"status": "partial", "segments": 3, "supported": 2, "unsupported": 1, "claims": 2, "supportedClaims": 1, "abstained": false},
  "evidence": {"citations": 1, "usable": 1, "invalid": []},
  "minOverlap": 0.45
}
FieldMeaning
segments[].textThe segment with its markers, list marker and emphasis removed
segments[].kindheading, lead_in, connective, closing, item or sentence
segments[].claimfalse for the four structural kinds, which need no citation
segments[].supportedEvery check passed
segments[].reasonsReason codes, in the order of the table below
segments[].detailsEach failing token as {reason, text}; for low_overlap, the content words no cited passage holds
segments[].overlapShare of the segment's content words found in its cited passages, or null
supportedTextThe supported segments with their headings and list markers, citation markers removed
coverage.statusmatched, partial or insufficient, as research uses them
coverage.supported / unsupportedSegments supported and refused, structural ones included
coverage.claims / supportedClaimsSegments that are claims, and those supported
coverage.abstainedThe answer was only INSUFFICIENT_EVIDENCE

coverage.status is matched when at least one claim is supported and no segment is refused, partial when some claim is supported and some segment is refused, and insufficient when no claim is supported. A refused heading or lead-in counts as refused. In supportedText a heading is kept only over a supported claim, a lead-in only before one, and a connective or closing sentence only when some claim survived. The segments keep their citationIds, so a reader can show the supported text with its sources.

Checks

A claim is any segment that is not structural. Each claim is checked against the usable passages it cites, and every failure is reported, not only the first.

ReasonWhen
missing_citationThe claim cites nothing
unknown_citationIt cites an id the evidence does not hold, such as an invented c9 or a source id like s3
invalid_citationIt cites a citation that failed the evidence checks
unsupported_numberA number, percentage, money amount, year or date is in no cited passage
unsupported_quoteA quoted string is in no cited passage
unsupported_nameA capitalised name is in no cited passage
low_overlapFewer than 0.45 of its content words occur in the cited passages

One unknown or invalid id refuses a claim even beside a good one. The content checks still run against the usable ids, so every problem is listed.

Numbers are compared as values. 1,200,000, 1.2 million and $1.2m are the same value; a currency symbol or word must match; a percentage must meet a percentage; a bare number may meet a passage number of any kind. Spelled numbers count (eleven, twenty-five, a thousand), and a lone one counts only as a quantity, not in one of or no one. A figure hedged with about, roughly, nearly or the like may round the passage's figure to its own precision, so about 40% meets 38.7% but about 99% does not meet 97%. An unhedged figure must match exactly.

Dates are read in their written forms (May 27, 1937, 27 May 1937, 2022-08-18, August 2022, May 27) and compared as dates. A date is supported by one at least as specific, so May 1937 meets May 27, 1937, and a passage's 12 March meets 12 March 2021 when the passage holds 2021 elsewhere.

Quotations in straight or curly double quotes, guillemets, and single quotes that are not apostrophes must occur as a word sequence in one cited passage, ignoring case, punctuation and diacritics.

Names are runs of capitalised words, joined across of, de, van and the like, matched as a word sequence in one cited passage, ignoring case and inflection. Month and weekday names are left to the date check. A sentence's first word joins a name only when the evidence writes it capitalised mid-sentence, or when it starts a longer run and is not an ordinary word of the cited passages, so Brooklyn Bridge is checked whole while Typically is not a name. Headings are title case, so their capitals are not read as names.

Content words are the segment's words less function words, discourse words such as overall and according, numbers, and month names, after a light suffix stripper on both sides (spans meets spanning, released meets releases). Diacritics and letters such as ł fold, so Sklodowska meets Skłodowska.

Structural text

Four kinds of segment make no claim of their own and need no citation, each only while it is 20 words or shorter.

KindWhat it is
headingA Markdown heading, or a bold-only line
lead_inAn uncited sentence ending in a colon, such as Bakers watch a few signals:
connectiveAn uncited segment with no content word, figure or quotation, such as Hope this helps!
closingThe answer's last segment, uncited, that passes every check, overlap included, against the passages the answer's supported claims cited

Structural text still answers for what it states. A number, quotation or name in a heading, lead-in or connective must occur in the passages it cites or, citing none, somewhere in the evidence; so # Opened in 1938 is refused when no passage says 1938. A closing sentence that fails any check is an uncited claim and is refused with missing_citation. A list item is a claim like any sentence.

Measured precision and recall

bash
npm run search:grounding-evaluate

The command verifies twelve labelled answers in platform/search/grounding-fixture.mjs and writes .search-data/reports/grounding.json (or --output path) with the threshold sweep, the results per split and per kind, and every misjudged segment. Each answer is written out as the Markdown a model would return and verified whole, so the parser is measured with the checks; a pack that does not parse into one segment per labelled row fails the run.

The fixture holds 230 segments over twelve evidence packs shaped like research responses: 115 supported and 115 adversarial. Six packs choose the threshold and six held-out packs only measure it. The threshold is the lowest with the best F0.5 of predicting supported on the development packs, precision weighted over recall because a passed unsupported sentence costs a reader more than a refused one. On the development packs F0.5 is 0.866 at both 0.45 and 0.5. Recall starts to fall at 0.55, and precision rises again only from 0.9, where the overlap refuses some named limits along with a third of the supported segments. The module's 0.45 is that choice, and a test holds the two equal.

PacksSegmentsPrecisionRecall
Development1140.8381.000
Held out1160.8710.931
All packs2300.8540.965
All packs, named limits left out2111.0000.965

Precision is held below 1 by the named limits below, which the adversarial half includes on purpose. Every other adversarial segment is refused.

KindLabelSegmentsJudged as labelled
verbatimSupported3131
paraphraseSupported3433
synthesisSupported66
headingSupported1212
lead_inSupported77
list_itemSupported1313
closingSupported129
changed_numberAdversarial3030
changed_dateAdversarial88
swapped_nameAdversarial1212
uncitedAdversarial1212
invented_idAdversarial1212
mergedAdversarial1212
wrong_quoteAdversarial77
embellishedAdversarial33
negationNamed limit60
comparisonNamed limit50
role_swapNamed limit50
substitutionNamed limit30

The four supported segments refused are all held out: three closing sentences that add an evaluation the evidence does not make (the budget grows modestly, two doses give strong protection, pressure, dose and time decide the shot), and one paraphrase sharing two of five content words with its passage (Its net loss shrank to $6.1 million, compared with $9.8 million a year before).

The fixture is synthetic and was written by one author who knew the checks. The two Calder County and Northwind Robotics packs are invented; the rest follow well-known public facts, but each pack asserts only what its own passages say. It measures the checks on the failure kinds named here, not a model's real error rate, and a model's errors will include subtler ones.

Limits

A lexical check confirms that tokens occur, not what a sentence does with them. These pass, and the fixture counts them as passed unsupported segments:

  • Negation. The bridge is not painted International Orange keeps every word of its passage; not, no and never are function words to the overlap, and no other check reads polarity.
  • Reversed comparison or change. falls for rises, cooler for warmer, widened for narrowed: one changed word in a sentence otherwise drawn from the passage leaves the overlap high.
  • Swapped roles. Two entities or two figures of the same passage exchanged, as in Aldrin became the first person to step onto the lunar surface, or $15.9 million, up from $18.2 million for the reverse. That includes a figure moved to another unit or quantity, such as 21.5 pounds where the passage has 47.5 pounds (21.5 kg): units are not compared.
  • Substitution. One ordinary word replaced in a long sentence, such as stroma for thylakoid membranes or second for first. Spelled ordinals are words, not numbers.

Other edges, measured or known:

  • A count in a lead-in, such as Two changes matter most:, must occur in the evidence, and can be met by coincidence. Both of the fixture's passed that way, one on The two astronauts and the other on the day of October 2.
  • A paraphrase that shares too few words with its passage is refused, and so is a closing sentence that evaluates rather than restates.
  • Only hedged figures round. 1.2 million without about must match exactly.
  • Stopwords, number words, month names and the suffix stripper are English.
  • The verifier trusts a passage as far as its length, and its slice when the source text is supplied. It fetches nothing, so it cannot tell whether the passage is still what the page says; a later crawl can change the offsets.