vitna.ai/docs/search/grounding Machine view
Raw .md llms.txt
> Source: https://vitna.ai/docs/search/grounding
> Raw Markdown: https://vitna.ai/docs/search/grounding.md
> Corpus index: https://vitna.ai/llms.txt

# Grounding check

`platform/search/grounding.mjs` decides, segment by segment, whether an answer
a model wrote from Vitna Search evidence is supported by the verbatim passages
it cites. It is deterministic and lexical: it uses no model and no network, and
it checks that what a sentence states occurs in the cited passages, not what
the sentence means.

## Where it sits

Search supplies the evidence. [Source brief research](overview.md#source-brief-research)
returns verbatim passages, each with a citation id, a source, and code-point
offsets into the stored text version read for that source. The answer is
written outside Search, by a model the caller runs, from those passages only,
citing them with inline markers such as `[c1]`. The verifier then reads the
evidence and the answer together and says which segments the evidence
supports, and returns the supported text alone.

It runs where the answer is written, in the caller's process, reads the two
values it is handed, and sends nothing anywhere. The file imports nothing and
uses no regular-expression lookbehind, so it can be copied whole into code
for Node, Deno, Bun, Workers or a browser; it is not yet published in the
JavaScript or Python SDK.

```js
import { verifyAnswer } from "./grounding.mjs";

const evidence = await response.json(); // the /v1/research response, as returned
const result = verifyAnswer(evidence, answer, { minOverlap: 0.45 });
```

`minOverlap` is optional and defaults to `GROUNDING_MIN_OVERLAP`, the
measured `0.45`. The module also exports `parseCitedAnswer(text)`, which
returns the answer's segments as `{text, citationIds, kind}` with markers
removed, and the `INSUFFICIENT_EVIDENCE` sentinel.

`supported` means the cited passages hold the segment's figures, quotations
and names and most of its words. It does not mean the segment is true:
passages are publisher text, and the checks below cannot see a reversed
meaning.

## Evidence

The verifier reads `sources`, `citations`, and the passage texts quoted in
`overview` and `sections[].passages`. A top-level `passages` list, or a
`text` on the citation itself, is read too. Several research responses merged
into one pack work the same way, provided their source and citation ids are
renumbered to stay unique. A citation is usable only when its
id is unique, `offsetUnit` is `unicode_code_points`, `start` and `end` are
integers with `start` below `end`, `sourceId` names a listed source, exactly
one passage text is quoted for it, and that text is `end - start` code points
long. When a source carries its `text`, the passage must also be that slice of
it.

A citation that fails is listed in `evidence.invalid` with its reason:
`duplicate_id`, `offset_unit`, `offsets`, `unknown_source`, `missing_text`,
`conflicting_text`, `too_long`, `length_mismatch` or `source_text_mismatch`. A
segment citing it is refused with `invalid_citation` instead of being checked
against it.

## Answer

The answer is Markdown text with inline markers, or a list of segments,
`[{ text, citationIds }]`.

```markdown
## Opening

It opened to pedestrians on May 27, 1937 [c1]. Cars crossed on May 29, 1937 [c1].
```

Text is parsed into segments: each heading (a `#` line, or a line that is only
bold text), and each sentence of a paragraph or list item. Sentences end at
`.`, `!` or `?` followed by a capital, a digit or a quotation mark, except
after common abbreviations such as `Dr.`, `e.g.` and `U.S.`, and after a single
initial. Markers just after a sentence's full stop belong to that sentence, and
a line holding only markers belongs to the sentence before it. A bracket is a
marker when each comma-separated part is an id in the evidence or has the
shape of one, letters then digits or digits alone, so `[1]` is refused as an
unknown id rather than read as the figure 1. A link's bracketed text is not a
marker. Fences and horizontal rules are markup; the text inside a fence is
read as a paragraph.

A segment list is used as given, with any inline markers in a segment's text
added to its ids. A segment whose text opens with `#`, or whose `kind` is
`heading`, is a heading; one opening with a list marker is a list item.

When the passages do not answer the question, the model should reply with
`INSUFFICIENT_EVIDENCE` alone. An answer that is only that word, allowing
surrounding whitespace, quotes, backticks or a full stop, has no segments:
`coverage.status` is `insufficient` and `coverage.abstained` is `true`. Beside
other text the word is an uncited sentence and is refused.

## Output

```json
{
  "segments": [
    {"index": 0, "kind": "heading", "text": "Opening", "citationIds": [], "claim": false, "supported": true, "reasons": [], "details": [], "overlap": null},
    {"index": 1, "kind": "sentence", "text": "It opened to pedestrians on May 27, 1937.", "citationIds": ["c1"], "claim": true, "supported": true, "reasons": [], "details": [], "overlap": 1},
    {"index": 2, "kind": "sentence", "text": "Cars crossed on May 29, 1937.", "citationIds": ["c1"], "claim": true, "supported": false,
     "reasons": ["unsupported_number", "low_overlap"], "details": [{"reason": "unsupported_number", "text": "May 29, 1937"}, {"reason": "low_overlap", "text": "cars crossed"}], "overlap": 0}
  ],
  "supportedText": "## Opening\n\nIt opened to pedestrians on May 27, 1937.",
  "coverage": {"status": "partial", "segments": 3, "supported": 2, "unsupported": 1, "claims": 2, "supportedClaims": 1, "abstained": false},
  "evidence": {"citations": 1, "usable": 1, "invalid": []},
  "minOverlap": 0.45
}
```

| Field | Meaning |
|---|---|
| `segments[].text` | The segment with its markers, list marker and emphasis removed |
| `segments[].kind` | `heading`, `lead_in`, `connective`, `closing`, `item` or `sentence` |
| `segments[].claim` | `false` for the four structural kinds, which need no citation |
| `segments[].supported` | Every check passed |
| `segments[].reasons` | Reason codes, in the order of the table below |
| `segments[].details` | Each failing token as `{reason, text}`; for `low_overlap`, the content words no cited passage holds |
| `segments[].overlap` | Share of the segment's content words found in its cited passages, or `null` |
| `supportedText` | The supported segments with their headings and list markers, citation markers removed |
| `coverage.status` | `matched`, `partial` or `insufficient`, as research uses them |
| `coverage.supported` / `unsupported` | Segments supported and refused, structural ones included |
| `coverage.claims` / `supportedClaims` | Segments that are claims, and those supported |
| `coverage.abstained` | The answer was only `INSUFFICIENT_EVIDENCE` |

`coverage.status` is `matched` when at least one claim is supported and no
segment is refused, `partial` when some claim is supported and some segment is
refused, and `insufficient` when no claim is supported. A refused heading or
lead-in counts as refused. In `supportedText` a heading is kept only over a
supported claim, a lead-in only before one, and a connective or closing
sentence only when some claim survived. The segments keep their
`citationIds`, so a reader can show the supported text with its sources.

## Checks

A claim is any segment that is not structural. Each claim is checked against
the usable passages it cites, and every failure is reported, not only the
first.

| Reason | When |
|---|---|
| `missing_citation` | The claim cites nothing |
| `unknown_citation` | It cites an id the evidence does not hold, such as an invented `c9` or a source id like `s3` |
| `invalid_citation` | It cites a citation that failed the evidence checks |
| `unsupported_number` | A number, percentage, money amount, year or date is in no cited passage |
| `unsupported_quote` | A quoted string is in no cited passage |
| `unsupported_name` | A capitalised name is in no cited passage |
| `low_overlap` | Fewer than `0.45` of its content words occur in the cited passages |

One unknown or invalid id refuses a claim even beside a good one. The content
checks still run against the usable ids, so every problem is listed.

Numbers are compared as values. `1,200,000`, `1.2 million` and `$1.2m` are
the same value; a currency symbol or word must match; a percentage must meet a
percentage; a bare number may meet a passage number of any kind. Spelled
numbers count (`eleven`, `twenty-five`, `a thousand`), and a lone `one` counts
only as a quantity, not in `one of` or `no one`. A figure hedged with `about`,
`roughly`, `nearly` or the like may round the passage's figure to its own
precision, so `about 40%` meets `38.7%` but `about 99%` does not meet `97%`.
An unhedged figure must match exactly.

Dates are read in their written forms (`May 27, 1937`, `27 May 1937`,
`2022-08-18`, `August 2022`, `May 27`) and compared as dates. A date is
supported by one at least as specific, so `May 1937` meets `May 27, 1937`, and
a passage's `12 March` meets `12 March 2021` when the passage holds 2021
elsewhere.

Quotations in straight or curly double quotes, guillemets, and single quotes
that are not apostrophes must occur as a word sequence in one cited passage,
ignoring case, punctuation and diacritics.

Names are runs of capitalised words, joined across `of`, `de`, `van` and the
like, matched as a word sequence in one cited passage, ignoring case and
inflection. Month and weekday names are left to the date check. A sentence's
first word joins a name only when the evidence writes it capitalised
mid-sentence, or when it starts a longer run and is not an ordinary word of the
cited passages, so `Brooklyn Bridge` is checked whole while `Typically` is not
a name. Headings are title case, so their capitals are not read as names.

Content words are the segment's words less function words, discourse words
such as `overall` and `according`, numbers, and month names, after a light
suffix stripper on both sides (`spans` meets `spanning`, `released` meets
`releases`). Diacritics and letters such as `ł` fold, so `Sklodowska` meets
`Skłodowska`.

## Structural text

Four kinds of segment make no claim of their own and need no citation, each
only while it is 20 words or shorter.

| Kind | What it is |
|---|---|
| `heading` | A Markdown heading, or a bold-only line |
| `lead_in` | An uncited sentence ending in a colon, such as `Bakers watch a few signals:` |
| `connective` | An uncited segment with no content word, figure or quotation, such as `Hope this helps!` |
| `closing` | The answer's last segment, uncited, that passes every check, overlap included, against the passages the answer's supported claims cited |

Structural text still answers for what it states. A number, quotation or name
in a heading, lead-in or connective must occur in the passages it cites or,
citing none, somewhere in the evidence; so `# Opened in 1938` is refused when
no passage says 1938. A closing sentence that fails any check is an uncited
claim and is refused with `missing_citation`. A list item is a claim like any
sentence.

## Measured precision and recall

```bash
npm run search:grounding-evaluate
```

The command verifies twelve labelled answers in
`platform/search/grounding-fixture.mjs` and writes
`.search-data/reports/grounding.json` (or `--output path`) with the threshold
sweep, the results per split and per kind, and every misjudged segment. Each
answer is written out as the Markdown a model would return and verified whole,
so the parser is measured with the checks; a pack that does not parse into one
segment per labelled row fails the run.

The fixture holds 230 segments over twelve evidence packs shaped like research
responses: 115 supported and 115 adversarial. Six packs choose the threshold
and six held-out packs only measure it. The threshold is the lowest with the
best F0.5 of predicting `supported` on the development packs, precision
weighted over recall because a passed unsupported sentence costs a reader more
than a refused one. On the development packs F0.5 is 0.866 at both 0.45 and
0.5. Recall starts to fall at 0.55, and precision rises again only from 0.9,
where the overlap refuses some named limits along with a third of the
supported segments. The module's `0.45` is that choice, and a test holds the
two equal.

| Packs | Segments | Precision | Recall |
|---|---|---|---|
| Development | 114 | 0.838 | 1.000 |
| Held out | 116 | 0.871 | 0.931 |
| All packs | 230 | 0.854 | 0.965 |
| All packs, named limits left out | 211 | 1.000 | 0.965 |

Precision is held below 1 by the named limits below, which the adversarial
half includes on purpose. Every other adversarial segment is refused.

| Kind | Label | Segments | Judged as labelled |
|---|---|---|---|
| `verbatim` | Supported | 31 | 31 |
| `paraphrase` | Supported | 34 | 33 |
| `synthesis` | Supported | 6 | 6 |
| `heading` | Supported | 12 | 12 |
| `lead_in` | Supported | 7 | 7 |
| `list_item` | Supported | 13 | 13 |
| `closing` | Supported | 12 | 9 |
| `changed_number` | Adversarial | 30 | 30 |
| `changed_date` | Adversarial | 8 | 8 |
| `swapped_name` | Adversarial | 12 | 12 |
| `uncited` | Adversarial | 12 | 12 |
| `invented_id` | Adversarial | 12 | 12 |
| `merged` | Adversarial | 12 | 12 |
| `wrong_quote` | Adversarial | 7 | 7 |
| `embellished` | Adversarial | 3 | 3 |
| `negation` | Named limit | 6 | 0 |
| `comparison` | Named limit | 5 | 0 |
| `role_swap` | Named limit | 5 | 0 |
| `substitution` | Named limit | 3 | 0 |

The four supported segments refused are all held out: three closing sentences
that add an evaluation the evidence does not make (`the budget grows
modestly`, `two doses give strong protection`, `pressure, dose and time decide
the shot`), and one paraphrase sharing two of five content words with its
passage (`Its net loss shrank to $6.1 million, compared with $9.8 million a
year before`).

The fixture is synthetic and was written by one author who knew the checks.
The two Calder County and Northwind Robotics packs are invented; the rest
follow well-known public facts, but each pack asserts only what its own
passages say. It measures the checks on the failure kinds named here, not a
model's real error rate, and a model's errors will include subtler ones.

## Limits

A lexical check confirms that tokens occur, not what a sentence does with them.
These pass, and the fixture counts them as passed unsupported segments:

- Negation. `The bridge is not painted International Orange` keeps every word
  of its passage; `not`, `no` and `never` are function words to the overlap,
  and no other check reads polarity.
- Reversed comparison or change. `falls` for `rises`, `cooler` for `warmer`,
  `widened` for `narrowed`: one changed word in a sentence otherwise drawn
  from the passage leaves the overlap high.
- Swapped roles. Two entities or two figures of the same passage exchanged,
  as in `Aldrin became the first person to step onto the lunar surface`, or
  `$15.9 million, up from $18.2 million` for the reverse. That includes a
  figure moved to another unit or quantity, such as `21.5 pounds` where the
  passage has `47.5 pounds (21.5 kg)`: units are not compared.
- Substitution. One ordinary word replaced in a long sentence, such as
  `stroma` for `thylakoid membranes` or `second` for `first`. Spelled ordinals
  are words, not numbers.

Other edges, measured or known:

- A count in a lead-in, such as `Two changes matter most:`, must occur in the
  evidence, and can be met by coincidence. Both of the fixture's passed that
  way, one on `The two astronauts` and the other on the day of `October 2`.
- A paraphrase that shares too few words with its passage is refused, and so
  is a closing sentence that evaluates rather than restates.
- Only hedged figures round. `1.2 million` without `about` must match
  exactly.
- Stopwords, number words, month names and the suffix stripper are English.
- The verifier trusts a passage as far as its length, and its slice when the
  source text is supplied. It fetches nothing, so it cannot tell whether the
  passage is still what the page says; a later crawl can change the offsets.
Human Machine Shift+M