96 generations: 12 tickers x 4 synthesis sections x 2 models, identical prompts
The compliance gate passed an inverted P/E reading without complaint. Aggregate filters measure compliance, not correctness; the rubric is where the gap shows.
DocumentationWeb API referenceSearch referenceInvoice reconcileOpenAPI specsllms.txt
Vitna is built inside Convexity, a market intelligence terminal whose assistant routes every call by intent across two vendors. This is the actual routing ledger from its git history, the decision pattern Vitna exists to automate. Reconstructed and retrieved 2026-09-15.
Every line traces to a dated commit or an in-code decision comment. Nothing is projected or invented.
Four internal evals, verbatim from the savings ledger, behind the entry that made Luna the synthesis primary. Scores are printed with their n and their date, and costs appear only as ratios, never as absolute spend.
96 generations: 12 tickers x 4 synthesis sections x 2 models, identical prompts
The compliance gate passed an inverted P/E reading without complaint. Aggregate filters measure compliance, not correctness; the rubric is where the gap shows.
84 generations: 3 tickers x 4 sections x 6 reasoning tiers, identical prompts
low · coverage 8 · bull 7 · bear 8 · criterion 8 · neutrality 8 · 1.0x cost · 6.6s
Quality is not monotone in reasoning budget, and cost is: xhigh pays 3x the per-call cost of low to score lower, and loses framing neutrality on the way. low ties high at half the cost and 10 seconds less latency, and was adopted.
6 generations: 1 thesis x 6 reasoning tiers, one shared bundle of 9,244 characters, byte-identical context
Prose runs 450 to 551 tokens across every tier; reasoning runs 0 to 4,095.
Same read, same six generations: what the output contains at each tier. The first tier to carry a capability is ruled under.
| Capability | nano | none | low | medium | high | xhigh |
|---|---|---|---|---|---|---|
| Circular-financing bear case | no | yes | yes | yes | yes | yes |
| Supply-constraint evidence | no | yes | yes | yes | yes | yes |
| Peer competitive margins | no | no | yes | yes | yes | yes |
| Non-circular decision criterion | no | partial | yes | yes | yes | yes |
| Reports that an empty source carries nothing | no | no | no | yes | yes | yes |
Prose is 450 to 550 tokens at every tier including nano, and reasoning spans zero to 4,095: nothing about effort makes the model write more, it makes it think more. low is where peer data and a falsifiable decision criterion appear; medium is where the model stops citing a source for claims it cannot carry. Above medium the returns collapse: high and xhigh spend 4x and 8x the reasoning tokens to add one mention and slightly fuller peer margins, at 2x and 4x the latency.
| Sample | n | median | p90 | worst |
|---|---|---|---|---|
| Paired eval | 48 | 4.3s | 7.2s | 38.0s |
| Effort-sweep control | 12 | 4.2s | · | · |
| Tier read | 1 | · | · | 18.1s |
nano at the median is the fastest configuration measured, and the cheapest. Its defect is tail variance: roughly one call in twenty spikes to 4-9x its median. For a four-hourly batch neither number matters much; for anything interactive, the tail is the number to watch.
Drawn to scale from the paired eval, n = 48: the worst call ran 8.8x its own median.
| Tier | overall | output tokens | latency | fits 2,000 |
|---|---|---|---|---|
| none | 7.5 | 426 to 480 | 4.4 to 5.2s | yes |
| low · adopted | 7.9 | 618 to 665 | 6.9s | yes |
| medium | 8.0 | 2,020 to 2,095 | 19 to 22s | no |
| high | 7.8 | 3,510 to 4,624 | 30 to 39s | no |
| xhigh | 7.9 | 8,170 to 10,296 | 66 to 81s | no |
none, at zero reasoning tokens, clears every gate: correctness came from the instruction, not the reasoning budget, and the old sweep was largely measuring which tiers rediscovered unstated requirements. The curve flattened to half a point between best and worst. The mandate roughly tripled reasoning tokens at a fixed tier, so medium now exceeds the production output cap in 3 of 3 tickers, and xhigh runs 66 to 81 seconds at about 8x the cost of low for at most a tenth of a point. Configuration settled: luna at low, with the revised instruction.
Compliance gates, every arm: 15 of 15. Scored by the author of the prompt against pre-registered gates, n = 3 tickers. Gate results are near-objective; graded means carry at least 0.3 of noise either way.
The lineup settled on 10 Aug 2026 and the routing table in the repository has not changed since; the retrieval on 2026-09-15 confirmed it. What changed is what the usage ledger could say about it: 4 readings, each traced to a merged pull request, and none of them a ruling.
Where a reading carries no n, the count is a call volume on one surface. Beside the token profile the register already carries for that lane and a public price, a volume is a spend figure, and this study keeps spend to ratios.
4 standing rules the savings ledger wrote down for itself on the day of the measurements above, in its own words.
The pattern above, run by hand for five months, is the one the engine automates: a threshold somebody armed, a run, a brief, and a ruling a person signs.