Get started

Router Challenge (#72)

Source

A router executes a routing policy per request. Vitna determines whether handing the choice to a router beats naming a model.

The Epistemic Problem of Pricing a Router

Every fixed model or provider infrastructure class has a published list price on Witness (witness.price_period). Vitna archives the raw bytes, timestamps the read, and tracks the price period.

A dynamic router (such as Not Diamond, Martian, RouteLLM, or OpenRouter Auto) has no single published list price:

  1. It routes dynamically across an arbitrary set of underlying models.
  2. It charges in one of several ways: nothing on top of the selected model's own rate, which OpenRouter's Auto Router documentation states for that router (retrieved 2026-08-26), a fee per request, or a percentage markup.
  3. Its effective cost per unit is knowable only retrospectively from observed deployments or controlled test runs.

A decision engine that assumes a static list price for a router would be inventing a figure the market does not publish.

Aperture solves this through Router Challenge (#72): evaluating a router candidate retrospectively from confirmed telemetry recorded in aperture.deployment_observation and evals recorded in aperture.evidence.

The /challenge page runs the same function over three illustrative scenarios. Nothing on it was observed, and it names no router and no model.

The Economics of a Router Candidate

For a window of observed requests, the total router cost is:

text
Total Spend = Underlying Token Spend + Router Surcharges
Router Unit Cost = Total Spend / Observed Units

When a router charges a fee per request, the platform fee can consume a substantial fraction of the token savings gained by picking cheaper models.

Router Challenge computes the fee drag:

text
Fee Drag = Router Surcharge per Unit / (Incumbent Token Cost - Router Underlying Token Cost)

If underlying model selection saves \$0.006 per unit, but the router fee is \$0.004 per unit, the fee drag is 66.7%, leaving only \$0.002 in net savings.

Like every decision in Aperture, candidates are compared on cost per successful workload unit:

text
Successful Unit Cost = Unit Cost / Task Success Rate

A router that cuts token cost in half but drops task accuracy from 95% to 85% may produce a higher cost per successful unit once failed retries are factored in.

Running a Router Challenge

From the command line:

bash
npm run aperture:challenge -- <workload-id> --router <router-sku> [--fixed <sku>] [--surcharge <usd>] [--format markdown|json]

Router ids the engine knows

KNOWN_ROUTERS in platform/engine/router-candidate.mjs carries a default fee for four router ids, used when --surcharge is not given. One of those defaults is a sourced fact. The other three are the engine's own assumptions, so a challenge run against one of them should pass --surcharge with the fee your own contract states.

Router identifierDefault feeSource
openrouter/auto\$0.000 / reqOpenRouter's Auto Router documentation, retrieved 2026-08-26: you pay the selected model's standard rate, and nothing more for the routing
notdiamond/custom-router\$0.001 / reqNot sourced. An engine default, never read from Not Diamond
martian/model-router\$0.004 / reqNot sourced. An engine default, never read from Martian
routellm/mf\$0.000 / reqNot sourced. An engine default

Minimum Evidence Guardrail

A router candidate evaluation requires at least 20 observed units on the workload (MIN_ROUTER_OBSERVATIONS = 20). Aperture refuses to evaluate a router on thinner data.

Verdicts

  • SWITCH: The router satisfies the quality floor, passes all edge case vetoes, and beats the incumbent by at least the workload's economicThresholdPct on cost per successful unit.
  • STAY: The fixed incumbent model remains more economical, or the router fails the quality floor.
  • TEST: The router achieved favorable economics, but triggered edge case vetoes or has marginal sample size requiring targeted validation before production rollout.