Router Challenge (#72)
A router executes a routing policy per request. Vitna determines whether handing the choice to a router beats naming a model.
The Epistemic Problem of Pricing a Router
Every fixed model or provider infrastructure class has a published list price on Witness (witness.price_period). Vitna archives the raw bytes, timestamps the read, and tracks the price period.
A dynamic router (such as Not Diamond, Martian, RouteLLM, or OpenRouter Auto) has no single published list price:
- It routes dynamically across an arbitrary set of underlying models.
- It charges in one of several ways: nothing on top of the selected model's own rate, which OpenRouter's Auto Router documentation states for that router (retrieved 2026-08-26), a fee per request, or a percentage markup.
- Its effective cost per unit is knowable only retrospectively from observed deployments or controlled test runs.
A decision engine that assumes a static list price for a router would be inventing a figure the market does not publish.
Aperture solves this through Router Challenge (#72): evaluating a router candidate retrospectively from confirmed telemetry recorded in aperture.deployment_observation and evals recorded in aperture.evidence.
The /challenge page runs the same function over three illustrative scenarios. Nothing on it was observed, and it names no router and no model.
The Economics of a Router Candidate
For a window of observed requests, the total router cost is:
Total Spend = Underlying Token Spend + Router Surcharges
Router Unit Cost = Total Spend / Observed UnitsWhen a router charges a fee per request, the platform fee can consume a substantial fraction of the token savings gained by picking cheaper models.
Router Challenge computes the fee drag:
Fee Drag = Router Surcharge per Unit / (Incumbent Token Cost - Router Underlying Token Cost)If underlying model selection saves \$0.006 per unit, but the router fee is \$0.004 per unit, the fee drag is 66.7%, leaving only \$0.002 in net savings.
Like every decision in Aperture, candidates are compared on cost per successful workload unit:
Successful Unit Cost = Unit Cost / Task Success RateA router that cuts token cost in half but drops task accuracy from 95% to 85% may produce a higher cost per successful unit once failed retries are factored in.
Running a Router Challenge
From the command line:
npm run aperture:challenge -- <workload-id> --router <router-sku> [--fixed <sku>] [--surcharge <usd>] [--format markdown|json]Router ids the engine knows
KNOWN_ROUTERS in platform/engine/router-candidate.mjs carries a default fee for four router ids, used when --surcharge is not given. One of those defaults is a sourced fact. The other three are the engine's own assumptions, so a challenge run against one of them should pass --surcharge with the fee your own contract states.
| Router identifier | Default fee | Source |
|---|---|---|
openrouter/auto | \$0.000 / req | OpenRouter's Auto Router documentation, retrieved 2026-08-26: you pay the selected model's standard rate, and nothing more for the routing |
notdiamond/custom-router | \$0.001 / req | Not sourced. An engine default, never read from Not Diamond |
martian/model-router | \$0.004 / req | Not sourced. An engine default, never read from Martian |
routellm/mf | \$0.000 / req | Not sourced. An engine default |
Minimum Evidence Guardrail
A router candidate evaluation requires at least 20 observed units on the workload (MIN_ROUTER_OBSERVATIONS = 20). Aperture refuses to evaluate a router on thinner data.
Verdicts
- SWITCH: The router satisfies the quality floor, passes all edge case vetoes, and beats the incumbent by at least the workload's
economicThresholdPcton cost per successful unit. - STAY: The fixed incumbent model remains more economical, or the router fails the quality floor.
- TEST: The router achieved favorable economics, but triggered edge case vetoes or has marginal sample size requiring targeted validation before production rollout.
