vitna.ai/docs/routing/router-challenge Machine view
Raw .md llms.txt
> Source: https://vitna.ai/docs/routing/router-challenge
> Raw Markdown: https://vitna.ai/docs/routing/router-challenge.md
> Corpus index: https://vitna.ai/llms.txt

# Router Challenge (#72)

A router executes a routing policy per request. Vitna determines whether handing the choice to a router beats naming a model.

## The Epistemic Problem of Pricing a Router

Every fixed model or provider infrastructure class has a published list price on Witness (`witness.price_period`). Vitna archives the raw bytes, timestamps the read, and tracks the price period.

A dynamic router (such as Not Diamond, Martian, RouteLLM, or OpenRouter Auto) has no single published list price:

1. It routes dynamically across an arbitrary set of underlying models.
2. It charges in one of several ways: nothing on top of the selected model's own rate, which [OpenRouter's Auto Router documentation](https://openrouter.ai/docs/guides/routing/routers/auto-router) states for that router (retrieved 2026-08-26), a fee per request, or a percentage markup.
3. Its effective cost per unit is knowable only retrospectively from observed deployments or controlled test runs.

A decision engine that assumes a static list price for a router would be inventing a figure the market does not publish.

Aperture solves this through Router Challenge (#72): evaluating a router candidate retrospectively from confirmed telemetry recorded in `aperture.deployment_observation` and evals recorded in `aperture.evidence`.

The [/challenge](https://vitna.ai/challenge) page runs the same function over three illustrative scenarios. Nothing on it was observed, and it names no router and no model.

## The Economics of a Router Candidate

For a window of observed requests, the total router cost is:

```
Total Spend = Underlying Token Spend + Router Surcharges
Router Unit Cost = Total Spend / Observed Units
```

When a router charges a fee per request, the platform fee can consume a substantial fraction of the token savings gained by picking cheaper models.

Router Challenge computes the fee drag:

```
Fee Drag = Router Surcharge per Unit / (Incumbent Token Cost - Router Underlying Token Cost)
```

If underlying model selection saves \$0.006 per unit, but the router fee is \$0.004 per unit, the fee drag is 66.7%, leaving only \$0.002 in net savings.

Like every decision in Aperture, candidates are compared on cost per successful workload unit:

```
Successful Unit Cost = Unit Cost / Task Success Rate
```

A router that cuts token cost in half but drops task accuracy from 95% to 85% may produce a higher cost per successful unit once failed retries are factored in.

## Running a Router Challenge

From the command line:

```bash
npm run aperture:challenge -- <workload-id> --router <router-sku> [--fixed <sku>] [--surcharge <usd>] [--format markdown|json]
```

## Router ids the engine knows

`KNOWN_ROUTERS` in `platform/engine/router-candidate.mjs` carries a default fee for four router ids, used when `--surcharge` is not given. One of those defaults is a sourced fact. The other three are the engine's own assumptions, so a challenge run against one of them should pass `--surcharge` with the fee your own contract states.

| Router identifier | Default fee | Source |
|---|---|---|
| `openrouter/auto` | \$0.000 / req | [OpenRouter's Auto Router documentation](https://openrouter.ai/docs/guides/routing/routers/auto-router), retrieved 2026-08-26: you pay the selected model's standard rate, and nothing more for the routing |
| `notdiamond/custom-router` | \$0.001 / req | Not sourced. An engine default, never read from Not Diamond |
| `martian/model-router` | \$0.004 / req | Not sourced. An engine default, never read from Martian |
| `routellm/mf` | \$0.000 / req | Not sourced. An engine default |

## Minimum Evidence Guardrail

A router candidate evaluation requires at least 20 observed units on the workload (`MIN_ROUTER_OBSERVATIONS = 20`). Aperture refuses to evaluate a router on thinner data.

## Verdicts

* **SWITCH:** The router satisfies the quality floor, passes all edge case vetoes, and beats the incumbent by at least the workload's `economicThresholdPct` on cost per successful unit.
* **STAY:** The fixed incumbent model remains more economical, or the router fails the quality floor.
* **TEST:** The router achieved favorable economics, but triggered edge case vetoes or has marginal sample size requiring targeted validation before production rollout.
Human Machine Shift+M