Get started
01 / 08OverviewShadow telemetry · on your own box

Switchyard · a model router · on your own box

Pick a model per request, by rules you can read.

An OpenAI-compatible router you run yourself. Send model: "auto" and written rules choose the model and the reasoning effort for each request without asking a model, keep a warm conversation on its cache, and write down why.

In shadowIt runs on one server, for Convexity, and nothing on this page installs yet.

Requests arrive from the left, pass the router's ladder, and leave on three tracks: light, standard and strong. One request, in amber, takes the standard track.

A requestin amber

Our payments service double-charges about one customer in 20,000. Walk through, step by step, every way an at-least-once queue, a retrying HTTP client and a non-idempotent charge endpoint can combine to cause it; prove which one fits a 1-in-20,000 rate given these numbers: 0.3% client retry rate, 0.02% queue redelivery, 40 ms p99 timeout; and design a fix with an idempotency key, including the exact schema, the race you must still handle, and a test that proves it.code, difficulty 43, precision 2

strongdifficulty 70 to 100

anthropic/claude-opus-5

None of the four goes this far.

standarddifficulty 40 to 69

anthropic/claude-sonnet-5
  • A payments bugstandard · mediumdifficulty 43 · ~$0.02124 a turn
  • A migration designstandard · lowdifficulty 40 · ~$0.01511 a turn

lightdifficulty 0 to 39

deepseek/deepseek-v4.1-flash
  • A quick questionlight · highdifficulty 16 · ~$0.00402 a turn
  • A thank-youlight · lowdifficulty 0 · ~$0.00055 a turn
Four requests written for this page, each decided by the router in this repository when the site was built. The picture is drawn, not measured: requests pass the router's ladder and leave on its three tracks. In amber, the request taken apart below.
+1.03 msadded at the median, one 1 KB chat request at a time, against an upstream that answers instantly
17 of 20held-out requests routed to an accepted tier, labelled before the first run and never tuned against
237tests, including a real router on a port in front of a scripted upstream
8 of 8deliberate breaks of its core properties turned the suite red
02 / 08How it decidesSix rules · no model call

How it decides

One decision, taken apart.

Six steps run between a request and its model, and none of them asks a model. The decision is a function of the request and the conversation it continues, so it can be run again and give the same answer, which is how this panel was made.

switchyard explain request.jsonrouter.config.json · digest 6308d99fc0f402f6 · prices read 2026-09-28 and 2026-09-30

The request

Our payments service double-charges about one customer in 20,000. Walk through, step by step, every way an at-least-once queue, a retrying HTTP client and a non-idempotent charge endpoint can combine to cause it; prove which one fits a 1-in-20,000 rate given these numbers: 0.3% client retry rate, 0.02% queue redelivery, 40 ms p99 timeout; and design a fix with an idempotency key, including the exact schema, the race you must still handle, and a test that proves it.

Class
code
Difficulty
43of 100
Precision
2of 3

What it was scored on

  • class:code+28
  • reasoning_cues+5
  • analysis_cues+3
  • rigor_words+7
  • precision: precision_words+1
  • precision: exact_classes+1

Decision

anthropic/claude-sonnet-5
effort mediumtier standard
  1. difficulty 43 -> standard/low
  2. precision 2 raises effort to medium
  3. no warm conversation

If a call fails, in this order

  1. anthropic/claude-sonnet-5medium
  2. openai/gpt-6-solmedium
  3. anthropic/claude-opus-5medium

~2,100 output tokens, ~$0.02124 this turn, with no cache warm

The request

We run Postgres 14 behind PgBouncer. Design a zero-downtime migration that splits a 2 TB orders table by region: the cutover plan, how to backfill, how to verify row counts, and how to roll back at each step.

Class
analysis
Difficulty
40of 100
Precision
0of 3

What it was scored on

  • class:analysis+30
  • analysis_cues+3
  • rigor_words+7

Decision

anthropic/claude-sonnet-5
effort lowtier standard
  1. difficulty 40 -> standard/low
  2. no warm conversation

If a call fails, in this order

  1. anthropic/claude-sonnet-5low
  2. openai/gpt-6-sollow
  3. anthropic/claude-opus-5low

~1,500 output tokens, ~$0.01511 this turn, with no cache warm

The request

What does HTTP status 429 mean?

Class
qa
Difficulty
16of 100
Precision
0of 3

What it was scored on

  • class:qa+16

Decision

deepseek/deepseek-v4.1-flash
effort hightier light
  1. difficulty 16 -> light/high
  2. no warm conversation

If a call fails, in this order

  1. deepseek/deepseek-v4.1-flashhigh
  2. openai/gpt-6-lunahigh
  3. openai/gpt-5.6-lunahigh

~3,350 output tokens, ~$0.00402 this turn, with no cache warm

The request

Thanks, that worked!

Class
chitchat
Difficulty
0of 100
Precision
0of 3

What it was scored on

  • class:chitchat0

Decision

deepseek/deepseek-v4.1-flash
effort lowtier light
  1. difficulty 0 -> light/none
  2. deepseek/deepseek-v4.1-flash takes low, not none
  3. no warm conversation

If a call fails, in this order

  1. deepseek/deepseek-v4.1-flashlow
  2. openai/gpt-6-lunalow
  3. openai/gpt-5.6-lunalow

~460 output tokens, ~$0.00055 this turn, with no cache warm

Four requests written for this page, each decided when the site is built by the router in this repository with the config it ships, making the calls switchyard explain makes: as the first turn of a conversation, so no cache is warm, and with no preference from the caller. The command prints these fields as text, and the layout is ours. Costs are at the config's prices, read 2026-09-28, and deepseek/deepseek-v4.1-flash's on 2026-09-30.

  1. 1FeaturesReads the request once and returns counts: message sizes, code, stack traces, math notation, cue words by family, constraints, media, tools, conversation depth. No text survives it.src/features.mjs
  2. 2ScoresA task class, a difficulty from 0 to 100 and a precision from 0 to 3, under weights written in the config. A short follow-up is scored by the task it continues.src/score.mjs
  3. 3LadderThe difficulty picks a rung, and the rung a tier and an effort. A caller's preference slides the request along the line, and precision raises the effort's floor.policy.ladder
  4. 4CapabilitiesThe first model on the track that can serve the request: images, audio or files, tools, structured output, the context window, the output cap.per model, in the config
  5. 5StickinessA warm conversation keeps its model and its effort, so its cache stays warm, unless moving is worth the cold start.src/policy.mjs
  6. 6FallbacksThe chosen model, its tier-mates, then one tier up, never a cheaper tier. A malformed request and a refusal are not retried elsewhere.429 · 5xx · timeouts
The ladder and the tracks, as shippedread from the config this repository ships, the one the benchmark measured at commit e27ca78d

The ladderpolicy.ladder

as shipped
DifficultyTierEffort
0 to 9lightnone
10 to 24lighthigh
25 to 39lighthigh
40 to 54standardlow
55 to 69standardmedium
70 to 84strongmedium
85 to 100stronghigh

A caller's preference moves a request along the line: cost -10, balanced 0, quality +10. Precision raises the effort's floor, as it did for the payments bug above.

The trackstiers

up on a failure, never down
  • light
    1. deepseek/deepseek-v4.1-flash
    2. openai/gpt-6-luna
    3. openai/gpt-5.6-luna
    4. anthropic/claude-haiku-4.5
  • standard
    1. anthropic/claude-sonnet-5
    2. openai/gpt-6-sol
  • strong
    1. anthropic/claude-opus-5
    2. openai/gpt-6-astra

The ladder and the tracks are read from the config this repository ships, switchyard/config/router.config.json, and are the ones the benchmark measured at commit e27ca78d.

03 / 08The benchmarkMT-Bench · MT-Bench-101

The benchmark · MT-Bench, then MT-Bench-101

Against Jev Router, judge by judge.

MT-Bench's public questions, answered through OpenRouter from 2026-09-28 to 2026-09-30 and graded blind by three judges from three vendors, then 80 questions from MT-Bench-101 that nobody here had read, to see whether the result held. Both are chat, not anyone's traffic.

80questions per turn in each set, first turns and follow-ups
2,547answers, graded blind 2,208 times
3judges decide, Grok the neutral one; three more graded beside them, none in the mean
$39.01spent on everything in the record
Switchyard since 2026-09-30 DeepSeek V4.1 Flash leads its light tierheld up, judge by judge · 2026-09-28 to 2026-09-30

Where Jev Router leads comes first. It answers faster, it costs less on MT-Bench-101's short questions, and the GPT judge puts it ahead on follow-ups: the lit cells in its columns. Everywhere else in this table Switchyard is ahead.

The table scrolls sideways.

Held up, byMT-Bench, first turnsof 80MT-Bench, follow-upsof 79MT-Bench-101, first turnsof 79MT-Bench-101, follow-upsof 75
SwitchyardJev RouterSwitchyardJev RouterSwitchyardJev RouterSwitchyardJev Router
Mean of the three judgeswhat the rules were decided on6654686270347350
Claude judge7056695462266943
GPT judge6964545872706768
Grokneutral6041666065276838
Qwenneutral, not in the mean5335544553165826
DeepSeek V4 Proan interested judge, MT-Bench only50455244not askednot asked
Gemini 3.8 Flashan interested judge, MT-Bench only64506258not askednot asked
Cost of the answersthe total for each turn's answers$0.117$0.123$0.215$0.288$0.056$0.028$0.052$0.033
Median time to answerDeepSeek's answers, for Switchyard6.4 s3.6 s11.0 s5.1 snot recordednot recorded

The brighter figure of each pair is ahead: more held up, less cost, less time. An answer holds up when it scores within one point, on a scale of 1 to 10, of the best answer the tiers gave to the same prompt. Out of 80 and 79 on MT-Bench and 79 and 75 on MT-Bench-101, except where a judge graded fewer on MT-Bench: the Claude judge and Qwen graded 77 and 73 of the follow-ups, DeepSeek V4 Pro 77 first turns and 56 follow-ups, and Gemini 3.8 Flash 79 and 78.

Which judge you ask moves the frontier. Five policies on MT-Bench under each reading. On the six-answer run the Claude judge put Opus 22 first-turn answers ahead of gpt-6-luna at high effort, and the GPT judge put it 14 behind, so a claim across vendors needs every column. It opens on the neutral judge's follow-ups.

Turn
Held up by
  • Switchyard's rules, since and before 2026-09-30
  • Jev Router, and one model always: DeepSeek V4.1 Flash, or Opus 5
  • The frontier: nothing holds up more for less

Follow-ups, held up by Grok, the neutral judge

$0.03$0.10$0.30$1$3304050607080What the 79 answers cost, log scaleAnswers held up, of 79Since 09-30: 66 of 79 held up · $0.215 for 79 answers$0.21566 of 7903 Since 09-3066 of 79 held up · $0.215 for 79 answersBefore 09-30: 49 of 79 held up · $0.090 for 79 answers$0.09049 of 7901 Before 09-3049 of 79 held up · $0.090 for 79 answersJev Router: 60 of 79 held up · $0.288 for 79 answers$0.28860 of 79Jev Router60 of 79 held up · $0.288 for 79 answersDeepSeek: 65 of 79 held up · $0.168 for 79 answers$0.16865 of 7902 DeepSeek65 of 79 held up · $0.168 for 79 answersOpus 5: 68 of 79 held up · $2.835 for 79 answers$2.83568 of 7904 Opus 568 of 79 held up · $2.835 for 79 answers

Not drawn: the best answer each time, 79 of 79 for $0.091, chosen in hindsight.

Follow-ups, held up by the Claude judge

$0.03$0.10$0.30$1$3304050607080What the 79 answers cost, log scaleAnswers held up, of 79Since 09-30: 69 of 77 held up · $0.215 for 79 answers$0.21569 of 77Since 09-3069 of 77 held up · $0.215 for 79 answersBefore 09-30: 42 of 77 held up · $0.090 for 79 answers$0.09042 of 7701 Before 09-3042 of 77 held up · $0.090 for 79 answersJev Router: 54 of 77 held up · $0.288 for 79 answers$0.28854 of 77Jev Router54 of 77 held up · $0.288 for 79 answersDeepSeek: 71 of 77 held up · $0.168 for 79 answers$0.16871 of 7702 DeepSeek71 of 77 held up · $0.168 for 79 answersOpus 5: 70 of 77 held up · $2.835 for 79 answers$2.83570 of 77Opus 570 of 77 held up · $2.835 for 79 answers

Not drawn: the best answer each time, 79 of 79 for $0.091, chosen in hindsight.

Follow-ups, held up by the GPT judge

$0.03$0.10$0.30$1$3304050607080What the 79 answers cost, log scaleAnswers held up, of 79Since 09-30: 54 of 79 held up · $0.215 for 79 answers$0.21554 of 79Since 09-3054 of 79 held up · $0.215 for 79 answersBefore 09-30: 68 of 79 held up · $0.090 for 79 answers$0.09068 of 7901 Before 09-3068 of 79 held up · $0.090 for 79 answersJev Router: 58 of 79 held up · $0.288 for 79 answers$0.28858 of 79Jev Router58 of 79 held up · $0.288 for 79 answersDeepSeek: 55 of 79 held up · $0.168 for 79 answers$0.16855 of 79DeepSeek55 of 79 held up · $0.168 for 79 answersOpus 5: 49 of 79 held up · $2.835 for 79 answers$2.83549 of 79Opus 549 of 79 held up · $2.835 for 79 answers

Not drawn: the best answer each time, 79 of 79 for $0.091, chosen in hindsight.

Follow-ups, held up by the mean of the three judges

$0.03$0.10$0.30$1$3304050607080What the 79 answers cost, log scaleAnswers held up, of 79Since 09-30: 68 of 79 held up · $0.215 for 79 answers$0.21568 of 79Since 09-3068 of 79 held up · $0.215 for 79 answersBefore 09-30: 55 of 79 held up · $0.090 for 79 answers$0.09055 of 7901 Before 09-3055 of 79 held up · $0.090 for 79 answersJev Router: 62 of 79 held up · $0.288 for 79 answers$0.28862 of 79Jev Router62 of 79 held up · $0.288 for 79 answersDeepSeek: 70 of 79 held up · $0.168 for 79 answers$0.16870 of 7902 DeepSeek70 of 79 held up · $0.168 for 79 answersOpus 5: 66 of 79 held up · $2.835 for 79 answers$2.83566 of 79Opus 566 of 79 held up · $2.835 for 79 answers

Not drawn: the best answer each time, 79 of 79 for $0.091, chosen in hindsight.

First turns, held up by Grok, the neutral judge

$0.03$0.10$0.30$1$3304050607080What the 80 answers cost, log scaleAnswers held up, of 80Since 09-30: 60 of 80 held up · $0.117 for 80 answers$0.11760 of 80Since 09-3060 of 80 held up · $0.117 for 80 answersBefore 09-30: 47 of 80 held up · $0.051 for 80 answers$0.05147 of 8001 Before 09-3047 of 80 held up · $0.051 for 80 answersJev Router: 41 of 80 held up · $0.123 for 80 answers$0.12341 of 80Jev Router41 of 80 held up · $0.123 for 80 answersDeepSeek: 62 of 80 held up · $0.098 for 80 answers$0.09862 of 8002 DeepSeek62 of 80 held up · $0.098 for 80 answersOpus 5: 72 of 80 held up · $1.882 for 80 answers$1.88272 of 8003 Opus 572 of 80 held up · $1.882 for 80 answers

Not drawn: the best answer each time, 80 of 80 for $0.047, chosen in hindsight.

First turns, held up by the Claude judge

$0.03$0.10$0.30$1$3304050607080What the 80 answers cost, log scaleAnswers held up, of 80Since 09-30: 70 of 80 held up · $0.117 for 80 answers$0.11770 of 80Since 09-3070 of 80 held up · $0.117 for 80 answersBefore 09-30: 52 of 80 held up · $0.051 for 80 answers$0.05152 of 8001 Before 09-3052 of 80 held up · $0.051 for 80 answersJev Router: 56 of 80 held up · $0.123 for 80 answers$0.12356 of 80Jev Router56 of 80 held up · $0.123 for 80 answersDeepSeek: 72 of 80 held up · $0.098 for 80 answers$0.09872 of 8002 DeepSeek72 of 80 held up · $0.098 for 80 answersOpus 5: 75 of 80 held up · $1.882 for 80 answers$1.88275 of 8003 Opus 575 of 80 held up · $1.882 for 80 answers

Not drawn: the best answer each time, 80 of 80 for $0.047, chosen in hindsight.

First turns, held up by the GPT judge

$0.03$0.10$0.30$1$3304050607080What the 80 answers cost, log scaleAnswers held up, of 80Since 09-30: 69 of 80 held up · $0.117 for 80 answers$0.11769 of 80Since 09-3069 of 80 held up · $0.117 for 80 answersBefore 09-30: 72 of 80 held up · $0.051 for 80 answers$0.05172 of 8001 Before 09-3072 of 80 held up · $0.051 for 80 answersJev Router: 64 of 80 held up · $0.123 for 80 answers$0.12364 of 80Jev Router64 of 80 held up · $0.123 for 80 answersDeepSeek: 70 of 80 held up · $0.098 for 80 answers$0.09870 of 80DeepSeek70 of 80 held up · $0.098 for 80 answersOpus 5: 64 of 80 held up · $1.882 for 80 answers$1.88264 of 80Opus 564 of 80 held up · $1.882 for 80 answers

Not drawn: the best answer each time, 80 of 80 for $0.047, chosen in hindsight.

First turns, held up by the mean of the three judges

$0.03$0.10$0.30$1$3304050607080What the 80 answers cost, log scaleAnswers held up, of 80Since 09-30: 66 of 80 held up · $0.117 for 80 answers$0.11766 of 80Since 09-3066 of 80 held up · $0.117 for 80 answersBefore 09-30: 60 of 80 held up · $0.051 for 80 answers$0.05160 of 8001 Before 09-3060 of 80 held up · $0.051 for 80 answersJev Router: 54 of 80 held up · $0.123 for 80 answers$0.12354 of 80Jev Router54 of 80 held up · $0.123 for 80 answersDeepSeek: 68 of 80 held up · $0.098 for 80 answers$0.09868 of 8002 DeepSeek68 of 80 held up · $0.098 for 80 answersOpus 5: 73 of 80 held up · $1.882 for 80 answers$1.88273 of 8003 Opus 573 of 80 held up · $1.882 for 80 answers

Not drawn: the best answer each time, 80 of 80 for $0.047, chosen in hindsight.

  1. where Jev Router leadsFaster, cheaper on short questions, and ahead under the GPT judge on follow-ups.A median 3.6 s and 5.1 s on MT-Bench against DeepSeek's 6.4 s and 11.0 s. On MT-Bench-101 it cost $0.028 and $0.033 against $0.056 and $0.052, sending most questions to gpt-6-luna. The GPT judge: 58 against 54 and 68 against 67 follow-ups.
  2. MT-BenchMore held up than Jev Router on both turns, for less.66 against 54 and 68 against 62 by the mean, and ahead under Grok and the Claude judge too, for $0.117 against $0.123 and $0.215 against $0.288. That is 2.3 and 2.4 times what the rules it replaced cost.
  3. fresh questionsAnd on questions nobody here had read, by more.70 against 34 and 73 against 50 by the mean, and ahead under Grok and Qwen. DeepSeek was chosen from MT-Bench's own prompts, so 10 dialogues from each of eight MT-Bench-101 tasks were drawn before any answer existed, and all five predictions written down with them held.
  4. the GPT judgeOne judge disagrees.The GPT judge prefers the rules the router shipped before 2026-09-30, when gpt-6-luna led the light tier, on both sets and both turns, and it is the only judge that rates luna's answers near the top. Every other judge on this page puts the rules since ahead on both turns of every set it graded, the two neutral ones among them.
  5. the router itselfOne model, always, did as well, for less.On MT-Bench always DeepSeek held up on 68 and 70 against 66 and 68, within the judges' noise, for $0.098 and $0.168: the rules' escalations to Sonnet 5, 7 first turns and 6 follow-ups, cost money and did not help. On MT-Bench-101 the rules sent every question to DeepSeek, so there the two are the same policy.

These five are the whole of what the record says its tables support. Read them with the terms above and with these: both sets are chat, not your traffic; eighty questions per turn, so a gap of two or three answers is within noise; the MT-Bench follow-ups are out of 79; the judges prefer their own vendor's answers; and Vitna built Switchyard and ran the benchmark. Every table, the predictions and all ten caveats are in the full record.

04 / 08Built to be checkedShadow · report · replay · route

Built to be checked

It shows its work.

A router you cannot read is a router you cannot trust with your traffic. This one writes down every decision, starts by watching rather than serving, and never trades a failure for a cheaper answer.

Shadow first, then route

the default

The incumbent keeps serving while the rules' pick is written down. Read what they would have done, try a new policy on the traffic already recorded, and only then does a person throw the switch.

  1. ShadowSWITCHYARD_MODE=shadow
  2. Reportswitchyard report
  3. Replayswitchyard replay --config candidate.json
  4. RouteSWITCHYARD_MODE=route

It writes down why

counts and hashes

One row per request, written from a finally so a request that fails still leaves it: the features as counts, the scores and the decision with their reasons, every attempt, the tokens and the cost, and where the milliseconds went.

It never keeps the prompt, the completion or an error message, so a row can be read, shared and replayed without exposing a word.

A warm conversation keeps its cache

cache kept

A provider drops a conversation's cached prefix when the model or the reasoning effort changes, so Switchyard keeps both on a warm conversation unless moving is worth the cold start. It needs no session id: a turn continues an earlier one when the earlier turn's messages are a prefix of its own.

A failure moves up, never down

fallbacks

A failed call moves along the same track, then one tier up, and never to a cheaper tier: a failure is not a reason to answer a request worse. A malformed request and a refusal are not retried elsewhere.

05 / 08The othersMeasured vs documented

The others

Only what was measured gets a score.

Jev Router is the one other router Switchyard was run against. OpenRouter's Auto Router and Not Diamond also choose a model per request, and are here as their own documentation describes them, read on 2026-09-29, with no figure, because nothing here asked either of them anything.

Four routers, one measured against anotherread 2026-09-29

no scores for what was read

The table scrolls sideways.

PropertySwitchyardmeasured hereJev Routermeasured hereAuto Routerread, not runNot Diamondread, not run
What it decidesThe model and the reasoning effort for each request sent as autopicks the best model and reasoning effort for each requestThe model for each request, within a cost tier the caller setsThe model for each request
How it decidesRules over counted features of the request, no model callruns on Jev, TypeSafe's first System One modelA classifier assigns each prompt one of about thirty task types, then ranks models by what OpenRouter's users spent on for that type over a trailing seven daysA router trained on the customer's own evaluation data: prompts, each model's response and a score for each, at least 25 samples
What a decision costsNone; about a millisecond in processThe listing prints no price of its ownThere is no additional fee for using the Auto RouterNot in the document read
What it says about a choiceEvery decision carries its reasonsNot in the document readThe response's model field names the model that answeredNot in the document read
A policy tried on past trafficswitchyard replayNot in the document readNot in the document readNot in the document read
Where it runsYour box, your provider keysOpenRouterOpenRouterNot in the document read
06 / 08QuestionsStatus · boundaries · fit

Questions

What people ask first.

Can I run it?Not yet. It runs in shadow on one server, and nothing on this page installs.

What runs todaynothing on this page installs

  • In shadow, on one serverin serviceIt answers on loopback only, for Convexity, and serves every auto request with the incumbent while it records its own decision.
  • The rules since 2026-09-30in this repositoryDeepSeek V4.1 Flash leads the light tier, the rules the benchmark measured. The server in shadow changes its decisions only when a release is published to it, and which release it runs is read on that server, not from here.
  • DeepSeek in route modenot yetBefore it serves a request, DeepSeek has to be pinned to providers that neither retain nor train on prompts, and the router has no per-model provider setting yet. Whether it takes tools while reasoning, through OpenRouter, has not been checked.
  • Installablenot yetIts repository is private and there is no package or image.
  • The namenot settledA working name. No trademark search has been run, and Red Hat's JBoss project SwitchYard exists.
  • Beside the runtimenot part of itvitna-runtime keeps its own per-request rules, declared on the request's shape and signed into a policy. Switchyard reads each request's text in process to count it, and keeps only the counts.
When does a warm conversation move?the stickiness rules, as the router's README states them

When a warm conversation movesfrom the README

cache kept
WhenWhat it does
The task needs a bigger tierupgrademove, forfeiting the cache
Same tier, effort two or more steps higherupgrade_effortsame model, new effort
Same tier otherwisestickysame model and effort
A cheaper tier would do, but the task continuessticky
The task changed and moving saves at least downgradeMargin over horizonTurns, cold start includeddowngrade
The task changed but the warm cache makes staying cheapersticky_cost
Past the TTL, or the model cannot serve the turn, or the caller excludes itcold, ineligible, constrainedfresh pick
  • upgradeEffortSteps 2
  • downgradeMargin 0.2
  • horizonTurns 3

The rows are the README's, copied at the pinned commit. A result in periwinkle keeps the cache and one in amber pays for a cold one. That a change of model or effort drops the cache is the providers' own rule: Anthropic's and OpenAI's prompt caching documentation, as Switchyard's decision log read them on 2026-09-28.

What does it keep about a request, and what never?counts and hashes, never the words

What a row holds

counts and hashes
  • featuresas counts
  • scoresand their reasons
  • decisionand its reasons
  • chain hashesHMACs, under the deployment's own key
  • attemptsevery upstream try, with its status
  • usagethe upstream's tokens and cost
  • timingswhere the milliseconds went
  • Jevits answer, as numbers, when asked

What it never holds

never
  • prompt textFeatures are counted, and the words are not kept.
  • completion textAn answer goes to the caller and nowhere else.
  • error messageA moderation error can quote the prompt, so only a type label is kept.

Two of its test files plant a word in a request and assert it never reaches the ledger. A change to how features are counted bumps a version, and replay skips the rows it can no longer compare.

What does the router add to a request?+1.03 ms at the median, one request at a time

Every scenario measured2026-09-28

c is requests in flight
ScenarioAdded p50Router p99Throughput
Chat, named model, 1 KB, c=1+1.03 ms2.83 ms732 req/s
Chat, named model, 1 KB, c=16+8.35 ms22.9 ms1,532 req/s
Chat, named model, 1 KB, c=64+30.8 ms63.3 ms1,770 req/s
Chat, auto in shadow, 1 KB, c=16+7.20 ms15.2 ms1,885 req/s
Chat stream, 1 KB, c=16+7.83 ms18.0 ms1,640 req/s
Chat, 200 KB, c=1+25.0 ms38.3 ms38 req/s
Chat, 200 KB, c=4+99.6 ms133 ms39 req/s
Anthropic messages, 1 KB, c=16+6.51 ms14.3 ms2,060 req/s
Jev on, chat 1 KB, c=1+1.20 ms4.28 ms686 req/s
Jev on, chat 1 KB, c=16+11.0 ms29.8 ms1,222 req/s
Jev on, Anthropic messages, c=16+9.31 ms20.9 ms1,453 req/s

Measured against an upstream that answers instantly, each in a process of its own, so every millisecond of difference is the router's; with more in flight it is mostly queueing on one Node thread. The full run of 2026-09-28, on a Windows 11 ARM64 laptop, Node 24.14, not the production server, with other work sharing it: both sides of every comparison ran under the same load, so the differences hold better than the absolute numbers. The router's p99 at one request at a time was 2.83 ms. It predates a faster pass over the features, which took a 200 KB prompt from +37.5 and +38.0 ms to +15.8 and +16.0 ms one at a time.

What is Jev, and does it route anything?a second opinion, asked beside the rules, that routes nothing

Jev is TypeSafe's decision model. With Jev on, Switchyard asks it one question per request, how capable a model the request needs, alongside the model call, so a caller waits for nothing and a slow or failing Jev changes nothing. The ledger keeps its answer as numbers.

What asking costs

sends text to TypeSafe
Calls
59
Cost, all of them
$0.002
Median
172 ms
p95
372 ms

It is off in the config this repository ships. Where it is on, the text a request is scored on also goes to TypeSafe, through OpenRouter; a client listed in jev.optInClients is asked about only the requests it marks, and Convexity's backend marks only its public-data surfaces.

It leans high: where it and the rules disagreed, 11 times, it chose the bigger tier 9 times.

Routed to an accepted tier2026-09-28

a smoke test

The labelled requests, measured on typesafe/jev-1.13-20260917 with the question as written and never tuned. A smoke test, not a benchmark.

Held out (20)
the rules17 of 20
Jev19 of 20
the higher of the two19 of 20
In-sample (39)
the rules39 of 39
Jev38 of 39
the higher of the two38 of 39

Whether it has earned a vote is in what the benchmark taught, in the full record. So far it has not.

Can it speculate on keystrokes, the way a 0.5-second agent does?0.49 ms median across 869 sections, in the browser, without sending unsubmitted words anywhere

Sanity's engineering write-up ("The 0.5-second agent", sanity.io/engineering/the-0-5-second-agent) shows that an agent feels instant when entry selection, source choice, model routing, and document staging finish during typing pauses (250 ms debounce) before the reader presses Enter. Sanity asks TypeSafe's Jev over the network on every pause (282 ms for one question, 340 ms for three bundled questions, $0.00008 per pause). Because Switchyard's scorer and policy engine are pure functions over an inverted token posting index with zero dependencies, the documentation shell runs all three decisions in the browser in 0.49 ms at the median (1.23 ms p95 across 869 sections), for $0, while keeping unsubmitted keystrokes on the reader's machine.

Switchyard in-browser speculation869 sections · 109 pages · N=500

live on /docs
Decision p50 / p95
0.49 / 1.23 ms
Held-out / regression
22/25 · 35/35
Cost per pause
$0.00000
Unsubmitted words sent
0 chars

Open /docs and choose Ask agent (0.5s) in the rail. Every 180 ms typing pause scores candidate postings (staging two or three pages when probability splits across concepts or multi-hop comparisons, bridging HTTP status symptoms like 429 and 503, or withholding when none_of_these leads), picks the source surface, and runs src/policy.mjs in process (0.24 to 2.29 ms warm on POST /docs/agent.json).

Sanity's 0.5s Jev stackpublished figures

network per pause
3-question Jev call
340 ms
Cost per pause
$0.00008
Haiku tool baseline
1,328 ms
Unsubmitted words sent
every pause

When pre-staged documentation already answers the question, both architectures route generation to a small, fast tier rather than a frontier model, and only escalate when multi-step reasoning or code synthesis demands it.

Live keystroke race: in-browser rules vs. network speculationType or pick a prompt below. Every typing pause runs the entry selector, source router, and Switchyard policy in your browser.
Switchyard + local index (in-browser)0.97 ms / pause
Waterfall at Enter
0 ms
Cost (3 pauses)
$0.00000
Keystrokes sent
0 chars
Selected route
light · high
Sanity 0.5s stack (TypeSafe Jev network call)340 ms / pause
Network per pause
340 ms
Cost (3 pauses)
$0.00024
Keystrokes sent
3 prefixes
Model ceiling
single small tier
Traditional waterfall (Haiku tool call after Enter)1828 ms after Enter
Waterfall at Enter
1828 ms
Haiku tool call
1328 ms
Corpus fetch
500 ms
Saved by staging
1828 ms
1. Entry probability distribution (1 staged)
  • Attestationstaged94%
  • Signed routing policy3%
  • none_of_these (skip retrieval)3%
2. Bundled source + Switchyard route
Source choice
docs_index (93% confidence)
Task class & difficulty
qa · difficulty 23
Chosen model
deepseek/deepseek-v4.1-flash (light · effort high)
Fast-path eligible
yes (context pre-staged)

Methodology and caveats: our in-browser lane runs deterministic lexical, BM25 IDF, character 3-gram typo, and symptom-to-concept scoring over an inverted token posting index plus Switchyard rule routing (0.49 ms median, 1.23 ms p95, 1.77 ms p99 across N=500 runs on the full 869-section, 109-page documentation corpus; 0.24 to 2.29 ms warm server decision time on POST /docs/agent.json at commit 775c6f06, after a one-time 44.2 ms cold index build on worker start). Sanity's 340 ms figure is a hosted neural classifier call (TypeSafe Jev) and their 350 ms post-Enter figure is generative LLM time-to-first-token on Cerebras (read from sanity.io/engineering/the-0-5-second-agent). On our frozen 25-query held-out evaluation split (20 in-domain questions and 5 adversarial adjacent-domain abstentions, never tuned against), the scorer hits 22 of 25 (19 of 20 in-domain, 3 of 5 abstentions), and 35 of 35 on the in-repo regression gate (tests/docs-agent.test.mjs). When an OpenRouter key is configured on /docs/agent.json, post-Enter generation streams live LLM tokens over SSE while keeping keystroke speculation 100% local.

What does it give up?judgement: difficulty that leaves no trace in the words

A model reading the prompt can see difficulty that leaves no trace in the words, and rules over counts cannot. The labelled sets say how much that costs: 17 of 20 held-out requests went to an accepted tier, and all three misses sent a hard request one tier too cheap, multi-part professional asks with no difficulty words in them. On the 39 in-sample requests it is 39, but the weights were tuned against that set, so it is a regression gate and not evidence. Past the core properties, 12 of 12 breaks of the fast cue check and 15 of 15 of the 2026-09-29 rules turned the suite red too.

Known limitationsfrom the README

  • Scoring sees only what the text shows.A short question that needs deep expertise scores as a short question. The held-out misses are that shape.
  • Conversation state lives in memory.A restart forgets warm conversations, which costs money rather than correctness, and replicas do not share it.
  • Auto routes over two dialects.OpenRouter and OpenAI chat completions. Anthropic's Messages format is passed through for a caller that names a Claude model; routing it as auto would need an adapter that does not exist yet. There is no Responses API.
  • No per-client limits yet.No rate limits or spend caps per client key.
  • Jev cannot be asked again.Its answers cannot be replayed under a changed question, because the ledger holds no text to ask it with.
07 / 08The full recordPinned commit · all caveats

The full record

Every table, every judge, and every caveat.

Each part is the record as its results README and decision log state it at the pinned commit, closed until you open it.

MT-Bench, every policyeight answers per question, 2026-09-28 to 2026-09-30: the three deciding judges, then all six who graded

MT-Bench, 2026-09-28 to 2026-09-30held up, and the cost of each turn's answers

every judge's column

The table scrolls sideways.

PolicyFirst turns, of 80Follow-ups, of 79
MeanClaudeGPTGrok, neutralCostMeanClaudeGPTGrok, neutralCost
Switchyard, since 2026-09-30DeepSeek V4.1 Flash leads the light tier66706960$0.11768695466$0.215
Switchyard, before 2026-09-30gpt-6-luna led the light tier60527247$0.05155426849$0.090
Switchyard, Gemini leadingtried and not taken59715063$0.55956674263$0.819
Jev Routertypesafe/jev-router54566441$0.12362545860$0.288
Always DeepSeek V4.1 Flash, high effort68727062$0.09870715565$0.168
Always gpt-6-luna, high effort61517248$0.02655417151$0.039
Always Claude Opus 573756472$1.88266704968$2.835
Best answer each timehindsight80808080$0.04779777979$0.091

Eight answers per question, every one graded with all eight in view. Costs are the total for each turn's answers. The follow-ups are out of 79: DeepSeek's answer to question 140 failed twice, so that question is left out for every policy, and the Claude judge's column there is out of 77. Re-grading with eight answers in view moved the figures of the six-answer run before it by up to five answers (Jev Router's follow-ups from 67 to 62), so compare within this table, not with that run's. The change costs time as well as money: DeepSeek's median answer took 6.4 s and 11.0 s, against 4.2 s and 7.0 s for gpt-6-luna at high effort. "Best answer each time" is the cheapest answer within a point of the best, which nobody can pick in advance, so it is a ceiling rather than a policy.

Every judge alone, MT-Benchheld up, graded 2026-09-30

six columns

The table scrolls sideways.

PolicyThe three who decideAdded afterwards, none in the meanAll six
ClaudeGPTGrok, neutralDeepSeek V4 ProGemini 3.8 FlashQwen, neutral
First turns
Switchyard, since 2026-09-3070696050645357
Switchyard, before 2026-09-3052724734463138
Switchyard, Gemini leading71506361706463
Jev Router56644145503538
Always DeepSeek V4.1 Flash, high effort72706255675559
Always gpt-6-luna, high effort51724837473239
Always Claude Opus 575647272717573
Questions graded80808077798080
Follow-ups
Switchyard, since 2026-09-3069546652625465
Switchyard, before 2026-09-3042684927382636
Switchyard, Gemini leading67426350705862
Jev Router54586044584552
Always DeepSeek V4.1 Flash, high effort71556552645667
Always gpt-6-luna, high effort41715126382636
Always Claude Opus 570496849676669
Questions graded77797956787379

DeepSeek's and Google's models are among the answers, so their judges are interested parties; Qwen's vendor has none, so it is a second neutral judge beside Grok. Their roles were fixed before they graded, and the mean the rules were decided on did not move. DeepSeek V4 Pro was an unreliable judge on follow-ups: 56 of its 79 verdicts parsed after one retry pass, so its column rests on fewer questions than the others.

MT-Bench-101, the confirmation10 two-turn questions from each of eight tasks, drawn before any answer existed

Two-turn dialogues from MT-Bench-101 (Bai et al., ACL 2024), drawn by a seeded shuffle before any answer existed. Every model answered fresh, and each follow-up continues the dataset's own first reply, which is how that benchmark is built. The judges are the same three, with Qwen beside them and not in the mean.

MT-Bench-101, 2026-09-30held up, and the cost of each turn's answers

all five predictions held

The table scrolls sideways.

PolicyFirst turns, of 79Follow-ups, of 75
MeanClaudeGPTGrok, neutralQwen, neutralCostMeanClaudeGPTGrok, neutralQwen, neutralCost
Switchyard, since 2026-09-30DeepSeek V4.1 Flash leads the light tier7062726553$0.0567369676858$0.052
Switchyard, DeepSeek with reasoning offtried and not taken5868535962$0.0206671536464$0.015
Switchyard, before 2026-09-30gpt-6-luna led the light tier4324752714$0.0124536703015$0.009
Jev Routertypesafe/jev-router3426702716$0.0285043683826$0.033
Always Claude Opus 57878677878$1.4506873486873$1.024

Here the rules before 2026-09-30 sent every question to gpt-6-luna and both DeepSeek policies sent every one to DeepSeek, so this tests which model leads the light tier and nothing about the ladder above it. One first turn is missing because a DeepSeek answer came back empty, and five follow-ups because the run's spending cap stopped it first. The dataset's own replies were references for its maths only. With its reasoning off DeepSeek answered at a median of 3.5 s and 2.6 s, against 6.9 s and 7.0 s with it on, for about a third of the cost, and held up on 12 and 7 fewer answers by the mean.

The predictions, including the one that failedeleven, each written down before any of its run's answers existed

DeepSeek leading the light tier had to hold up at least as often as Jev Router on the follow-ups, beat the rules before under two of the three judges there, and not fall behind them on first turns. It did, and its higher cost was accepted before it became the default.

The eight-answer run, MT-Bench

one failed, cheaper than predicted
  1. HoldsAt least as many follow-ups held up as Jev Router.68 against 62.
  2. HoldsMore follow-ups than the rules before under the Claude judge and Grok, fewer under the GPT judge.27 more, 17 more and 14 fewer.
  3. HoldsAt least as many first turns as the rules before.66 against 60.
  4. FailsAt least three times the rules before in cost, more than Jev Router on first turns, and within a quarter of it on follow-ups.It cost less: 2.3 and 2.4 times the rules before, and 0.96 and 0.75 times Jev Router.
  5. HoldsGemini leading: more follow-ups than the rules before, fewer first turns.56 against 55, and 59 against 60, by one question each way.
  6. HoldsGemini leading costs more than DeepSeek leading.$0.559 and $0.819 against $0.117 and $0.215.

The confirmation, MT-Bench-101

all five held
  1. HoldsAt least as many as Jev Router on both turns.70 against 34, and 73 against 50.
  2. HoldsMore than the rules before on both turns, by the mean and under Grok.70 against 43 and 73 against 45; under Grok, 65 against 27 and 68 against 30.
  3. HoldsUnder the GPT judge, fewer than the rules before on at least one turn.The rules before lead there on both, 75 against 72 and 70 against 67.
  4. HoldsWith its reasoning off, fewer on both turns, at under half the cost.58 and 66 against 70 and 73, at $0.020 and $0.015 against $0.056 and $0.052.
  5. HoldsUnder Qwen, more than the rules before and Jev Router on both turns.53 against 14 and 16, and 58 against 15 and 26.

One thing changed mid-test on the six-answer run and is on the record: after 38 of the 80 follow-ups were graded, Jev Router's token cap went from 4,000 to 16,000, because DeepSeek had spent the whole cap thinking on 17 of its 74 follow-up answers and 4 of its 80 first ones. That graded the cap rather than the router, and the change could only help Jev Router. Both runs are kept, and the tables here use its answers as asked again.

What the benchmark taughtno signal we tried tells, prompt by prompt, when the cheap answer will fall short

The gap is one of prediction.

Two studies on 2026-09-29, over the same 160 graded MT-Bench prompts, graded then with six answers per question: four predictors of when gpt-6-luna's answer at high effort falls short, for $0.22, and two stricter labels to test them on, 1,125 judge calls for $5.49. The protocol and the prediction were written down before any judging, and the ones that failed are below.

Held up, of 80 by the judges' meanthe room a better router would have to win
First turns
  1. Always gpt-6-luna, high effort62$0.026
  2. Jev Router57$0.123
  3. Always Claude Opus 572$1.882
  4. The right one each time, in hindsight80$0.207
Follow-ups
  1. Always gpt-6-luna, high effort60$0.039
  2. Jev Router67$0.288
  3. Always Claude Opus 566$2.847
  4. The right one each time, in hindsight80$0.220

A router that always picked the right one of its own models would hold up on every question. On first turns the right answer each time costs more than Jev Router, and on follow-ups less. Either way it needs a signal that says, before answering, which prompts the light tier will lose. No signal tried here says it.

What ranked the first turns did not hold on the follow-ups.

Jev before answering, Jev shown luna's answer, Claude Haiku 4.5 scoring it, and gpt-6-luna scoring its own. None saw the grades or the reference answers. Thresholds were fitted on the first turns and applied unchanged to the follow-ups. A ranking score is how often a predictor rates a real failure as more likely than a success.

Ranking scores, fitted and then tested0.5 is chance

falls to chance
0.400.500.600.700.800.90chanceFirst turns, where the thresholds were fittedFollow-ups, where they were testedJev before answering0.800.57Jev before answeringJev shown the answer0.820.60Jev shown the answerClaude Haiku 4.50.670.57Claude Haiku 4.5gpt-6-luna on itself0.560.58gpt-6-luna on itselfJev before answering: ranking score 0.80 on the first turns it was fitted onJev before answering: ranking score 0.57 on the follow-ups it was tested onJev shown the answer: ranking score 0.82 on the first turns it was fitted onJev shown the answer: ranking score 0.60 on the follow-ups it was tested onClaude Haiku 4.5: ranking score 0.67 on the first turns it was fitted onClaude Haiku 4.5: ranking score 0.57 on the follow-ups it was tested ongpt-6-luna on itself: ranking score 0.56 on the first turns it was fitted ongpt-6-luna on itself: ranking score 0.58 on the follow-ups it was tested on

Each predictor, as a policyescalating to Opus on its threshold

The table scrolls sideways.

PredictorRanking, first turnsRanking, follow-upsFollow-ups held up, escalating to OpusCost
Jev before answering0.800.5760 of 80, never escalated$0.043
Jev shown the answer0.820.6064 of 80$0.380
Claude Haiku 4.50.670.5759 of 79, never escalated$0.167
gpt-6-luna on itself0.560.5862 of 75$0.459

No policy built on them beat Jev Router's 67 held-up follow-ups for $0.288. Part of the reason is the label: the three judges agree on whether luna's answer held up on only 43 of the 80 first turns and 34 of the 79 follow-ups, and any two of them agree 57% to 71% of the time. A predictor cannot learn a label the judges do not agree on.

Two stricter labels, and how far the judges agree on each.

An answer key, where MT-Bench has a reference for math, reasoning and coding: each judge marked luna-high's, Opus's and Jev Router's answers right or wrong by their final result. And a pairwise label on all 160 prompts: is luna-high's answer at least as good as Opus's, asked in both orders. Kappa discounts the agreement chance alone would give: 0 is chance, 1 is perfect.

All three judges agreeof the prompts each label covers

  1. Answer key, luna-high right or wrong50 of 55kappa 0.47
  2. 1-10 held up, the same 55 prompts33 of 55kappa 0.17
  3. Pairwise against Opus, first turns47 of 80kappa 0.41
  4. 1-10 held up, first turns43 of 80kappa 0.20
  5. Pairwise against Opus, follow-ups28 of 79kappa 0.14
  6. 1-10 held up, follow-ups34 of 79kappa 0.10

The answer key is cleaner than the 1 to 10 label on the same prompts, by the study's own rule of fifteen points more agreement and a higher kappa. The pairwise label is not cleaner on either turn: asking which answer is better did not remove the judges' disagreement.

Where an answer can be checked, the cheap one is right.

By the judges' majority, on the 55 prompts with a key, gpt-6-luna at high effort is right on 52, Claude Opus 5 on 52 and Jev Router on 51. The judges agree on every math and reasoning prompt; their only splits are 5 coding prompts. With so few failures there is nothing for a predictor to find.

  • 52 of 55gpt-6-luna, high effort, right by the judges' majority28 of 29 first turns, 24 of 26 follow-ups
  • 52 of 55Claude Opus 5, right by the judges' majority28 of 29 first turns, 24 of 26 follow-ups
  • 51 of 55Jev Router, right by the judges' majority27 of 29 first turns, 24 of 26 follow-ups

The judges prefer Opus, and split by vendor.

Asked which answer is better, the judges together rate luna-high at least as good as Opus on 20 of 80 first turns and 27 of 80 follow-ups, even in math, where its answers are right. The 1 to 10 label's point of slack had counted 43 and 35 of those losses as holding up.

The Claude judge (Claude Sonnet 5)

First turns21 of 80
Follow-ups20 of 80

The GPT judge (GPT-6 Sol)

First turns47 of 80
Follow-ups63 of 80

Grok 4.7, the neutral judge

First turns19 of 80
Follow-ups32 of 80

How often each judge rates gpt-6-luna's answer at high effort at least as good as Opus's, in both orders. The GPT judge is the most lenient toward OpenAI's model on both turns.

No predictor finds the losses.

Against the pairwise label the four predictors rank at chance, while a judge predicting the other two reaches 0.74 on first turns and 0.67 on follow-ups. Escalating on their thresholds gains on the follow-ups what escalating as many prompts at random would.

Against the pairwise labelranking scores, and what escalating gains

at chance

The table scrolls sideways.

PredictorRanking, first turnsRanking, follow-upsFollow-ups at least as good as Opus, escalatingAs many at random
Jev before answering0.520.4935 of 8034.3
Jev shown the answer0.610.4034 of 8034.3
Claude Haiku 4.50.550.4627 of 7927.0
gpt-6-luna on itself0.540.5333 of 7532.5

gpt-6-luna at high effort alone ties or beats Opus on 27 of 80 follow-ups, so any escalation gains when it loses two comparisons in three.

A smarter per-prompt predictor built from these signals will not close the gap.

Where MT-Bench can check an answer there is no gap in right answers. On follow-ups, Jev Router's lead over luna at high effort, 67 to 60 held up, is 1 prompt on the 26 with a key and 6 on the 54 without, where "better" is the judges' call and they split by vendor.

Nothing changes in the router on the strength of this. The benchmark that would say more is one on real traffic, with answers that can be checked.

Five predictions for the label study, two of them failed.

Fixed before any judging, and printed whatever they showed.

Fixed before any judging

two failed
  1. FailsThe answer key is far cleaner: all three judges agree on at least 85% of its prompts, against 60% for the 1 to 10 label, with a kappa of at least 0.6.Agreement is 91%, over the bar, but kappa is 0.47. All 5 splits are coding prompts.
  2. HoldsThe pairwise label is not cleaner: within 10 points of the 1 to 10 label's agreement on each turn.The GPT judge was the most lenient on both turns, as predicted.
  3. HoldsOn the key, luna-high is right on at least 80% of each turn's prompts.28 of 29 first turns and 24 of 26 follow-ups, so it has 2 follow-up failures, too few for the key to decide the predictor question.
  4. HoldsOn the pairwise label no predictor separates luna-high's follow-up failures, and Jev shown the answer stays below 0.70.It scores 0.40.
  5. FailsThe key's ceiling is above 0.85 on each turn.0.99 on first turns and 0.50 on follow-ups, a figure that rests on 2 failures.

Jev's one vote: one prompt in 56, at five times the cost.

The one rule change that reads the prompt lets Jev raise a request the rules scored at the floor, by one tier. It was scored on the 56 questions the other 2026-09-29 changes were not fitted to, and fired on 6: it helped on one coding prompt, hurt none, and changed nothing on the other five, at five times the rules' cost.

Nothing serves on the raise. It stays in the ledger as a count of what it would have raised, and the question on Jev says how that is asked.

Jev Router's answers, model by modelit picks its own model and effort per question

Jev Router chooses its own models

as it answered

It picked its model and effort per question, and on MT-Bench its answers came from gpt-6-luna, DeepSeek V4.1 Flash, Gemini 3.8 Flash and, rarely, Sonnet 5.5 or GPT-6 Sol. DeepSeek's and Google's models answered for it and for Switchyard's candidates, and xAI's, Alibaba's, DeepSeek's and Google's models judged. Every one of those vendors saw only public benchmark prompts.

First turnsMT-Bench answers

  • openai/gpt-6-luna43
  • deepseek/deepseek-v4.1-flash23
  • google/gemini-3.8-flash13
  • openai/gpt-6-sol1

Follow-upsMT-Bench answers

  • openai/gpt-6-luna25
  • deepseek/deepseek-v4.1-flash41
  • google/gemini-3.8-flash11
  • openai/gpt-6-sol1
  • anthropic/claude-sonnet-5.51
How every figure was madesix rules, the same for every policy

Six rulesthe same for every policy

  • QuestionsMT-Bench's 80 questions, first turns and follow-ups: fastchat/llm_judge/data/mt_bench/question.jsonl in lm-sys/FastChat, retrieved 2026-09-28, sha256 119565adbab82227089cefdb44c8d7e2cf04dc0a0ec233634c82e7d4e2a944f7. And MT-Bench-101's two-turn dialogues, 10 from each of eight tasks, drawn with a shuffle seeded mtbench101-confirmation-2026-09-30 from github.com/mtbench101/mt-bench-101 at bc18b3e2 (Apache-2.0), whose file has sha256 1a1e433acf7e2ef013b5001b60fd8e48bc3e4202460727e25b2ea8575aae7d80 and the sample drawn from it c5db7ce5a943bdb29481b78b4f7b85060d1ea1a03b88b75abfe189a6381708f5.
  • Answersgpt-6-luna at low, medium and high effort; Claude Sonnet 5 and Claude Opus 5 at low effort; DeepSeek V4.1 Flash at high effort, and on MT-Bench-101 also with its reasoning off; Gemini 3.8 Flash at high effort on MT-Bench; and Jev Router, which picks its own model and effort. All through OpenRouter, with room for 4,000 tokens, 8,000 for luna at medium, and 16,000 for luna at high, DeepSeek and Gemini. Jev Router had 4,000 on MT-Bench, where the answers that cap cut off were asked again with 16,000, and 16,000 on MT-Bench-101.
  • JudgesClaude Sonnet 5, GPT-6 Sol and Grok 4.7 decide, each scoring every answer to a question from 1 to 10 in one blind prompt, with the benchmark's reference answer in view where it has one. The answer order was meant to be shuffled per question and judge, and was not: later places scored up to 0.44 points lower, Jev Router lost up to 0.13 to where it sat, and gpt-6-luna at low effort sat last in nearly every prompt, at a cost not measured. Grok is the neutral judge: xAI has no model among the answers. Qwen 3.8 Max, from a vendor with none either, graded both sets beside them, and DeepSeek V4 Pro and Gemini 3.8 Flash graded MT-Bench. None of the three is in the mean.
  • Held upWithin one point of the best answer the tiers gave to the same prompt. Measured against the best answer rather than the strongest model's, so a prompt where the strong model stumbles cannot flatter a cheaper pick.
  • Follow-upsEvery answer continues the same conversation, so the answers stay comparable: on MT-Bench the first question and Opus 5's first answer, on MT-Bench-101 the dataset's own first reply. The rules decide each follow-up with their own first-turn decision as the warm conversation a minute before, as the live router would.
  • PoliciesEach policy is scored on the answers it would have served. The rules and Jev Router are decided per question; a decision at an effort nothing answered at is scored on the nearest answer, and the record counts it as such. The reports were written while gpt-6-luna led the light tier, so their shipped rules are the rules before 2026-09-30, and the DeepSeek candidate is what ships now.
The runs, with what each cost2,547 answers, 2,208 gradings, $39.01

The record also holds 1,125 label verdicts, 316 Jev decisions and 633 escalation predictions, which what the benchmark taught reads.

The runs in the recordbench/results/README.md

The table scrolls sideways.

RunDateWhatAnswersGradingsCost
2026-09-28-turn1-four-answers2026-09-28/29First turns: luna, Sonnet 5 and Opus 5 at low effort, plus Jev Router; Claude and GPT judges. The first run stopped at 24 of 80 on an OpenRouter daily guardrail and a second finished it320160$3.25
2026-09-29-turn1-six-answers2026-09-29The same questions with luna also at medium and high effort; all six answers re-graded together, then graded again by Grok. Jev Router's answers had a 4,000-token cap, which cut off 4160 more240$1.75
2026-09-29-turn1-six-answers-reask2026-09-29The final first-turn data: Jev Router's 4 cut-off answers asked again with 16,000 tokens, and their questions re-graded by all three judges. report-with-candidate.* scores the shipped rules4 more12$0.20
2026-09-29-turn2-first-measured2026-09-29Follow-ups as first measured: 38 of 80 before the credit ran out, and Jev Router cut off on 17 of 74 answers. Kept as the record411110$3.74
2026-09-29-turn22026-09-29The final follow-up data: all 80, Jev Router's cut-off answers asked again, three judges86 more152$3.67
overhead-2026-09-28.md2026-09-28Router latency and throughput against an instant upstream
2026-09-29-escalation2026-09-29Four predictors of when luna's high-effort answer falls short, over the 160 graded prompts (below)633 predictions$0.22
2026-09-29-labels2026-09-29Cleaner labels from the graded answers: the answer key checked on 55 prompts, and luna-high against Opus in both orders on all 160 (below)1,125 verdicts$5.49
2026-09-30-turn1-vendors2026-09-30First turns: DeepSeek V4.1 Flash and Gemini 3.8 Flash at high effort beside the six recorded answers, which bench/regrade.mjs copied from 2026-09-29-turn1-six-answers-reask, and all eight re-graded by all three judges (below)160 more240$3.06
2026-09-30-turn2-vendors2026-09-30Follow-ups, the same way from 2026-09-29-turn2. 79 of 80 complete: DeepSeek's answer to question 140 failed twice with an empty response159 more235$4.88
2026-09-30-turn1-vendors, judges added2026-09-30DeepSeek V4 Pro, Gemini 3.8 Flash and Qwen 3.8 Max grade the same eight answers afterwards. Their grades follow the panel's in grades.jsonl and never enter the panel's mean (below)236$1.75
2026-09-30-turn2-vendors, judges added2026-09-30The same over the follow-ups. The $5 cap for both turns stopped the retry pass, so DeepSeek V4 Pro has 56 of 79207$3.09
2026-09-30-confirm-turn12026-09-30The confirmation on fresh questions: 80 MT-Bench-101 first turns (2026-09-30-confirm-questions), every model answering fresh, with DeepSeek at high effort and with reasoning off, graded by the panel and Qwen 3.8 Max (below). 79 complete: one DeepSeek answer came back empty639316$4.35
2026-09-30-confirm-turn22026-09-30The follow-ups, each continuing the dataset's own first reply. The $3.50 cap stopped the run at 75 of 80 complete608300$3.56

Not listed: one run that called no model and cost nothing, measuring a kind of request this page does not cover. The totals are the same without it.

Ten caveatsread every figure on this page with these

Ten caveats

a party's claim
  1. The judges prefer their own vendor's answers.On MT-Bench's first turns the Claude judge put Opus 22 answers ahead of gpt-6-luna at high effort, and the GPT judge 14 behind. A claim across vendors needs the per-judge columns, not the mean alone.
  2. Both sets are chat.80 public prompts per turn, not real traffic, and not the traffic any router here was built for.
  3. Eighty questions per turn is a small sample.One question is 1.25 points of held up, so a gap of two or three answers is within noise.
  4. Longer answers may be part of the gap.On MT-Bench-101 luna's answers have a median of 419 and 282 characters, against DeepSeek's 1,178 and 673, and judges tend to reward length though told not to. On the maths, graded against references, DeepSeek still scored above luna, 8.5 against 7.9 and 9.1 against 8.6 by the judges' mean score.
  5. DeepSeek was chosen from MT-Bench's own prompts.Jev Router's DeepSeek answers did well on them, which is why it was tried, and why MT-Bench-101 was run with its predictions written down first.
  6. Not every question is in every figure.The MT-Bench follow-ups are out of 79, after an answer that failed twice, and MT-Bench-101 has 79 first turns and 75 follow-ups, after an empty answer and a spending cap that stopped the run.
  7. The follow-ups continue a fixed first answer.Opus's on MT-Bench, which may favour Opus continuing its own text, and the dataset's on MT-Bench-101.
  8. Jev Router's price and model mix are its own.It chose its models itself, and on MT-Bench-101 sent most questions to gpt-6-luna: 70 of 80 first turns and 55 of 76 follow-ups.
  9. The rules measured are the ones this repository ships.DeepSeek has led the light tier here since 2026-09-30. Switchyard's shadow deployment runs whichever release was last published to it, and which one that is is read on that server, not from the record.
  10. Vitna built Switchyard and ran this benchmark.That makes every figure here a party's claim, which is why each judge's column is printed and the failed prediction is kept.
What was runthe commands, as the record prints them

The code is in a private repository, so these say what was run rather than something to run today. A rerun asks every model again, so its answers and scores will differ in detail from these; the runs that call models stop at their budget and resume from their own files.

OPENROUTER_API_KEY=... node bench/quality.mjs --questions question.jsonl --out out1 --budget 10
OPENROUTER_API_KEY=... node bench/quality.mjs --questions question.jsonl --out out2 --turn 2 \
  --history out1/answers.jsonl --history-arm strong --candidate bench/candidates/2026-09-29-light-high.json --budget 10
node bench/quality.mjs --report --out out1 --questions question.jsonl
node bench/overhead.mjs
node bench/labels.mjs --plan --questions question.jsonl
OPENROUTER_API_KEY=... node bench/labels.mjs --questions question.jsonl --out out3 --budget 7.5
node bench/regrade.mjs --from bench/results/2026-09-29-turn1-six-answers-reask --to out-v1
OPENROUTER_API_KEY=... node bench/quality.mjs --questions question.jsonl --out out-v1 --vendors --budget 4 \
  --candidate bench/candidates/2026-09-30-light-deepseek.json --candidate bench/candidates/2026-09-30-light-gemini.json
node bench/mtbench101.mjs --in mtbench101.jsonl --out mt101
A=deepseek,deepseek_off
J=sonnet,sol,grok,qwen-max
C="--candidate bench/candidates/2026-09-30-light-deepseek.json --candidate bench/candidates/2026-09-30-light-deepseek-off.json"
OPENROUTER_API_KEY=... node bench/quality.mjs --questions mt101/question.jsonl --out out-c1 --extra-arms $A --judges $J --budget 8 $C

The decisions at the head of this page are computed from this repository when the site is built. Every measured figure here is copied from Switchyard's benchmark record in convexityos/vitna at commit e27ca78d, committed 2026-09-30, and checked against that commit before it is published. The repository is private, so the method and the caveats are on this page rather than behind a link.

08 / 08The waitlistShadow waitlist · full record

The waitlist

Tell us what you would route.

Nothing installs yet. Leave a work email and the traffic you would point at Switchyard, and we will write when it can be run outside Convexity.

Your work email and what you describe, stored to write to you when Switchyard can be run outside Convexity. Nothing else is collected, and no date is promised.