Technical note8 min read
How Beam 1.0 got past its tokens-per-second ceiling
Beam 1.0 runs at 348.8 tokens a second across its hosts and 537.3 on the fastest. One token a pass stops near 300. What sets that ceiling, and the ways past it.
Beam's two fastest hosts run past what one token a pass has reached.
Output tokens a second for each host serving Beam 1.0, timed by Artificial Analysis on a 10,000-token prompt, against one request in Reflection's and vLLM's own tests.
- A host, as Artificial Analysis times it
- One request, in its makers' and vLLM's tests
Where these figures come from. Every speed here is cited and was read on 20 October 2026: the hosts' from Artificial Analysis, which times each one on a 10,000-token prompt and reports the median of its last 72 hours, and the single-request figures from Reflection, which made Beam, and vLLM, which serves it. Arithmetic on Beam's weights is ours, from the tensor sizes in its published files, and says so where it appears. Beam is not on Vitna's boards yet, so nothing here is a reading of the record.
Beam 1.0's 348.8 tokens a second is the middle of its market, not its top. Reflection announced Beam, a 501 billion parameter open model, on 5 October,4 and Artificial Analysis times it at 348.8 output tokens a second, the median across the four hosts that serve it.1 The fastest, Baseten, serves it at 537.3; the slowest, Nebius, at 215.3.2 Same weights, 2.49 times the speed.
How high can it go? Three numbers bound it, and the one people usually blame, memory, is the loosest. One token per pass of the model stops near 300 a second for a single request on Nvidia's newest GPUs. The memory on those GPUs would allow, by our arithmetic, about 2,749. And by drafting several tokens and checking them in one pass, Beam's makers have run one request at 874.
Artificial Analysis, read 20 October 202612; vLLM and Reflection, 16 October 202689. 303 is our arithmetic: one second over vLLM's 3.30 ms pass.
Four hosts, one set of weights
| Host | Weights | Tokens a second | First answer token | Blended price |
|---|---|---|---|---|
| Baseten | NVFP4 | 537.3 | 4.09 s | $0.78 |
| CoreWeave | NVFP4 | 354.5 | 6.48 s | $0.57 |
| Together AI | NVFP4 | 301.1 | 7.15 s | $0.23 |
| Nebius | FP8 | 215.3 | 10.59 s | $0.33 |
Artificial Analysis, read 20 October 2026.2 Speed is output tokens a second while the model is writing; the first answer token comes after Beam's reasoning; the blend weights cache hits, input and output 7:2:1, per million tokens.
Nothing about the model differs between those rows. Three hosts run the four-bit NVFP4 checkpoint, and Nebius the eight-bit FP8 one, which is 503 GB of weights against 293 GB.7 The fastest host has the dearest blend, and the cheapest is third. What separates them is how each one serves the model: the chips, the software, and how many people's requests share each pass. None of the four has published how it serves Beam, as far as we could find.
One pass, one token
A language model writes one token at a time, and every token is a full trip through the model: for Beam, 52 layers in a row, each waiting on the one before.5 The next token cannot start until this one is chosen, so a single request goes exactly as fast as one pass allows.
vLLM, which served Beam from its first day, timed that pass for one request on eight Nvidia GB200 GPUs at 3.30 milliseconds, with about 100,000 tokens of conversation behind it, down from 8.95 before coding agents worked through its kernels.9 That is 303 tokens a second at most. Reflection's own tests, on four of Nvidia's B300s, put one request at 288 to 289 tokens a second.8
Not the memory
The usual explanation for that limit is memory: every pass reads the model's weights out of the GPUs' memory, and reading takes time. For Beam it accounts for little of the time.
Beam holds 501.3 billion parameters but uses 23.4 billion on each token, because each of its 49 expert layers sends a token to 8 of its 256 experts.56 Adding up the tensors one token touches in the published four-bit files gives 23.27 GB a pass, and the biggest share is not the four-bit experts but attention, which those files keep at 16 bits.7
| What a token reads | Precision | GB a pass |
|---|---|---|
| Attention, all 52 layers | BF16 | 10.90 |
| Experts, 8 of 256 in each of 49 layers | NVFP4 | 8.67 |
| Feed-forward, the first 3 layers | BF16 | 1.88 |
| Output head | BF16 | 1.55 |
| Expert routers, 49 layers | FP32 | 0.25 |
| Total | 23.27 |
Our tally from the tensor sizes in the NVFP4 files.7 Each row is rounded down, and the total is the sum before rounding. Norms, biases and the one embedding row a token looks up add under 3 MB.
Eight Blackwell GPUs read their memory at 64 terabytes a second between them, 8 TB/s each.1011 At that rate 23.27 GB takes 0.363 milliseconds; with the cache of a 100,000-token conversation, another 1.88 GB, it takes 0.393. By our arithmetic that is room for 2,543 to 2,749 tokens a second. vLLM's pass takes 3.30 milliseconds, so the reading fills under an eighth of it.
The rest is the pass waiting on itself. A layer's work is split across the eight GPUs, and twice in every layer they stop to add their pieces together before the next step can start, 104 stops a pass. Most of vLLM's seven changes fused the small steps around those stops; the first, which merged the work at each of the 104 boundaries into one kernel, cut the pass by 40%, and by the end an expert layer ran as 13 kernels.9 Reflection's four-GPU figure points the same way: half the GPUs and half the memory bandwidth, and vLLM's eight are under 5% faster. That compares different software on different prompts, so it is a hint rather than a test.
More than one token a pass
If most of a pass is fixed cost, the way past the ceiling is to get more than one token out of each. Beam ships with a helper for exactly that, DFlash2: a 1.93 billion parameter drafter that guesses the next 7 tokens at once. Beam checks all 7 in a single pass and keeps the guesses it agrees with, adding one token of its own, so a pass yields up to 8. Beam still decides every token, and Reflection says the output is distributed exactly as Beam's own would be.8
How many guesses survive depends on how predictable the writing is.
| Task | Tokens a pass | Without the drafter | With it |
|---|---|---|---|
| GSM8Kmath word problems | 5.88 | 288 | 874 |
| MATH-500competition math | 5.39 | 289 | 864 |
| HumanEvalcode | 4.68 | 289 | 733 |
| MBPPcode | 4.32 | 289 | 665 |
| MT-Benchchat | 4.11 | 289 | 565 |
Reflection's tests: one request at a time, greedy decoding, Beam NVFP4 on four B300 GPUs.8 Tokens a pass is the mean each checking pass yields, out of 8.
A checking pass does cost more than a plain one. 874 tokens a second at 5.88 tokens a pass is 148 passes a second, 6.72 milliseconds each, against 3.47 without the drafter. It is still a good trade: 3.0 times the speed on GSM8K and 1.9 times on chat. Long prompts make the guesses worse; on recorded agentic coding sessions they fell from 5.72 tokens a pass with prompts under 2,000 tokens long to 4.23 at 64,000 to 128,000.8 vLLM's own run on eight GPUs reached 708 tokens a second at 3.7 a pass.9
This method has its own ceiling. If every guess held, each 6.72 millisecond pass would yield 8 tokens, about 1,189 a second by our arithmetic.
Speed is sold by the seat
None of those runs is what a host does all day. A host earns its money by putting many requests through each pass, since the weights it reads once serve them all, and every request in the batch waits for the whole pass. Reflection's table shows the trade. With the drafter, one request ran at 874 tokens a second; 32 at once ran at 10,799 between them, 337 each. Without it, 288 alone and 122 each among 32.8 The host gets 12 times the tokens; each user gets well under half the speed.
Some hosts sell that trade as a tier.
| Model and host | Tier | Tokens a second | Cost per task |
|---|---|---|---|
| GPT-6.1 Sol (max)OpenAI | Standard | 55 | $0.72 |
| Ultrafast | 384 | $4.35 | |
| DeepSeek V4.1 Flash (max)Baseten | Standard | 444 | $1.01 |
| FAST | 697 | $2.03 |
Artificial Analysis's provider board, read 20 October 2026.3 Cost per task is its weighted average cost of one Intelligence Index task.
OpenAI charges 6.0 times as much for 6.9 times the speed, and Baseten 2.0 times as much for 1.5 times. Neither says what changes behind the faster tier, a smaller batch or other hardware, but both charge more for the faster seat than for the slower one.
Where the ceiling is
For Beam on today's Nvidia GPUs, the ceilings stack up like this. One token a pass stops near 300 tokens a second for one request, set by how long a pass takes rather than by memory. Memory alone would allow about 2,749. The drafter has reached 874, and about 1,189 if every guess held. And the hosts, at 215.3 to 537.3, sit where their batches put them.
A model near Beam's size has been pushed further. In May 2025 Nvidia ran Meta's 400 billion parameter Llama 4 Maverick at over 1,000 tokens a second per user on one eight-GPU B200 server, measured by Artificial Analysis, with a drafter and a long list of fused kernels.11
Past that, the routes change the machine or the model. Cerebras builds a chip from a whole silicon wafer with 44 GB of memory on it, read at 43.2 petabytes a second, which is 675 times the eight-GPU server's by our arithmetic.12 Its gpt-oss-120b is the fastest endpoint on Artificial Analysis's board, at 1,789 tokens a second.3 Beam's four-bit files would fill more than 6 of those wafers, and none of the four hosts Artificial Analysis times runs it on one.2 K2 Horizon 375B A23B, from the Institute of Foundation Models, is named for the same 23 billion active parameters as Beam, and its own endpoint runs at 1,109 tokens a second;3 its makers say it writes blocks of tokens in parallel rather than one at a time.13
Of the 502 endpoints with a measured speed on that board, the median runs at 130 tokens a second, 6 run above 1,000, and Baseten's Beam ranks 22nd, by our count.3 So 348.8 is not Beam's ceiling. It is where four hosts' choices about how to serve it landed in its first week, with room above it for anyone willing to pay for a faster seat.
Sources
- 1PageBeam 1.0: intelligence, performance and price analysisArtificial Analysisread
- 2PageBeam 1.0: API provider performance benchmarking and price analysisArtificial Analysisread
- 3PageLLM API providers leaderboardArtificial Analysisread
- 4ReleaseIntroducing BeamReflection, 5 October 2026read
- 5CardBeam-1.0-501B-A23BReflection, Hugging Faceread
- 6ConfigBeam-1.0-501B-A23B config.jsonReflection, Hugging Faceread
- 7FilesBeam-1.0-501B-A23B-NVFP4Reflection, Hugging Face (tensor sizes read from the safetensors headers)read
- 8CardBeam-1.0-501B-A23B-DFlash2, the drafter and its evaluationReflection, Hugging Faceread
- 9PostServing Reflection Beam on vLLM from day 0vLLM, with Inferact, Reflection and Nvidia, 16 October 2026read
- 10SpecGB200 NVL72 specificationsNvidiaread
- 11PostBlackwell breaks the 1,000 TPS/user barrier with Meta's Llama 4 MaverickNvidia Technical Blog, 22 May 2025read
- 12SpecWafer-Scale Engine 3Cerebrasread
- 13ReleaseMBZUAI's Institute of Foundation Models launches K2 HorizonMBZUAI, via Zawya, 3 September 2026read
More posts
Why a Google SERP API now returns five results, not ten
Google now unlocks each result link a little later than the one above it, the tenth about 24 seconds after the page. Five results arrive before the long waits: 2,273 of 2,500 live searches answered, 12 at a time, at a median of 4.19 seconds.
Claude API pricing: how Opus fell from $75 to $20
A million tokens of Opus output cost $75 in May 2025. Opus 5.5 launched at $20 in September. Every Claude model's price today, every change since, and what one coding session costs.
Who owns the AI companies, and who still holds the votes
Elon Musk owns under half of SpaceX, home of xAI, and casts 82% of its votes. Microsoft owns about a quarter of OpenAI and names none of its directors. How the money and the votes came apart at 13 AI labs.
How DeepSeek went from price war to price rise
DeepSeek's cheapest model costs $1.20 per million output tokens at peak, up from $0.28 in April. Every seller's price today, and every release's price and score.
How a stock trader came to own DeepSeek
Liang Wenfeng still controls DeepSeek. His hedge fund paid for it for three years, and the outside investors who bought in this June hold under 9%.
