Open models, served from your own machine.
One command installs Anchor with a model that fits your machine and starts a local OpenAI and Anthropic server on 127.0.0.1. With an 8 GB NVIDIA GPU and 16 GB of memory, it serves Qwen3-30B-A3B by streaming experts from memory or drive.

Amber is the experts picked off the drive.
- 76.7
- tokens a second after "The capital of France is", Qwen3-30B-A3B decoding on an RTX 3070 with 8 GB, where llama.cpp's best on the same file was 29.2
- 5.1 GiB
- of memory to serve that model with its experts read off the drive, on a Ryzen 7 3700X and that RTX 3070, against 17.5 holding the whole file
- 17.3 ms
- a token of OLMoE-1B-7B on the RTX 3070, where AirLLM 4.0.0 took 11,249 ms on the same checkpoint
A download, with nothing to sign up for.
Release v0.2.1 is the latest on GitHub. It needs no account and no key, and the installer chooses the first model in its catalogue whose needs the machine meets. Every figure that names a machine was measured; the rest describes the design.
- Release
- v0.2.1 on GitHub (not on npm): installers for Windows and Linux on x86-64, plus a binary for macOS on Apple silicon.
- Licence
- Apache-2.0, with the full C11 and CUDA source public and built with Make or CMake.
- Access
- A direct download: no account, no API key, no telemetry, and nothing billed.
- Processors
- x86-64 with AVX2, or ARM with NEON; every model the engine supports runs on a CPU.
- GPUs
- NVIDIA CUDA (Turing to Blackwell), checked by hand on an RTX 3070. macOS runs on CPU; AMD HIP portability and multi-GPU layer planning are on main (Gate A13 foundation), with AMD/multi-GPU benchmarks pending.
- Models
- The two installer catalogue models below, plus SmolLM2-135M, OLMoE-1B-7B, and bge-small-en-v1.5 embeddings.
| Model | The machine needs | Runs on | Weights | Context | Experts |
|---|---|---|---|---|---|
| Qwen3-30B-A3B at Q4_K_Mqwen3-30b-a3b | an NVIDIA GPU of 7.5 GiB and 14 GiB of memory | the NVIDIA GPU | Qwen3-30B-A3B-Q4_K_M.gguf18.6 GB, from Qwen/Qwen3-30B-A3B-GGUF | 32,768 tokens with 46 GiB of memory or more, else 16,384 tokens | the whole file held in memory with 30 GiB of memory or more, else read off the drive into 4,096 MiB |
| SmolLM2-360M-Instructsmollm2-360m-instruct | 3 GiB of memory | the CPU | model.safetensors724 MB, from HuggingFaceTB/SmolLM2-360M-Instruct | 4,096 tokens | none: a dense model, held in memory |
VITNA_MODEL names a model of the catalogue in place of the installer's choice.Read off install/catalog.json at release v0.2.1; sizes from each file's pin.A recorded install, then a REPL in your browser.
The two runs are what the installer did on fresh machines and on one GPU, as the engine's README records them. The REPL answers in this tab, and asks your machine nothing until you type /connect.
Recorded on 2026-10-06 against release v0.2.0, in CI on GitHub's hosted runners and by hand: the engine's README, How gate A12 was checked.
Illustrative: no model runs in this tab, and the panel is the design's, not a reading of your machine. /models, /install and /endpoints print the release's own facts. /connect asks http://127.0.0.1:8765/v1/health once; the engine sends no CORS headers, so a page on another origin cannot read its replies even while it runs, and its own chat page at http://127.0.0.1:8765/ is where to talk to it.
A reply in the shape your client already reads.
A chat completion from Qwen3-30B-A3B that calls a tool, beside the request that asked for it. The fields are the engine's, from its README; the words, ids and counts are written for this page.
curl http://127.0.0.1:8765/v1/chat/completions \
-H "content-type: application/json" -d '{
"model": "qwen3-30b-a3b",
"messages": [{ "role": "user",
"content": "What does notes.md say about the release?" }],
"tools": [{ "type": "function", "function": {
"name": "read_file",
"parameters": { "type": "object",
"properties": { "path": { "type": "string" } } } } }]
}'HTTP/1.1 200 OK
x-vitna-seed: 4185073029631457214
{
"id": "chatcmpl-6f1c2a9e04b7d358",
"object": "chat.completion",
"model": "qwen3-30b-a3b",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"reasoning_content": "They want notes.md read first.",
"tool_calls": [{
"id": "call_9a3e51c07d2b4f86",
"type": "function",
"function": {
"name": "read_file",
"arguments": "{\"path\": \"notes.md\"}"
}
}]
},
"logprobs": null,
"finish_reason": "tool_calls"
}],
"usage": {
"prompt_tokens": 172, "completion_tokens": 38,
"total_tokens": 210,
"prompt_tokens_details": { "cached_tokens": 0 }
}
}reasoning_content- Qwen3's reasoning, the text it wrote between its think tags, kept apart from the answer. Thinking is on unless a request asks it off, through
chat_template_kwargs.enable_thinkingorreasoning_effort. tool_calls,finish_reason- Each call the model wrote, for your code to run; the reply ends
tool_calls, and you send the result back as atoolmessage. Anthropic's Messages returns the same reply as athinkingblock and atool_use, and the Responses API as areasoningitem and afunction_call. usage.prompt_tokens_details.cached_tokens- The prompt tokens whose keys and values were reused from the request before rather than computed again: none on a first turn, most of the prompt on the next.
prompt_tokenscounts after the chat template; Anthropic counts the reuse ascache_read_input_tokens. x-vitna-seed- The seed the reply was sampled with, so a sampled reply can be reproduced by sending it back as
seed. x-vitna-ignored- Sent only when a request carried a field the server does not know, and names it rather than dropping it unseen. A field that would change the output and is not implemented is refused with a 400 instead.
curl http://127.0.0.1:8765/v1/embeddings \
-H "content-type: application/json" -d '{
"model": "bge-small-en-v1.5",
"input": ["Represent this sentence for searching
relevant passages: how do tides work"]
}'{
"object": "list",
"data": [{ "object": "embedding", "index": 0,
"embedding": [ /* 384 values */ ] }],
"model": "bge-small-en-v1.5",
"usage": { "prompt_tokens": 18, "total_tokens": 18 }
}Illustrative: the shapes are the engine's, an embedding 384 values normalized to length one (README, Serve the model and Serve an embedding model); no model wrote these replies. A model that generates answers /v1/embeddings with a 404 that says so, and an embedding model answers generation the same way.
One command, on Windows or Linux.
Release v0.2.1's installer fetches the engine and checks it against the SHA-256 the release wrote into it, asks the engine which model fits the machine, fetches that model's files at their pinned revisions, resuming any part already there, checks every file and starts the server with its chat page. Which model is the catalogue's call, above: Qwen3-30B-A3B with an NVIDIA GPU, the whole file in memory or its experts read off the drive, and SmolLM2-360M-Instruct on any other machine.
The installer, in the engine's README# Windows, in PowerShell
irm https://github.com/convexityos/vitna-anchor/releases/latest/download/install.ps1 | iex
# Linux on x86-64
curl -fsSL https://github.com/convexityos/vitna-anchor/releases/latest/download/install.sh | sh
# macOS on Apple silicon: no installer. The release's binary runs on the CPU;
# fetch a model with the engine's scripts/fetch-model.mjs, then serve it:
./vitna-anchor-macos-arm64 serve --model models/smollm2-360m-instruct# Any OpenAI client. Only the base URL changes. from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8765/v1", api_key="placeholder", ) res = client.chat.completions.create( model="qwen3-30b-a3b", messages=[{"role": "user", "content": "Hello"}], )
The model is the name the installer gave it, qwen3-30b-a3b or smollm2-360m-instruct, which GET /v1/models lists; any other name answers 404. The key is a placeholder: nothing checks it, which is why the server listens only on 127.0.0.1 unless told otherwise.
# Anthropic's base URL has no /v1.
ANTHROPIC_BASE_URL=http://127.0.0.1:8765 \
ANTHROPIC_API_KEY=placeholder \
ANTHROPIC_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_SONNET_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_OPUS_MODEL=qwen3-30b-a3b claudeAs the engine's README sets Claude Code up, every tier it asks for named as the one model. Its prompts are long, so give the server a context of 32K; each turn after the first reuses nearly all of the one before.
- Package
- vitna-anchorrelease binaries; not on npm
- Server
- 127.0.0.1:8765OpenAI and Anthropic APIs
- Dependencies
- 0libc; CUDA linked in
- Checkpoints
- SafeTensors, GGUFQ2_K to Q8_0, multi-shard GGUF
- Engine
- C11 coreCPU, CUDA; AMD HIP layer in main
- Licence
- Apache-2.0source on GitHub
Three jobs it has been checked doing.
Each is one the engine's README records as checked, by hand or in CI, with the route it rides and where a client points.
- 01
A coding agent that stays on your machine
Claude Code speaks Anthropic's Messages and Codex OpenAI's Responses, and both work against the server with only a base URL and a model name set. Against Qwen3-30B-A3B on an RTX 3070, Claude Code 2.1.283 read and edited a file and Codex 0.160.0 called its tools, checked by hand for gate A11. Their prompts are long, about 14,000 tokens for Claude Code's, so the README serves them a context of 32K.
base URLs http://127.0.0.1:8765 and http://127.0.0.1:8765/v1POST /v1/messagesPOST /v1/responses - 02
A private chat in the browser
The chat page is built into the engine: it streams a reply with Qwen3's reasoning folded away above the answer, renders the model's Markdown from escaped text, stops on Esc, and loads nothing from anywhere, its content security policy letting it talk to its own server alone. The installer opens it once the server is up (gate A12).
http://127.0.0.1:8765/ in any browserGET /POST /v1/chat/completions - 03
Embeddings for search over your own documents
bge-small-en-v1.5 runs on the CPU and serves OpenAI's embeddings request and response, 384 values a text, within 1e-5 of its pinned reference (gate A14). It is not in the installer's catalogue: fetch it with the engine's scripts/fetch-model.mjs and serve it with --model, as its README's "Serve an embedding model" shows.
a server started with the embedding modelPOST /v1/embeddings
Four places a weight can live, fastest first.
A model is not one file that has to fit somewhere. It is layers that are always hot, a cache of everything you have already said, and a long tail of experts that each token needs a few of. Anchor puts each on the store that suits it; the bar beside each store is its nominal bus figure.
/v1/chat/completions/v1/messages/v1/embeddingsTokenizes prompt, picks active experts each layer, streams SSE replies back.
zero dependenciesFallback: CPU AVX2 / NEON when no qualifying GPU exists
- 01
Host RAMthe layers every token touches, and each conversation's cache
Embeddings, attention and the router, the parts every token passes through, stay resident for the life of the process, and on an NVIDIA GPU they live on the card instead. Each sequence of the server keeps the key-value cache of the last request it served, so a conversation sent back with one more turn reuses its history and computes only what is new.
44.8 GB/sDDR5-5600, one module's peak - 02
NVMe, stripedexpert slabs across two or four drives, as designed
Slabs would be interleaved round-robin across physical drives in 4,096-byte sectors, so parallel unbuffered reads land on every drive at once. The command line lays a file out that way today. The engine does not read across drives yet, and nothing has measured what a second drive adds.
not measuredthe engine reads one drive today - 03
NVMe, one driveexperts read as the layers want them
Built and measured. A mixture's experts are read with direct I/O, past the page cache, by four threads into a cache in memory of the size you give it, and the experts the next layer will probably want start reading while this one runs. On a Gen3 drive that read them at 1.7 GB/s, OLMoE-1B-7B decoded 5.8 tokens a second on a Ryzen 7 3700X through an 8 GiB cache, against 6.1 with the whole model in memory, and 2.2 through 4 GiB, where each token waits on its reads.
7.45 GB/sPCIe Gen4 x4, 12.4 on Gen5 - 04
The wirecloud providers, on overflow only
Nothing leaves the machine during local generation: the engine's server listens on 127.0.0.1 unless told otherwise and makes no outbound connection. The proof the design adds, a count of socket egress bytes and a rolling SHA-256 over the trajectory on every completion, is not built. When a turn cannot fit locally, it would leave as a lane on vitna-runtime, sent to the model a signed policy names.
0 bytessocket egress during local generation
Eighteen parts, each named after the file it lives in.
Every module maps to a real file under engine/src/, runtime/, or platform/runtime/ so every claim can be checked against the source. Sixteen of the parts run in the engine today; the two designed and not built say so.
api.c · server.c · chat_page.cjsonpfx.c · grammar.mjsencoder.c · wordpiece.ccrypto.c · not wired yetmodel_cuda.cu · kernels.capi.c · model.c · --parallelwarp.c · exact.c · pool.cresolve.mjs · policy-storegguf.c · safetensors.c · quant.cexpert_stream.c · model.cinstall.c · catalog.cruntime/stripe.mjs- 01BuiltDirect NVMe DMA Streamingexpert_stream.c
A mixture's experts are read from the drive with direct I/O, past the operating system's file cache, by four threads with handles of their own, into a cache in memory of a set size that gives up the least used expert first. The rest of the model stays mapped. The logits are the same, bit for bit, as with every weight in memory.
In plain wordsA model's experts need not all fit in memory. OLMoE-1B-7B decodes 5.8 tokens a second on a Ryzen 7 3700X through an 8 GiB cache and 2.2 through 4 GiB, read from a Gen3 NVMe drive.
- 02BuiltSpeculative MoE Draftingapi.c · model_cuda.cu
Up to k tokens are drafted from the place the text's last two or three tokens occur earlier in it, then checked together, on a GPU in one pass of up to 63 for a mixture of experts. Each token is taken from logits identical, byte for byte, to a token at a time's, greedy or sampled with a seed.
In plain wordsDrafting changes how fast an answer arrives and never what it is. How much faster has not been published yet.
- 03BuiltAsync Overlapped Pre-fetchingexpert_stream.c · model.c
Before a layer's experts run, the next layer's router scores the residual stream as it stands and the experts it would choose start being read. The guess named 80.4% of the experts the layers went on to use. On a GPU, the half of the guess the router ranks highest is copied in behind.
In plain wordsThe next layer's experts are usually on their way before it asks for them. A wrong guess costs a read and decides nothing.
- 04DesignedMulti-Drive NVMe Stripingruntime/stripe.mjs
The command line splits a file round-robin, in 4,096-byte chunks, across several directories, and its layout is tested. The engine does not read a striped file yet, and read throughput across drives has not been measured.
In plain wordsDesigned so a second SSD adds read speed. Not wired into the engine, and not measured.
- 05BuiltQuantized Weightsquant.c · model_cuda.cu · runtime/quantize.mjs
Models run quantized from GGUF files across Q2_K, Q3_K, Q4_K_M, Q5_K, Q6_K, and Q8_0 on both CPU and CUDA: each block is widened to float32 as it is used, to the bits gguf-py gives, keeping experts in their stored format so more of them fit. The command line's INT4 and INT8 quantizer is tested and used by no model yet.
In plain wordsThe installer's Qwen3-30B-A3B is Qwen's own Q4_K_M file, run as stored, and main also supports sub-4-bit Q2_K, Q3_K, and Q5_K GGUF block formats on CPU and CUDA.
- 06BuiltMulti-Format Ingestionsafetensors.c · gguf.c · runtime/ingest.mjs
The engine reads SafeTensors in place, one file or shards through their index, and single-file or multi-shard split GGUF v2 and v3 checkpoints across CPU, expert streaming, and CUDA, with tensor metadata checked against config.json. It refuses a config asking for anything it does not implement. The command line can also rewrite a checkpoint with every tensor at a 4,096-byte offset.
In plain wordsPoint it at a SafeTensors download from Hugging Face, or a single or multi-shard split GGUF checkpoint with F32, F16, BF16, Q8_0, Q6_K, Q5_K, Q4_K, Q3_K, or Q2_K tensors.
Show all 18 source modules (12 more)Hide extra 12 source modules
- 07BuiltPrefix KV Reuseapi.c · model.c
Each sequence of the server keeps the key-value cache of the last request it served, and a request goes to the free sequence holding the longest prefix of its prompt, recomputing at least its last token. The prefix tree the design shares across requests, radix_kv.c, holds no tensors yet.
In plain wordsA conversation sent back with one more turn does not pay for its history again. Replies match a server with reuse off exactly on a CPU, and on a GPU to within float32 rounding of the logits.
- 08BuiltGPU Expert Cachemodel_cuda.cu
On an NVIDIA GPU everything but the experts goes to the card. It keeps as many experts as its memory holds, the least used given up first, and copies the rest in from page-locked memory as a layer wants them. A prompt's tokens run together as rows, each expert once a layer for every row routed to it.
In plain wordsOLMoE-1B-7B decodes 61 tokens a second on an RTX 3070 with 8 GB, ten times its CPU speed, and reads a 2,000-token prompt in 3.0 seconds.
- 09BuiltCPU Beside the GPUwarp.c · exact.c · pool.c
With --cpu-experts, the experts a token wants that the GPU lacks are shared between copies to the card and CPU threads running them from memory at the same time, by what each has cost lately. The CPU computes an expert in the GPU kernel's own order, so the logits are the GPU alone's, byte for byte, whichever experts ran where.
In plain wordsWhere copying an expert is slow, the CPU takes some of the work. Where it is cheap, as from Q4_K_M on the RTX 3070, the CPU is given nothing, and nothing changes.
- 10BuiltLong Contextmodel_cuda.cu
A key-value cache too large for the GPU keeps as many layers on the card as fit and the rest in page-locked memory on the host, copied in for each layer's attention; past 1,024 positions a prompt's tokens share the cache's tiles. --precision fast, opt-in, rounds a prompt's products to float16 on the tensor cores, and decoding stays exact.
In plain wordsA prompt of 32,768 tokens runs on a GPU of 8 GB with its logits unchanged, bit for bit. It reads that prompt slower than llama.cpp unless you opt into the fast precision.
- 11BuiltThree APIs and Tool Callsapi.c · convert.c · chat.c
OpenAI's chat completions and Responses and Anthropic's Messages, streamed or not, are converted into one conversation and written in the model's chat template, Qwen3's byte for byte as transformers renders it. The reply is read back as it streams: its reasoning, its text and each tool call, in the shape of the API that asked.
In plain wordsClaude Code and Codex work against it as they are. SmolLM2, whose template has no tools, refuses a request that sends them.
- 12BuiltInstaller and Chat Pageinstall.c · catalog.c · chat_page.c
The engine describes the machine (hardware), picks the first model of its built-in catalogue whose needs the machine meets (plan), and checks every fetched file against its pinned size and SHA-256 (verify). The install scripts fetch, resume and start the server, whose chat page is built into the engine and served at /.
In plain wordsOne command gets a checked model that fits, and a page in the browser that talks to it and loads nothing from anywhere else.
- 13BuiltText Embeddingsencoder.c · wordpiece.c
A BERT-architecture encoder in float32 on the CPU, several texts as the rows of one matrix product, pooled and normalized as the model's sentence-transformers files say, and served at /v1/embeddings. Its tokenizer matches the model's own on every code point.
In plain wordsbge-small-en-v1.5 gives 384 values a text, within 1e-5 of its pinned reference, the same bits on any number of threads.
- 14BuiltRequests at Onceapi.c · server.c
With --parallel n, up to n requests run together, each in a sequence of the key-value cache of its own, and each round runs every request's next token, on a GPU in passes that read each weight once. A new prompt runs in pieces between the others' rounds.
In plain wordsSeveral clients can share one server, and running together changes no reply: every token's logits are the ones that request alone would get.
- 15BuiltKernel Pushdown Grammarjsonpfx.c · runtime/grammar.mjs
In JSON mode, before each token is chosen, the logit of every token that could not continue a JSON object is set to minus infinity. The command line compiles a JSON Schema to a pushdown automaton, but the engine refuses a schema with a 400 rather than ignore it.
In plain wordsAsk for a JSON object and the reply is one, or a valid start of one if it is cut short. Schemas are not enforced yet.
- 16BuiltCheckpoint Attestation & Air-Gap Proofapi.c · crypto.c
SHA-256 is tested against the FIPS 180-2 vectors and computes SHA-256 checkpoint attestation exposed on /health and API response headers in main, so callers and Vitna Code receipts can verify which exact weights served a turn. Per-token trajectory digests and socket egress byte counters are still unbuilt.
In plain wordsEvery /health check and API response on main carries the loaded checkpoint's SHA-256 digest, while the server listens on 127.0.0.1 and makes no outbound connection. Rolling trajectory digests remain designed.
- 17BuiltAMD HIP & Multi-GPU Plannergpu_plan.c · model_cuda.cu
Main includes an AMD HIP portability layer (gpu_compat.h) and a multi-GPU layer-split planner (gpu_plan.c) as the Gate A13 foundation, alongside single-card NVIDIA CUDA execution.
In plain wordsSingle-card NVIDIA CUDA is measured on an RTX 3070; AMD HIP portability and multi-GPU layer planning are in source on main with hardware benchmarks still pending.
- 18DesignedCloud Overflow Laneresolve.mjs · policy-store.mjs
A turn that cannot fit locally would leave as a lane on vitna-runtime, sent with the operator's own provider key to the model a signed policy names for that lane. The runtime consults no price while the turn waits, and computes no saving.
In plain wordsThe turns that do go to the cloud go to the model your signed policy names, on your own provider account.
Every step it has measured, oldest first.
Each row is one change to the engine and the figure it moved, copied from the engine's own benchmark history, which its tests hold to its README: the machine it ran on, how many runs a figure summarizes, and where the runs were kept, their range. Nothing here is a target.
- Gate A5
OLMoE-1B-7B decodes on the CPU with its experts read from the drive, through a cache of 4 GiB, rather than held in memory.
OLMoE-1B-7B, BF16, on the CPU · AMD Ryzen 7 3700X and an RTX 30702.22tokens a secondthe median of 63 steps in the median of 3 runsAgainst, the whole model in the file cache: 6.11 tokens a secondEvery figure, and how each was taken
- 450.2 ms a tokenthe median of 63 steps in the median of 3 runs
- 6.11 tokens a second, the whole model in the file cachethe median of 63 steps in the median of 3 runs
- Gate A5
OLMoE-1B-7B on the GPU: everything but the experts on the device, and the experts copied in as the layers want them.
OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 307061.3tokens a secondthe median of 63 steps in the median of 5 runsBefore, the same model on the CPU: 163.7 ms a tokenten times as fastEvery figure, and how each was taken
- 16.3 ms a tokenthe median of 63 steps in the median of 5 runs
- 163.7 ms a token, the same model on the CPUthe median of 63 steps in the median of 3 runs
- Prompt rows
A prompt's tokens run together as rows on the GPU, each expert once a layer for every row routed to it.
OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 30702,994ms for a prompt of 2,000 tokensthe median of 3 runsBefore, a token at a time: 53,071 ms for a prompt of 2,000 tokens17.7x as fastEvery figure, and how each was taken
- 53,071 ms for a prompt of 2,000 tokens, a token at a timethe median of 3 runs
- Gate A7
OLMoE-1B-7B from its own Q4_K_M file, every expert then fitting on the GPU.
OLMoE-1B-7B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070213.6tokens a secondthe median of 63 steps in the median of 5 runsBefore, BF16: 60.3 tokens a secondEvery figure, and how each was taken
- 4.7 ms a tokenthe median of 63 steps in the median of 5 runs
- 60.3 tokens a second, BF16the median of 63 steps in the median of 5 runs
- Gate A8
The CPU widens quantized blocks eight weights at a time with AVX2, to the same bits.
OLMoE-1B-7B, Q4_K_M, on the CPU · AMD Ryzen 7 3700X and an RTX 3070326.9ms a tokenthe median of 63 steps in the median of 3 runsBefore, a scalar loop: 1,140.1 ms a token3.5 times as fastEvery figure, and how each was taken
- 1,140.1 ms a token, a scalar loopthe median of 63 steps in the median of 3 runs
- Gate A8
The CPU runs the experts the GPU lacks, at the same time as the GPU runs the others, with the same logits byte for byte.
OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 307033.5ms a token2,040 MiB of experts on the GPU, 4 CPU threadsthe median of 63 steps in the median of 5 runsBefore, the GPU alone: 37.7 ms a tokenEvery figure, and how each was taken
- 37.7 ms a token, the GPU alonethe median of 63 steps in the median of 5 runs
- Gate A9
Qwen3-30B-A3B, 30B parameters, on a GPU of 8 GB that holds under a third of its experts.
Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 307076.7tokens a secondafter "The capital of France is"the engine's timing line, the median of 3 runsBeside, llama.cpp b11433 at its best -ncmoe: 29.2 tokens a secondEvery figure, and how each was taken
- 47.2 tokens a second, after the Fibonacci codethe engine's timing line, the median of 3 runs
- 81.7 tokens a second, after the lighthouse storythe engine's timing line, the median of 3 runs
- 29.2 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 3 runs
- Gate A9
Qwen3-30B-A3B reads a prompt of 512 tokens, every chunk copying in the experts it wants.
Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070345tokens a secondthe median of 3 runsBeside, llama.cpp b11433 at its best -ncmoe: 505 tokens a secondEvery figure, and how each was taken
- 505 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 3 runs
- Gate A10
A long context's attention in tiles a prompt's tokens share, with the key-value cache the GPU has no room for in host memory.
Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070353tokens a seconda prompt of 4,096 tokensthe median of 3 runsBefore, every token in the attention the engine had before: 226 tokens a secondEvery figure, and how each was taken
- 182 tokens a second, a prompt of 32,768 tokensone run
- 226 tokens a second, every token in the attention the engine had beforethe median of 3 runs
- 573 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 2 runs
- Fast precision
--precision fast, opt-in: a prompt's products on the GPU's tensor cores in float16, decoding still exact.
Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 30701,014tokens a seconda prompt of 4,096 tokensthe median of 3 runsBefore, --precision exact: 349 tokens a secondabout three times as fastEvery figure, and how each was taken
- 521 tokens a second, a prompt of 32,768 tokensone run
- 1.03e-3 KL divergence from the reference at the last position, the mean of 6 promptsthe mean of 6 prompts
- 234 of 239 top tokens agreeing with the referenceevery position of 6 prompts
- 349 tokens a second, --precision exactthe median of 3 runs
- 573 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 2 runs
- 1.4e-3 KL divergence, Q8_0 against BF16, for scalethe mean of 6 prompts
- Gate A14
bge-small-en-v1.5's embeddings on the CPU, several texts as the rows of one matrix product.
bge-small-en-v1.5, F32, on the CPU · an OVHcloud b3-16 instance3,673tokens a second on 4 threadsone runBefore, one thread: 1,086 tokens a secondEvery figure, and how each was taken
- 1,086 tokens a second, one threadone run
- The installer
A machine of 16 GB gets Qwen3-30B-A3B, served with its experts read from the drive into 4 GiB of memory rather than the whole file held there.
Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 30705.1GiB of memorythe process's peak working setthe median of 3 runs, all 5.1Against, the whole file held in memory: 17.5 GiB of memoryEvery figure, and how each was taken
- 59.3 ms a token after a prompt of 5 tokensthe median of 3 runs, 59.0 to 59.3
- 54.1 s for a prompt of 8,000 tokensthe median of 3 runs, 53.2 to 54.3
- 111.7 ms a token after itthe median of 3 runs, 111.5 to 112.2
- 17.5 GiB of memory, the whole file held in memorythe median of 3 runs, all 17.5
- 22.3 ms a token after a prompt of 5 tokens, the whole file held in memorythe median of 3 runs, 22.1 to 23.4
- 26.7 s for a prompt of 8,000 tokens, the whole file held in memorythe median of 3 runs, 26.5 to 26.8
- 33.9 ms a token after it, the whole file held in memorythe median of 3 runs, 33.6 to 34.0
- Beside AirLLM
OLMoE-1B-7B in BF16 beside AirLLM 4.0.0, which loads every layer from the drive for every token.
OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 307017.3ms a tokenthe median of 3 runs, 17.0 to 17.6Beside, AirLLM 4.0.0: 11,249 ms a tokenEvery figure, and how each was taken
- 61.8 ms a token with 256 MiB of experts on the GPUone run
- 11,249 ms a token, AirLLM 4.0.0the mean of 31 steps, 9,670 to 13,355
- Beside AirLLM
Qwen3-30B-A3B in BF16 beside AirLLM 4.0.0: 61 GB, more than the machine's memory, read off the drive.
Qwen3-30B-A3B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070141ms a token over 128 tokensthe experts read from the drive through 16 GiBone runBeside, AirLLM 4.0.0: 73,399 ms a tokenEvery figure, and how each was taken
- 269 ms a token over 16 tokensone run
- 73,399 ms a token, AirLLM 4.0.0the mean of 15 steps, 68,098 to 79,930
- Beside AirLLM
OLMoE-1B-7B in 4 bits beside AirLLM 4.0.0's own 4-bit compression.
OLMoE-1B-7B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 30705.1ms a tokenone runBeside, AirLLM 4.0.0, compression='4bit': 8,161 ms a tokenEvery figure, and how each was taken
- 16.6 ms a token with 256 MiB of experts on the GPUone run
- 8,161 ms a token, AirLLM 4.0.0, compression='4bit'the mean of 7 steps, 7,653 to 8,782
Where it stops.
- One machine, by hand
- Every speed on this page was measured on one machine, a Ryzen 7 3700X with an RTX 3070 of 8 GB, but the embeddings', on an OVHcloud b3-16 instance. CI compiles the GPU path and has no GPU to run it on.
- Prompts read slower than llama.cpp
- On the RTX 3070, Qwen3-30B-A3B read a prompt of 512 tokens at 345 tokens a second where llama.cpp's best was 505, and a prompt of 4,096 tokens at 353 against 573. Only the opt-in fast precision reads a prompt faster than llama.cpp.
- Experts off the drive are slower
- Read off the drive into 4 GiB, Qwen3-30B-A3B took 59.3 ms a token after a prompt of 5 tokens on the RTX 3070, against 22.3 with the whole file in memory. The 16 GB plan was checked on a larger machine described as 15.9 GiB, never on one with only 16 GB, and it does not measure the drive.
- Context
- Started by hand, the server holds the model's own maximum or 4,096 tokens, whichever is less, unless --ctx says more; the installer passes its catalogue's. A request that would not fit is refused, never cut short.
- No authentication
- The server listens on 127.0.0.1 unless --host says otherwise, and warns when it does: anyone who can reach the port can use it.
- Refused, not ignored
- A JSON Schema, log probabilities, penalties, logit bias and more than one choice are refused with a 400 that names them, as are images and documents in Anthropic's Messages.
- No proof of the air gap
- Nothing leaves during generation because the server opens no outbound connection, which its source shows. The proof the design adds, carried on every completion, is not built.
- Not started
- AMD GPUs and a model split across two GPUs (gate A13), and any GPU on macOS.
Vitna charges nothing. The machine is yours.
What running Anchor costs, line by line, with who charges for each. No line of it is a Vitna price, and none is a provider's: nothing about it calls one.
- Vitna
- Nothing
- Anchor is Apache-2.0: no licence fee, no account, no key, and nothing metered or billed.
- The model's download
- Your connection
- 18.6 GB for Qwen3-30B-A3B at Q4_K_M or 726 MB for SmolLM2-360M-Instruct, weights, tokenizer and config together, fetched once from Hugging Face at pinned revisions. Run again, the installer fetches only what is missing.
- Hardware and power
- You
- The machine it runs on and the electricity it draws. Vitna sells no hardware, and nothing about Anchor is hosted: the server is a process on your machine.
- Model providers
- None
- Anchor calls no provider. The overflow lane that would send a turn to one, on your own key and account, is designed and not built.
Asked before running it.
Everything about accounts, telemetry, supported GPUs and failures, answered plainly.
I am not a systems engineer. How hard is this to set up?
On Windows or Linux it is one command: release v0.2.1's installer picks a model that fits the machine, fetches it with checks, starts the server and opens a chat page that talks to it. On macOS on Apple silicon there is no installer yet: the release's binary runs on the CPU, with a model fetched as the engine's README shows, or build from source with Make or CMake. Building the GPU path yourself needs NVIDIA's CUDA toolkit; the release's Windows and Linux binaries carry it already.
Does it need a dedicated GPU?
No. Every model the engine runs runs on a CPU with AVX2 or NEON, and CI checks the small ones against their references on Linux, macOS and Windows. The installer gives Qwen3-30B-A3B only to a machine with an NVIDIA GPU of 7.5 GiB and 14 GiB of memory, and SmolLM2-360M-Instruct, on the CPU, to any other. A GPU makes a mixture of experts about ten times faster: OLMoE-1B-7B decoded 61 tokens a second on an RTX 3070 against 6.1 on a Ryzen 7 3700X, both with the model in memory. CUDA was checked by hand on that one card; an AMD HIP portability layer and multi-GPU layer-split planner are on main with hardware benchmarks still pending.
Can I point Claude Code or Codex at it?
Yes, with Qwen3-30B-A3B. Claude Code speaks Anthropic's Messages, at http://127.0.0.1:8765, and Codex OpenAI's Responses, at http://127.0.0.1:8765/v1; tools work in both and in chat completions. Claude Code 2.1.283 read and edited a file and Codex 0.160.0 called its tools through it, checked by hand. Their prompts are long, so serve a context of 32K. What the engine lacks is refused with a 400 naming it rather than ignored: a JSON Schema, log probabilities, penalties, logit bias, more than one choice, and tools for SmolLM2, whose template has none.
Which checkpoint formats does it read?
SafeTensors, one file or shards through their index, read in place, and GGUF v2 and v3 with F32, F16, BF16, Q8_0, Q4_K and Q6_K tensors, which is how the installer's Qwen3-30B-A3B arrives. It runs Llama, OLMoE and Qwen3's mixture of experts, and refuses on load a config asking for anything it does not implement, saying what.
Do I need an account or a key?
No. There is nothing to sign up for, and the server has no authentication: it listens on 127.0.0.1 unless --host says otherwise, and warns when it does, since anyone who can reach the port can then use it. A client that insists on a key can send any placeholder; nothing checks it.
What leaves my machine?
Installing fetches the engine from GitHub and the model from Hugging Face, at pinned revisions, and checks each file. After that, generating opens no outbound connection: the server answers on 127.0.0.1, and its chat page loads nothing from anywhere and may talk to its own server alone. Nothing is sent to Vitna. That is true of the code, which its source shows, rather than proved per turn: the count of socket egress bytes and the digest over each completion the design adds are not built.
What happens when something fails?
A request whose step fails ends with an error, never with the reply so far: a 500 of type server_error, or, in a stream, an error event in place of the final chunk with no [DONE] after it. If the GPU can run nothing more, waiting requests get a 503 with the code device_lost and the server exits with status 75, so whatever started it can start it again. A request the context cannot hold is refused, not cut short. The installer resumes a part fetched before, checks every file against its pinned SHA-256, and fetches a file that fails anew, once.
How is this different from Ollama or LM Studio?
They run a large catalogue of models today, and Anchor is checked on five, two of them in its installer, so it replaces neither yet. What it is built around is a mixture of experts whose experts live on the drive and in a cache you size, with the busiest on the GPU, and logits that stay the same, bit for bit, however the experts are cached or read. Measured, a model read off the drive is slower than one held in memory: 2.2 tokens a second against 6.1 on a Ryzen 7 3700X, and 8.7 against 61 on an RTX 3070.
Will streaming wear out my SSD or slow down my other apps?
No. The engine only reads the checkpoint, with direct I/O, and reads do not wear a drive the way writes do. Because they bypass the operating system's file cache, the memory they take is the cache you give the engine and no more.
Does speculative drafting change the model's answers?
No, not by a bit. Every drafted token is checked against logits identical, byte for byte, to the ones decoding a token at a time gives, greedy or sampled with a seed, and the engine's tests compare the two. Drafting changes how fast the answer arrives, never what it is.
Where do I get help?
Through the contact form, Talk to us. The engine's README on GitHub documents every command, flag and route, and how each gate was checked.
Run models on your own infrastructure. Anchor is one of 3 ways.
A model that fits the machine, on the machine.
Release v0.2.1 installs with one command on Windows or Linux, picks the model the machine can run, and serves it on 127.0.0.1 with a chat page beside it. Nothing to sign up for, and nothing billed.
