Get started

Open models, served from your own machine.

One command installs Anchor with a model that fits your machine and starts a local OpenAI and Anthropic server on 127.0.0.1. With an 8 GB NVIDIA GPU and 16 GB of memory, it serves Qwen3-30B-A3B by streaming experts from memory or drive.

A field of weight tiles on a drive runs into the distance. A few of them, the experts the router picks, glow amber and lift off it toward memory.

Amber is the experts picked off the drive.

76.7
tokens a second after "The capital of France is", Qwen3-30B-A3B decoding on an RTX 3070 with 8 GB, where llama.cpp's best on the same file was 29.2
5.1 GiB
of memory to serve that model with its experts read off the drive, on a Ryzen 7 3700X and that RTX 3070, against 17.5 holding the whole file
17.3 ms
a token of OLMoE-1B-7B on the RTX 3070, where AirLLM 4.0.0 took 11,249 ms on the same checkpoint
01 / 11AvailabilityEarly release

A download, with nothing to sign up for.

Release v0.2.1 is the latest on GitHub. It needs no account and no key, and the installer chooses the first model in its catalogue whose needs the machine meets. Every figure that names a machine was measured; the rest describes the design.

Release
v0.2.1 on GitHub (not on npm): installers for Windows and Linux on x86-64, plus a binary for macOS on Apple silicon.
Licence
Apache-2.0, with the full C11 and CUDA source public and built with Make or CMake.
Access
A direct download: no account, no API key, no telemetry, and nothing billed.
Processors
x86-64 with AVX2, or ARM with NEON; every model the engine supports runs on a CPU.
GPUs
NVIDIA CUDA (Turing to Blackwell), checked by hand on an RTX 3070. macOS runs on CPU; AMD HIP portability and multi-GPU layer planning are on main (Gate A13 foundation), with AMD/multi-GPU benchmarks pending.
Models
The two installer catalogue models below, plus SmolLM2-135M, OLMoE-1B-7B, and bge-small-en-v1.5 embeddings.
The installer's catalogue, in order of preferenceRead off install/catalog.json at e72ee25
ModelThe machine needsRuns onWeightsContextExperts
Qwen3-30B-A3B at Q4_K_Mqwen3-30b-a3ban NVIDIA GPU of 7.5 GiB and 14 GiB of memorythe NVIDIA GPUQwen3-30B-A3B-Q4_K_M.gguf18.6 GB, from Qwen/Qwen3-30B-A3B-GGUF32,768 tokens with 46 GiB of memory or more, else 16,384 tokensthe whole file held in memory with 30 GiB of memory or more, else read off the drive into 4,096 MiB
SmolLM2-360M-Instructsmollm2-360m-instruct3 GiB of memorythe CPUmodel.safetensors724 MB, from HuggingFaceTB/SmolLM2-360M-Instruct4,096 tokensnone: a dense model, held in memory
A 30B mixture of experts, in Qwen's own Q4_K_M quantization, for chat and for coding agents, with its reasoning and tool calls in each API's shape. A small model for trying the server, not a coding assistant. It runs on the CPU, on one thread. A machine that meets neither is told so, and why. VITNA_MODEL names a model of the catalogue in place of the installer's choice.Read off install/catalog.json at release v0.2.1; sizes from each file's pin.
02 / 11Try itRecorded runs and browser REPL

A recorded install, then a REPL in your browser.

The two runs are what the installer did on fresh machines and on one GPU, as the engine's README records them. The REPL answers in this tab, and asks your machine nothing until you type /connect.

recorded · a fresh Windows machine, no GPU
Typed, in Windows PowerShellirm https://github.com/convexityos/vitna-anchor/releases/download/v0.2.0/install.ps1 | iex
The planSmolLM2-360M-Instruct, on the CPU, and why: no NVIDIA GPU was found
The filesresumed a part fetched before it ran, then checked each against its pinned SHA-256
Asked "What is the capital of France?", the server answeredThe capital of France is Paris.
recorded · by hand, an RTX 3070 with 8 GB
The planQwen3-30B-A3B at Q4_K_M with a 32K context, on the GPU
The files18.6 GB from Hugging Face at the pinned revisions, every file checked
The chat pagenamed the model and its context from the server, and streamed a reply with its reasoning folded away

Recorded on 2026-10-06 against release v0.2.0, in CI on GitHub's hosted runners and by hand: the engine's README, How gate A12 was checked.

Illustrative: no model runs in this tab, and the panel is the design's, not a reading of your machine. /models, /install and /endpoints print the release's own facts. /connect asks http://127.0.0.1:8765/v1/health once; the engine sends no CORS headers, so a page on another origin cannot read its replies even while it runs, and its own chat page at http://127.0.0.1:8765/ is where to talk to it.

03 / 11What comes backOpenAI and Anthropic shapes

A reply in the shape your client already reads.

A chat completion from Qwen3-30B-A3B that calls a tool, beside the request that asked for it. The fields are the engine's, from its README; the words, ids and counts are written for this page.

the request
curl http://127.0.0.1:8765/v1/chat/completions \
  -H "content-type: application/json" -d '{
  "model": "qwen3-30b-a3b",
  "messages": [{ "role": "user",
    "content": "What does notes.md say about the release?" }],
  "tools": [{ "type": "function", "function": {
    "name": "read_file",
    "parameters": { "type": "object",
      "properties": { "path": { "type": "string" } } } } }]
}'
the reply, illustrative
HTTP/1.1 200 OK
x-vitna-seed: 4185073029631457214

{
  "id": "chatcmpl-6f1c2a9e04b7d358",
  "object": "chat.completion",
  "model": "qwen3-30b-a3b",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "reasoning_content": "They want notes.md read first.",
      "tool_calls": [{
        "id": "call_9a3e51c07d2b4f86",
        "type": "function",
        "function": {
          "name": "read_file",
          "arguments": "{\"path\": \"notes.md\"}"
        }
      }]
    },
    "logprobs": null,
    "finish_reason": "tool_calls"
  }],
  "usage": {
    "prompt_tokens": 172, "completion_tokens": 38,
    "total_tokens": 210,
    "prompt_tokens_details": { "cached_tokens": 0 }
  }
}
reasoning_content
Qwen3's reasoning, the text it wrote between its think tags, kept apart from the answer. Thinking is on unless a request asks it off, through chat_template_kwargs.enable_thinking or reasoning_effort.
tool_calls, finish_reason
Each call the model wrote, for your code to run; the reply ends tool_calls, and you send the result back as a tool message. Anthropic's Messages returns the same reply as a thinking block and a tool_use, and the Responses API as a reasoning item and a function_call.
usage.prompt_tokens_details.cached_tokens
The prompt tokens whose keys and values were reused from the request before rather than computed again: none on a first turn, most of the prompt on the next. prompt_tokens counts after the chat template; Anthropic counts the reuse as cache_read_input_tokens.
x-vitna-seed
The seed the reply was sampled with, so a sampled reply can be reproduced by sending it back as seed.
x-vitna-ignored
Sent only when a request carried a field the server does not know, and names it rather than dropping it unseen. A field that would change the output and is not implemented is refused with a 400 instead.
an embedding request
curl http://127.0.0.1:8765/v1/embeddings \
  -H "content-type: application/json" -d '{
  "model": "bge-small-en-v1.5",
  "input": ["Represent this sentence for searching
    relevant passages: how do tides work"]
}'
its reply, illustrative
{
  "object": "list",
  "data": [{ "object": "embedding", "index": 0,
             "embedding": [ /* 384 values */ ] }],
  "model": "bge-small-en-v1.5",
  "usage": { "prompt_tokens": 18, "total_tokens": 18 }
}

Illustrative: the shapes are the engine's, an embedding 384 values normalized to length one (README, Serve the model and Serve an embedding model); no model wrote these replies. A model that generates answers /v1/embeddings with a 404 that says so, and an embedding model answers generation the same way.

04 / 11InstallWindows, Linux and macOS

One command, on Windows or Linux.

Release v0.2.1's installer fetches the engine and checks it against the SHA-256 the release wrote into it, asks the engine which model fits the machine, fetches that model's files at their pinned revisions, resuming any part already there, checks every file and starts the server with its chat page. Which model is the catalogue's call, above: Qwen3-30B-A3B with an NVIDIA GPU, the whole file in memory or its experts read off the drive, and SmolLM2-360M-Instruct on any other machine.

The installer, in the engine's README
# Windows, in PowerShell
irm https://github.com/convexityos/vitna-anchor/releases/latest/download/install.ps1 | iex

# Linux on x86-64
curl -fsSL https://github.com/convexityos/vitna-anchor/releases/latest/download/install.sh | sh

# macOS on Apple silicon: no installer. The release's binary runs on the CPU;
# fetch a model with the engine's scripts/fetch-model.mjs, then serve it:
./vitna-anchor-macos-arm64 serve --model models/smollm2-360m-instruct
after it runs
It prints the base URLs and the model's nameOpenAI clients: http://127.0.0.1:8765/v1 Anthropic clients: http://127.0.0.1:8765 model: qwen3-30b-a3b
And opens the chat pagehttp://127.0.0.1:8765/
Later, beside what it installedserve.cmd on Windows, serve.sh on Linux, starts the server again
Where it installsVITNA_HOME, by default %LOCALAPPDATA%\vitna-anchor or ~/.vitna-anchor
What else it readsVITNA_MODEL, a model of the catalogue; VITNA_PORT, 8765 by default; VITNA_NO_START=1, stop once all is checked
Run againit fetches only what is missing
client.py
# Any OpenAI client. Only the base URL changes.
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8765/v1",
    api_key="placeholder",
)

res = client.chat.completions.create(
    model="qwen3-30b-a3b",
    messages=[{"role": "user", "content": "Hello"}],
)

The model is the name the installer gave it, qwen3-30b-a3b or smollm2-360m-instruct, which GET /v1/models lists; any other name answers 404. The key is a placeholder: nothing checks it, which is why the server listens only on 127.0.0.1 unless told otherwise.

claude code
# Anthropic's base URL has no /v1.
ANTHROPIC_BASE_URL=http://127.0.0.1:8765 \
ANTHROPIC_API_KEY=placeholder \
ANTHROPIC_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_SONNET_MODEL=qwen3-30b-a3b \
ANTHROPIC_DEFAULT_OPUS_MODEL=qwen3-30b-a3b claude

As the engine's README sets Claude Code up, every tier it asks for named as the one model. Its prompts are long, so give the server a context of 32K; each turn after the first reuses nearly all of the one before.

Package
vitna-anchorrelease binaries; not on npm
Server
127.0.0.1:8765OpenAI and Anthropic APIs
Dependencies
0libc; CUDA linked in
Checkpoints
SafeTensors, GGUFQ2_K to Q8_0, multi-shard GGUF
Engine
C11 coreCPU, CUDA; AMD HIP layer in main
Licence
Apache-2.0source on GitHub
05 / 11Where it fitsThree checked jobs

Three jobs it has been checked doing.

Each is one the engine's README records as checked, by hand or in CI, with the route it rides and where a client points.

  1. 01

    A coding agent that stays on your machine

    Claude Code speaks Anthropic's Messages and Codex OpenAI's Responses, and both work against the server with only a base URL and a model name set. Against Qwen3-30B-A3B on an RTX 3070, Claude Code 2.1.283 read and edited a file and Codex 0.160.0 called its tools, checked by hand for gate A11. Their prompts are long, about 14,000 tokens for Claude Code's, so the README serves them a context of 32K.

    POST /v1/messagesPOST /v1/responses
    base URLs http://127.0.0.1:8765 and http://127.0.0.1:8765/v1
  2. 02

    A private chat in the browser

    The chat page is built into the engine: it streams a reply with Qwen3's reasoning folded away above the answer, renders the model's Markdown from escaped text, stops on Esc, and loads nothing from anywhere, its content security policy letting it talk to its own server alone. The installer opens it once the server is up (gate A12).

    GET /POST /v1/chat/completions
    http://127.0.0.1:8765/ in any browser
  3. 03

    Embeddings for search over your own documents

    bge-small-en-v1.5 runs on the CPU and serves OpenAI's embeddings request and response, 384 values a text, within 1e-5 of its pinned reference (gate A14). It is not in the installer's catalogue: fetch it with the engine's scripts/fetch-model.mjs and serve it with --model, as its README's "Serve an embedding model" shows.

    POST /v1/embeddings
    a server started with the embedding model
06 / 11How it worksMemory ladder

Four places a weight can live, fastest first.

A model is not one file that has to fit somewhere. It is layers that are always hot, a cache of everything you have already said, and a long tail of experts that each token needs a few of. Anchor puts each on the store that suits it; the bar beside each store is its nominal bus figure.

Weight placement and expert streaming127.0.0.1:8765 · zero outbound socket
Local clientsSDKs & Chat UI
/v1/chat/completions/v1/messages/v1/embeddings
Your machine · 127.0.0.1:8765 · loopback onlyNo outbound connection
C11 servervitna-anchor

Tokenizes prompt, picks active experts each layer, streams SSE replies back.

zero dependencies
01 · Host RAMHot layers & KV cache
AttnKV treeNorms
03 · NVMe driveGGUF / SafeTensors
Cold MoE expert weights
NVIDIA GPU · CUDA VRAMActive token decode
Pinned on deviceAttention + active KV slots
Streamed expert cacheTop-k routed experts only

Fallback: CPU AVX2 / NEON when no qualifying GPU exists

Always-hot layers & KV cache Routed MoE experts read on demand Loopback boundary (no outbound egress)
  1. 01

    Host RAMthe layers every token touches, and each conversation's cache

    Embeddings, attention and the router, the parts every token passes through, stay resident for the life of the process, and on an NVIDIA GPU they live on the card instead. Each sequence of the server keeps the key-value cache of the last request it served, so a conversation sent back with one more turn reuses its history and computes only what is new.

    44.8 GB/sDDR5-5600, one module's peak
  2. 02

    NVMe, stripedexpert slabs across two or four drives, as designed

    Slabs would be interleaved round-robin across physical drives in 4,096-byte sectors, so parallel unbuffered reads land on every drive at once. The command line lays a file out that way today. The engine does not read across drives yet, and nothing has measured what a second drive adds.

    not measuredthe engine reads one drive today
  3. 03

    NVMe, one driveexperts read as the layers want them

    Built and measured. A mixture's experts are read with direct I/O, past the page cache, by four threads into a cache in memory of the size you give it, and the experts the next layer will probably want start reading while this one runs. On a Gen3 drive that read them at 1.7 GB/s, OLMoE-1B-7B decoded 5.8 tokens a second on a Ryzen 7 3700X through an 8 GiB cache, against 6.1 with the whole model in memory, and 2.2 through 4 GiB, where each token waits on its reads.

    7.45 GB/sPCIe Gen4 x4, 12.4 on Gen5
  4. 04

    The wirecloud providers, on overflow only

    Nothing leaves the machine during local generation: the engine's server listens on 127.0.0.1 unless told otherwise and makes no outbound connection. The proof the design adds, a count of socket egress bytes and a rolling SHA-256 over the trajectory on every completion, is not built. When a turn cannot fit locally, it would leave as a lane on vitna-runtime, sent to the model a signed policy names.

    0 bytessocket egress during local generation
Illustrative. A bar is a store's nominal bus figure as a share of the fastest, not a measurement of your machine or of the engine; a rung's text names the machine wherever it gives a measured figure.Relay takes no overflow from this engine: it serves only the model a request names.
07 / 11Inside16 built · 2 designed

Eighteen parts, each named after the file it lives in.

Every module maps to a real file under engine/src/, runtime/, or platform/runtime/ so every claim can be checked against the source. Sixteen of the parts run in the engine today; the two designed and not built say so.

Engine module layout16 built in C11 / CUDA · 2 designed
01 · Serving
Three APIs & Chat Pageapi.c · server.c · chat_page.c
JSON Pushdown Grammarjsonpfx.c · grammar.mjs
Text Embeddingsencoder.c · wordpiece.c
Air-Gap Proof (designed)crypto.c · not wired yet
02 · Compute
GPU Cache & Draftingmodel_cuda.cu · kernels.c
Prefix KV & Parallelapi.c · model.c · --parallel
CPU Beside GPUwarp.c · exact.c · pool.c
Cloud Overflow (designed)resolve.mjs · policy-store
03 · Weights
GGUF, SafeTensors & Quantgguf.c · safetensors.c · quant.c
Direct NVMe Expert Streamexpert_stream.c · model.c
Installer & Catalogueinstall.c · catalog.c
Drive Striping (designed)runtime/stripe.mjs
  1. 01Built
    Direct NVMe DMA Streamingexpert_stream.c

    A mixture's experts are read from the drive with direct I/O, past the operating system's file cache, by four threads with handles of their own, into a cache in memory of a set size that gives up the least used expert first. The rest of the model stays mapped. The logits are the same, bit for bit, as with every weight in memory.

    In plain wordsA model's experts need not all fit in memory. OLMoE-1B-7B decodes 5.8 tokens a second on a Ryzen 7 3700X through an 8 GiB cache and 2.2 through 4 GiB, read from a Gen3 NVMe drive.

  2. 02Built
    Speculative MoE Draftingapi.c · model_cuda.cu

    Up to k tokens are drafted from the place the text's last two or three tokens occur earlier in it, then checked together, on a GPU in one pass of up to 63 for a mixture of experts. Each token is taken from logits identical, byte for byte, to a token at a time's, greedy or sampled with a seed.

    In plain wordsDrafting changes how fast an answer arrives and never what it is. How much faster has not been published yet.

  3. 03Built
    Async Overlapped Pre-fetchingexpert_stream.c · model.c

    Before a layer's experts run, the next layer's router scores the residual stream as it stands and the experts it would choose start being read. The guess named 80.4% of the experts the layers went on to use. On a GPU, the half of the guess the router ranks highest is copied in behind.

    In plain wordsThe next layer's experts are usually on their way before it asks for them. A wrong guess costs a read and decides nothing.

  4. 04Designed
    Multi-Drive NVMe Stripingruntime/stripe.mjs

    The command line splits a file round-robin, in 4,096-byte chunks, across several directories, and its layout is tested. The engine does not read a striped file yet, and read throughput across drives has not been measured.

    In plain wordsDesigned so a second SSD adds read speed. Not wired into the engine, and not measured.

  5. 05Built
    Quantized Weightsquant.c · model_cuda.cu · runtime/quantize.mjs

    Models run quantized from GGUF files across Q2_K, Q3_K, Q4_K_M, Q5_K, Q6_K, and Q8_0 on both CPU and CUDA: each block is widened to float32 as it is used, to the bits gguf-py gives, keeping experts in their stored format so more of them fit. The command line's INT4 and INT8 quantizer is tested and used by no model yet.

    In plain wordsThe installer's Qwen3-30B-A3B is Qwen's own Q4_K_M file, run as stored, and main also supports sub-4-bit Q2_K, Q3_K, and Q5_K GGUF block formats on CPU and CUDA.

  6. 06Built
    Multi-Format Ingestionsafetensors.c · gguf.c · runtime/ingest.mjs

    The engine reads SafeTensors in place, one file or shards through their index, and single-file or multi-shard split GGUF v2 and v3 checkpoints across CPU, expert streaming, and CUDA, with tensor metadata checked against config.json. It refuses a config asking for anything it does not implement. The command line can also rewrite a checkpoint with every tensor at a 4,096-byte offset.

    In plain wordsPoint it at a SafeTensors download from Hugging Face, or a single or multi-shard split GGUF checkpoint with F32, F16, BF16, Q8_0, Q6_K, Q5_K, Q4_K, Q3_K, or Q2_K tensors.

Show all 18 source modules (12 more)Hide extra 12 source modules
  1. 07Built
    Prefix KV Reuseapi.c · model.c

    Each sequence of the server keeps the key-value cache of the last request it served, and a request goes to the free sequence holding the longest prefix of its prompt, recomputing at least its last token. The prefix tree the design shares across requests, radix_kv.c, holds no tensors yet.

    In plain wordsA conversation sent back with one more turn does not pay for its history again. Replies match a server with reuse off exactly on a CPU, and on a GPU to within float32 rounding of the logits.

  2. 08Built
    GPU Expert Cachemodel_cuda.cu

    On an NVIDIA GPU everything but the experts goes to the card. It keeps as many experts as its memory holds, the least used given up first, and copies the rest in from page-locked memory as a layer wants them. A prompt's tokens run together as rows, each expert once a layer for every row routed to it.

    In plain wordsOLMoE-1B-7B decodes 61 tokens a second on an RTX 3070 with 8 GB, ten times its CPU speed, and reads a 2,000-token prompt in 3.0 seconds.

  3. 09Built
    CPU Beside the GPUwarp.c · exact.c · pool.c

    With --cpu-experts, the experts a token wants that the GPU lacks are shared between copies to the card and CPU threads running them from memory at the same time, by what each has cost lately. The CPU computes an expert in the GPU kernel's own order, so the logits are the GPU alone's, byte for byte, whichever experts ran where.

    In plain wordsWhere copying an expert is slow, the CPU takes some of the work. Where it is cheap, as from Q4_K_M on the RTX 3070, the CPU is given nothing, and nothing changes.

  4. 10Built
    Long Contextmodel_cuda.cu

    A key-value cache too large for the GPU keeps as many layers on the card as fit and the rest in page-locked memory on the host, copied in for each layer's attention; past 1,024 positions a prompt's tokens share the cache's tiles. --precision fast, opt-in, rounds a prompt's products to float16 on the tensor cores, and decoding stays exact.

    In plain wordsA prompt of 32,768 tokens runs on a GPU of 8 GB with its logits unchanged, bit for bit. It reads that prompt slower than llama.cpp unless you opt into the fast precision.

  5. 11Built
    Three APIs and Tool Callsapi.c · convert.c · chat.c

    OpenAI's chat completions and Responses and Anthropic's Messages, streamed or not, are converted into one conversation and written in the model's chat template, Qwen3's byte for byte as transformers renders it. The reply is read back as it streams: its reasoning, its text and each tool call, in the shape of the API that asked.

    In plain wordsClaude Code and Codex work against it as they are. SmolLM2, whose template has no tools, refuses a request that sends them.

  6. 12Built
    Installer and Chat Pageinstall.c · catalog.c · chat_page.c

    The engine describes the machine (hardware), picks the first model of its built-in catalogue whose needs the machine meets (plan), and checks every fetched file against its pinned size and SHA-256 (verify). The install scripts fetch, resume and start the server, whose chat page is built into the engine and served at /.

    In plain wordsOne command gets a checked model that fits, and a page in the browser that talks to it and loads nothing from anywhere else.

  7. 13Built
    Text Embeddingsencoder.c · wordpiece.c

    A BERT-architecture encoder in float32 on the CPU, several texts as the rows of one matrix product, pooled and normalized as the model's sentence-transformers files say, and served at /v1/embeddings. Its tokenizer matches the model's own on every code point.

    In plain wordsbge-small-en-v1.5 gives 384 values a text, within 1e-5 of its pinned reference, the same bits on any number of threads.

  8. 14Built
    Requests at Onceapi.c · server.c

    With --parallel n, up to n requests run together, each in a sequence of the key-value cache of its own, and each round runs every request's next token, on a GPU in passes that read each weight once. A new prompt runs in pieces between the others' rounds.

    In plain wordsSeveral clients can share one server, and running together changes no reply: every token's logits are the ones that request alone would get.

  9. 15Built
    Kernel Pushdown Grammarjsonpfx.c · runtime/grammar.mjs

    In JSON mode, before each token is chosen, the logit of every token that could not continue a JSON object is set to minus infinity. The command line compiles a JSON Schema to a pushdown automaton, but the engine refuses a schema with a 400 rather than ignore it.

    In plain wordsAsk for a JSON object and the reply is one, or a valid start of one if it is cut short. Schemas are not enforced yet.

  10. 16Built
    Checkpoint Attestation & Air-Gap Proofapi.c · crypto.c

    SHA-256 is tested against the FIPS 180-2 vectors and computes SHA-256 checkpoint attestation exposed on /health and API response headers in main, so callers and Vitna Code receipts can verify which exact weights served a turn. Per-token trajectory digests and socket egress byte counters are still unbuilt.

    In plain wordsEvery /health check and API response on main carries the loaded checkpoint's SHA-256 digest, while the server listens on 127.0.0.1 and makes no outbound connection. Rolling trajectory digests remain designed.

  11. 17Built
    AMD HIP & Multi-GPU Plannergpu_plan.c · model_cuda.cu

    Main includes an AMD HIP portability layer (gpu_compat.h) and a multi-GPU layer-split planner (gpu_plan.c) as the Gate A13 foundation, alongside single-card NVIDIA CUDA execution.

    In plain wordsSingle-card NVIDIA CUDA is measured on an RTX 3070; AMD HIP portability and multi-GPU layer planning are in source on main with hardware benchmarks still pending.

  12. 18Designed
    Cloud Overflow Laneresolve.mjs · policy-store.mjs

    A turn that cannot fit locally would leave as a lane on vitna-runtime, sent with the operator's own provider key to the model a signed policy names for that lane. The runtime consults no price while the turn waits, and computes no saving.

    In plain wordsThe turns that do go to the cloud go to the model your signed policy names, on your own provider account.

08 / 11Measured2026-10-10

Every step it has measured, oldest first.

Each row is one change to the engine and the figure it moved, copied from the engine's own benchmark history, which its tests hold to its README: the machine it ran on, how many runs a figure summarizes, and where the runs were kept, their range. Nothing here is a target.

  1. Gate A5

    OLMoE-1B-7B decodes on the CPU with its experts read from the drive, through a cache of 4 GiB, rather than held in memory.

    OLMoE-1B-7B, BF16, on the CPU · AMD Ryzen 7 3700X and an RTX 3070
    2.22tokens a secondthe median of 63 steps in the median of 3 runsAgainst, the whole model in the file cache: 6.11 tokens a second
    Every figure, and how each was taken
    • 450.2 ms a tokenthe median of 63 steps in the median of 3 runs
    • 6.11 tokens a second, the whole model in the file cachethe median of 63 steps in the median of 3 runs
  2. Gate A5

    OLMoE-1B-7B on the GPU: everything but the experts on the device, and the experts copied in as the layers want them.

    OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    61.3tokens a secondthe median of 63 steps in the median of 5 runsBefore, the same model on the CPU: 163.7 ms a tokenten times as fast
    Every figure, and how each was taken
    • 16.3 ms a tokenthe median of 63 steps in the median of 5 runs
    • 163.7 ms a token, the same model on the CPUthe median of 63 steps in the median of 3 runs
  3. Prompt rows

    A prompt's tokens run together as rows on the GPU, each expert once a layer for every row routed to it.

    OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    2,994ms for a prompt of 2,000 tokensthe median of 3 runsBefore, a token at a time: 53,071 ms for a prompt of 2,000 tokens17.7x as fast
    Every figure, and how each was taken
    • 53,071 ms for a prompt of 2,000 tokens, a token at a timethe median of 3 runs
  4. Gate A7

    OLMoE-1B-7B from its own Q4_K_M file, every expert then fitting on the GPU.

    OLMoE-1B-7B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    213.6tokens a secondthe median of 63 steps in the median of 5 runsBefore, BF16: 60.3 tokens a second
    Every figure, and how each was taken
    • 4.7 ms a tokenthe median of 63 steps in the median of 5 runs
    • 60.3 tokens a second, BF16the median of 63 steps in the median of 5 runs
  5. Gate A8

    The CPU widens quantized blocks eight weights at a time with AVX2, to the same bits.

    OLMoE-1B-7B, Q4_K_M, on the CPU · AMD Ryzen 7 3700X and an RTX 3070
    326.9ms a tokenthe median of 63 steps in the median of 3 runsBefore, a scalar loop: 1,140.1 ms a token3.5 times as fast
    Every figure, and how each was taken
    • 1,140.1 ms a token, a scalar loopthe median of 63 steps in the median of 3 runs
  6. Gate A8

    The CPU runs the experts the GPU lacks, at the same time as the GPU runs the others, with the same logits byte for byte.

    OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    33.5ms a token2,040 MiB of experts on the GPU, 4 CPU threadsthe median of 63 steps in the median of 5 runsBefore, the GPU alone: 37.7 ms a token
    Every figure, and how each was taken
    • 37.7 ms a token, the GPU alonethe median of 63 steps in the median of 5 runs
  7. Gate A9

    Qwen3-30B-A3B, 30B parameters, on a GPU of 8 GB that holds under a third of its experts.

    Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    76.7tokens a secondafter "The capital of France is"the engine's timing line, the median of 3 runsBeside, llama.cpp b11433 at its best -ncmoe: 29.2 tokens a second
    Every figure, and how each was taken
    • 47.2 tokens a second, after the Fibonacci codethe engine's timing line, the median of 3 runs
    • 81.7 tokens a second, after the lighthouse storythe engine's timing line, the median of 3 runs
    • 29.2 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 3 runs
  8. Gate A9

    Qwen3-30B-A3B reads a prompt of 512 tokens, every chunk copying in the experts it wants.

    Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    345tokens a secondthe median of 3 runsBeside, llama.cpp b11433 at its best -ncmoe: 505 tokens a second
    Every figure, and how each was taken
    • 505 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 3 runs
  9. Gate A10

    A long context's attention in tiles a prompt's tokens share, with the key-value cache the GPU has no room for in host memory.

    Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    353tokens a seconda prompt of 4,096 tokensthe median of 3 runsBefore, every token in the attention the engine had before: 226 tokens a second
    Every figure, and how each was taken
    • 182 tokens a second, a prompt of 32,768 tokensone run
    • 226 tokens a second, every token in the attention the engine had beforethe median of 3 runs
    • 573 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 2 runs
  10. Fast precision

    --precision fast, opt-in: a prompt's products on the GPU's tensor cores in float16, decoding still exact.

    Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    1,014tokens a seconda prompt of 4,096 tokensthe median of 3 runsBefore, --precision exact: 349 tokens a secondabout three times as fast
    Every figure, and how each was taken
    • 521 tokens a second, a prompt of 32,768 tokensone run
    • 1.03e-3 KL divergence from the reference at the last position, the mean of 6 promptsthe mean of 6 prompts
    • 234 of 239 top tokens agreeing with the referenceevery position of 6 prompts
    • 349 tokens a second, --precision exactthe median of 3 runs
    • 573 tokens a second, llama.cpp b11433 at its best -ncmoellama-bench's mean of 2 runs
    • 1.4e-3 KL divergence, Q8_0 against BF16, for scalethe mean of 6 prompts
  11. Gate A14

    bge-small-en-v1.5's embeddings on the CPU, several texts as the rows of one matrix product.

    bge-small-en-v1.5, F32, on the CPU · an OVHcloud b3-16 instance
    3,673tokens a second on 4 threadsone runBefore, one thread: 1,086 tokens a second
    Every figure, and how each was taken
    • 1,086 tokens a second, one threadone run
  12. The installer

    A machine of 16 GB gets Qwen3-30B-A3B, served with its experts read from the drive into 4 GiB of memory rather than the whole file held there.

    Qwen3-30B-A3B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    5.1GiB of memorythe process's peak working setthe median of 3 runs, all 5.1Against, the whole file held in memory: 17.5 GiB of memory
    Every figure, and how each was taken
    • 59.3 ms a token after a prompt of 5 tokensthe median of 3 runs, 59.0 to 59.3
    • 54.1 s for a prompt of 8,000 tokensthe median of 3 runs, 53.2 to 54.3
    • 111.7 ms a token after itthe median of 3 runs, 111.5 to 112.2
    • 17.5 GiB of memory, the whole file held in memorythe median of 3 runs, all 17.5
    • 22.3 ms a token after a prompt of 5 tokens, the whole file held in memorythe median of 3 runs, 22.1 to 23.4
    • 26.7 s for a prompt of 8,000 tokens, the whole file held in memorythe median of 3 runs, 26.5 to 26.8
    • 33.9 ms a token after it, the whole file held in memorythe median of 3 runs, 33.6 to 34.0
  13. Beside AirLLM

    OLMoE-1B-7B in BF16 beside AirLLM 4.0.0, which loads every layer from the drive for every token.

    OLMoE-1B-7B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    17.3ms a tokenthe median of 3 runs, 17.0 to 17.6Beside, AirLLM 4.0.0: 11,249 ms a token
    Every figure, and how each was taken
    • 61.8 ms a token with 256 MiB of experts on the GPUone run
    • 11,249 ms a token, AirLLM 4.0.0the mean of 31 steps, 9,670 to 13,355
  14. Beside AirLLM

    Qwen3-30B-A3B in BF16 beside AirLLM 4.0.0: 61 GB, more than the machine's memory, read off the drive.

    Qwen3-30B-A3B, BF16, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    141ms a token over 128 tokensthe experts read from the drive through 16 GiBone runBeside, AirLLM 4.0.0: 73,399 ms a token
    Every figure, and how each was taken
    • 269 ms a token over 16 tokensone run
    • 73,399 ms a token, AirLLM 4.0.0the mean of 15 steps, 68,098 to 79,930
  15. Beside AirLLM

    OLMoE-1B-7B in 4 bits beside AirLLM 4.0.0's own 4-bit compression.

    OLMoE-1B-7B, Q4_K_M, on the GPU · AMD Ryzen 7 3700X and an RTX 3070
    5.1ms a tokenone runBeside, AirLLM 4.0.0, compression='4bit': 8,161 ms a token
    Every figure, and how each was taken
    • 16.6 ms a token with 256 MiB of experts on the GPUone run
    • 8,161 ms a token, AirLLM 4.0.0, compression='4bit'the mean of 7 steps, 7,653 to 8,782
The periwinkle bar is the row's figure and the grey one what it is set against, on one scale per row: a time or a memory is better shorter, a rate longer, and a bar under 2px is drawn at 2px so it is seen.Read off bench/history.json at 3d9150b.

Where it stops.

One machine, by hand
Every speed on this page was measured on one machine, a Ryzen 7 3700X with an RTX 3070 of 8 GB, but the embeddings', on an OVHcloud b3-16 instance. CI compiles the GPU path and has no GPU to run it on.
Prompts read slower than llama.cpp
On the RTX 3070, Qwen3-30B-A3B read a prompt of 512 tokens at 345 tokens a second where llama.cpp's best was 505, and a prompt of 4,096 tokens at 353 against 573. Only the opt-in fast precision reads a prompt faster than llama.cpp.
Experts off the drive are slower
Read off the drive into 4 GiB, Qwen3-30B-A3B took 59.3 ms a token after a prompt of 5 tokens on the RTX 3070, against 22.3 with the whole file in memory. The 16 GB plan was checked on a larger machine described as 15.9 GiB, never on one with only 16 GB, and it does not measure the drive.
Context
Started by hand, the server holds the model's own maximum or 4,096 tokens, whichever is less, unless --ctx says more; the installer passes its catalogue's. A request that would not fit is refused, never cut short.
No authentication
The server listens on 127.0.0.1 unless --host says otherwise, and warns when it does: anyone who can reach the port can use it.
Refused, not ignored
A JSON Schema, log probabilities, penalties, logit bias and more than one choice are refused with a 400 that names them, as are images and documents in Anthropic's Messages.
No proof of the air gap
Nothing leaves during generation because the server opens no outbound connection, which its source shows. The proof the design adds, carried on every completion, is not built.
Not started
AMD GPUs and a model split across two GPUs (gate A13), and any GPU on macOS.
09 / 11CostZero Vitna charge

Vitna charges nothing. The machine is yours.

What running Anchor costs, line by line, with who charges for each. No line of it is a Vitna price, and none is a provider's: nothing about it calls one.

Vitna
Nothing
Anchor is Apache-2.0: no licence fee, no account, no key, and nothing metered or billed.
The model's download
Your connection
18.6 GB for Qwen3-30B-A3B at Q4_K_M or 726 MB for SmolLM2-360M-Instruct, weights, tokenizer and config together, fetched once from Hugging Face at pinned revisions. Run again, the installer fetches only what is missing.
Hardware and power
You
The machine it runs on and the electricity it draws. Vitna sells no hardware, and nothing about Anchor is hosted: the server is a process on your machine.
Model providers
None
Anchor calls no provider. The overflow lane that would send a turn to one, on your own key and account, is designed and not built.
10 / 11QuestionsBefore running it

Asked before running it.

Everything about accounts, telemetry, supported GPUs and failures, answered plainly.

I am not a systems engineer. How hard is this to set up?

On Windows or Linux it is one command: release v0.2.1's installer picks a model that fits the machine, fetches it with checks, starts the server and opens a chat page that talks to it. On macOS on Apple silicon there is no installer yet: the release's binary runs on the CPU, with a model fetched as the engine's README shows, or build from source with Make or CMake. Building the GPU path yourself needs NVIDIA's CUDA toolkit; the release's Windows and Linux binaries carry it already.

Does it need a dedicated GPU?

No. Every model the engine runs runs on a CPU with AVX2 or NEON, and CI checks the small ones against their references on Linux, macOS and Windows. The installer gives Qwen3-30B-A3B only to a machine with an NVIDIA GPU of 7.5 GiB and 14 GiB of memory, and SmolLM2-360M-Instruct, on the CPU, to any other. A GPU makes a mixture of experts about ten times faster: OLMoE-1B-7B decoded 61 tokens a second on an RTX 3070 against 6.1 on a Ryzen 7 3700X, both with the model in memory. CUDA was checked by hand on that one card; an AMD HIP portability layer and multi-GPU layer-split planner are on main with hardware benchmarks still pending.

Can I point Claude Code or Codex at it?

Yes, with Qwen3-30B-A3B. Claude Code speaks Anthropic's Messages, at http://127.0.0.1:8765, and Codex OpenAI's Responses, at http://127.0.0.1:8765/v1; tools work in both and in chat completions. Claude Code 2.1.283 read and edited a file and Codex 0.160.0 called its tools through it, checked by hand. Their prompts are long, so serve a context of 32K. What the engine lacks is refused with a 400 naming it rather than ignored: a JSON Schema, log probabilities, penalties, logit bias, more than one choice, and tools for SmolLM2, whose template has none.

Which checkpoint formats does it read?

SafeTensors, one file or shards through their index, read in place, and GGUF v2 and v3 with F32, F16, BF16, Q8_0, Q4_K and Q6_K tensors, which is how the installer's Qwen3-30B-A3B arrives. It runs Llama, OLMoE and Qwen3's mixture of experts, and refuses on load a config asking for anything it does not implement, saying what.

Do I need an account or a key?

No. There is nothing to sign up for, and the server has no authentication: it listens on 127.0.0.1 unless --host says otherwise, and warns when it does, since anyone who can reach the port can then use it. A client that insists on a key can send any placeholder; nothing checks it.

What leaves my machine?

Installing fetches the engine from GitHub and the model from Hugging Face, at pinned revisions, and checks each file. After that, generating opens no outbound connection: the server answers on 127.0.0.1, and its chat page loads nothing from anywhere and may talk to its own server alone. Nothing is sent to Vitna. That is true of the code, which its source shows, rather than proved per turn: the count of socket egress bytes and the digest over each completion the design adds are not built.

What happens when something fails?

A request whose step fails ends with an error, never with the reply so far: a 500 of type server_error, or, in a stream, an error event in place of the final chunk with no [DONE] after it. If the GPU can run nothing more, waiting requests get a 503 with the code device_lost and the server exits with status 75, so whatever started it can start it again. A request the context cannot hold is refused, not cut short. The installer resumes a part fetched before, checks every file against its pinned SHA-256, and fetches a file that fails anew, once.

How is this different from Ollama or LM Studio?

They run a large catalogue of models today, and Anchor is checked on five, two of them in its installer, so it replaces neither yet. What it is built around is a mixture of experts whose experts live on the drive and in a cache you size, with the busiest on the GPU, and logits that stay the same, bit for bit, however the experts are cached or read. Measured, a model read off the drive is slower than one held in memory: 2.2 tokens a second against 6.1 on a Ryzen 7 3700X, and 8.7 against 61 on an RTX 3070.

Will streaming wear out my SSD or slow down my other apps?

No. The engine only reads the checkpoint, with direct I/O, and reads do not wear a drive the way writes do. Because they bypass the operating system's file cache, the memory they take is the cache you give the engine and no more.

Does speculative drafting change the model's answers?

No, not by a bit. Every drafted token is checked against logits identical, byte for byte, to the ones decoding a token at a time gives, greedy or sampled with a seed, and the engine's tests compare the two. Drafting changes how fast the answer arrives, never what it is.

Where do I get help?

Through the contact form, Talk to us. The engine's README on GitHub documents every command, flag and route, and how each gate was checked.

A model that fits the machine, on the machine.

Release v0.2.1 installs with one command on Windows or Linux, picks the model the machine can run, and serves it on 127.0.0.1 with a chat page beside it. Nothing to sign up for, and nothing billed.