39.3 Tokens a Second on a Laptop GPU. Codex Averages 33.

FreeToken is an open-source inference engine from UC Berkeley and UT Austin, released under Apache 2.0 on August 17, 2026. It runs Mixture-of-Experts models far larger than your GPU’s memory by treating the whole machine as one system: GPU, CPU, system RAM plus PCIe together. The paper reports a 35B model at 39.3 tokens per second on an 8GB laptop GPU, a 284B model on a single RTX 5090, plus the 753B GLM-5.2 on one workstation card. For comparison, the median decode speed in real Codex production traces is 33 tokens per second. It exposes Anthropic and OpenAI compatible APIs, so Claude Code, Codex, OpenClaw plus DeepSeek Harness can point at it directly. Best for anyone with a decent gaming PC. Not ideal for people who want an installer and no terminal.

The number that matters is not the 753 billion parameter model on one graphics card, impressive as that is.

It is 39.3.

That is the tokens per second FreeToken’s paper reports for a 35 billion parameter model running on an 8GB laptop GPU. A laptop. The kind people already own.

The median decode speed measured across real Codex production traces is 33 tokens per second.

So a laptop running a local model, for free, generates text faster than the cloud service you pay for. Not on a benchmark designed to flatter it. Against production traces from the actual product.

What It Actually Is

Verified against the paper, the repository plus multiple independent writeups.

The author list is the part that should make you pay attention. Co-first authors are Shuo Yang, a Berkeley EECS PhD student, plus Xiaoze Fan at UT Austin. The senior authors include Ion Stoica, Matei Zaharia, Kurt Keutzer plus Song Han.

Stoica and Zaharia created Apache Spark. Stoica also co-created Ray and vLLM, which is the serving engine most of the industry runs on. This is not a weekend project from an anonymous account. It is the group that built the infrastructure layer, aiming at consumer hardware.

What Ion Stoica Building This Tells You

Worth pausing on the author list, because it changes how you should read the paper.

Ion Stoica co-created Apache Spark, Ray plus vLLM. Matei Zaharia created Spark and is CTO at Databricks. Kurt Keutzer has been at Berkeley for decades. Song Han’s group at MIT produced much of the foundational work on model compression.

vLLM in particular matters here. It is the serving engine a very large share of the industry runs in production, built by this same group, designed for datacenter GPUs. FreeToken’s paper explicitly says it builds on the GPU-centric serving substrate of vLLM plus SGLang.

So the people who built the datacenter serving layer have now published the consumer one, using the same foundations.

That is a meaningful signal about where they think this goes. Not a startup pitching a product, not enthusiasts optimising a hobby, but the group whose software already runs frontier inference deciding that personal hardware is a serious target.

It also means the engineering is likely sound even where the marketing numbers get quoted loosely. The bandwidth-adaptive split, the closed-form ratio staying resident inside a captured CUDA graph, the elastic cache resizing without a restart. Those are systems-research answers to systems-research problems, from people with a long record of them.

None of that makes the community benchmark accurate. It does mean the paper deserves reading rather than skimming.

The Idea, Without the Jargon

Big models used to mean big GPUs. FreeToken’s argument is that this was always a software problem rather than a hardware one.

Mixture-of-Experts models are the key. Instead of one enormous network where every parameter fires on every token, an MoE splits its weights across hundreds of small expert sub-networks per layer. A tiny router picks a handful of experts for each token. Qwen3.6-35B activates roughly 3 billion parameters per token out of 35 billion total.

So the model is huge on disk while only a fraction of it does any work at any moment.

Where the old approach stalls

Traditional offloading noticed this and did the obvious thing: keep the full expert pool in system RAM, cache some experts on the GPU, fetch the rest over PCIe when needed. That works, slowly, because every cache miss becomes a transfer and PCIe is the bottleneck.

FreeToken’s change is that a cache miss has two possible answers rather than one. It can copy the expert to the GPU, or it can just run that expert on your CPU.

Both routes compete for the same host memory bandwidth. Saturate PCIe and you starve the CPU. Send everything to the CPU and the GPU sits idle. So the system measures your specific machine with a command called ft bench bw, then splits misses between the two paths based on real numbers instead of a fixed percentage somebody guessed.

Why That Matters More Than the Headline

A fixed offload ratio is a guess about hardware the author does not have.

PCIe bandwidth varies enormously. A desktop RTX 5090 on full lanes behaves nothing like a laptop GPU wired through fewer, or a desktop where a second device shares chipset lanes. A policy tuned on one configuration falls apart on another, which is why local inference advice is full of people arguing about settings that worked for them.

Measuring the machine removes the argument. The split is derived as a closed-form ratio from two measured bandwidths, cheap enough to stay resident inside a captured CUDA graph.

There is a second design choice worth knowing about. When VRAM runs short, most systems throw a CUDA out-of-memory error and die. FreeToken shrinks its GPU expert cache instead, without restarting or reloading weights, pushing more misses to CPU execution. Inference continues, slower.

Changing the failure mode from crash to degrade is a small thing that makes local inference feel usable rather than fragile.

The Numbers, With Sources Attached

Being careful here, because three different sets of figures are circulating.

From the paper. An 8GB laptop GPU serves a 35B model at 39.3 tokens per second. A gaming desktop with an RTX 5090 serves DeepSeek-V4-Flash at 284B. A single workstation GPU serves GLM-5.2 at 753B. Decode throughput runs 1.5 to 2.3 times faster than llama.cpp, Ollama plus KTransformers, with tail time-to-first-token under 44 seconds, per MarkTechPost’s summary.

From a community test, which is where the viral post came from. An RTX 5080 with 16GB VRAM, a Ryzen 9 9950X3D plus 64GB of system RAM ran Qwen3.6-35B-A3B in NVFP4 at roughly 20GB, hitting about 100 tokens per second. One run with a 1,028 token prompt reached around 110.

The person who posted it flagged the caveat themselves, which is worth repeating: the ~100 figure is a community result, not a benchmark reported in the paper.

From an independent Japanese writeup, an RTX 3080 laptop GPU with 16GB running at 80 watts served a 23.5GB Qwen3.6-35B-A3B build. That is a thermally limited laptop chip handling a model larger than its VRAM.

Three separate sources, three different machines, consistent direction.

The Claim You Should Check Hardest

One line in the reporting deserves more scrutiny than it has received.

Expert output stays bit-exact. No router changes, no expert substitution, no precision relaxation.

That is a strong claim. Most techniques for running big models on small hardware trade accuracy for fit. Quantization rounds weights. Pruning removes them. Expert-skipping approximates the router. Every one of those changes what the model produces, which is why “runs on your laptop” usually means “runs a worse version on your laptop.”

FreeToken is claiming it changes where computation happens rather than what gets computed. Same experts, same precision, same routing, different hardware executing it.

If that holds, it is a categorically different proposition from the 2-bit quantization work we covered last week, where the entire question was how much quality survives compression. Here there should be nothing to lose.

Nobody outside the team has verified bit-exactness independently yet. It is the single most checkable claim in the paper, plus the one most worth checking.

The Agent Angle Is the Real One

One design detail in the paper is aimed squarely at agents rather than chatbots, plus it explains why this arrives now.

Traditional offloading systems get evaluated on single-shot, short-prompt inference. Ask a question, get an answer, done. That is not what agent workloads look like.

An agent session re-enters prefill at every tool-calling turn. It writes code, calls a tool, reads the result, thinks, calls another tool. Each turn drags the whole growing context back through the system. The paper points out that none of the existing offloading systems provide cross-request prefix reuse for exactly this pattern.

Worse, agent workloads keep changing their execution pattern. Writing code activates one distribution of experts. Calling tools activates another. A cache tuned for one is wrong for the other, minute by minute.

FreeToken’s answer is semantic anchor checkpoints for recurrent state plus KV caches, letting agentic context edits like tool calls and thinking blocks avoid redundant recomputation. Combined with elastic memory management that shifts VRAM between expert caches plus KV memory at runtime without restarting.

That is a system built for the thing people actually do with these models now, rather than for the benchmark everyone still quotes.

It also explains the timing. Local inference has existed for years. Local inference that survives an eight-hour agent session is a different problem, plus it only became worth solving once people started running eight-hour agent sessions.

Claude Code Can Point At This

Here is the part with immediate practical consequences.

FreeToken exposes both Anthropic-compatible and OpenAI-compatible APIs. The README lists the agents it works with: Codex, Claude Code, OpenCode, OpenClaw plus DeepSeek Harness.

Which means your Claude Code client can point its backend at a gaming PC in the next room running a 284B model.

Sit with what that does to the economics. Agent workloads burn tokens continuously. A long coding session with subagents, tool calls plus repeated context can run into real money, which is the whole reason people watch their usage. Running that against local hardware turns a per-token bill into a fixed electricity cost.

If you would rather keep the whole stack local, our OpenClaw deep dive covers the containment side of running agents on your own machine.

The paper’s own framing points at exactly this. They identify solo developers, startups, small engineering teams plus regulated industries like healthcare, legal and defense as the intended users. The common thread is people facing either large agent token bills or data that cannot leave the building.

Our Claude Pro review covers what the subscription actually gets you. This does not replace it for most people. But for anyone running agents at volume, a one-time hardware cost against a monthly bill is now a real calculation rather than a theoretical one.

What It Actually Takes to Run

Honest setup expectations, because the demo numbers assume things.

The easy path is the desktop app. Windows or Linux, one-click, with a GUI for running models, chatting plus tuning the engine. It sets everything up for you.

The command line path is Linux x86_64 with an NVIDIA GPU. Install with uv pip install "freetoken[accel]", then ft serve pointed at a model directory.

You need the CUDA 13 Toolkit, specifically nvcc, because the CUDA kernels get compiled just-in-time on first run. That is a real dependency and a common failure point.

System RAM is the quiet requirement. The expert pool lives there, so the 753B configuration needs hundreds of gigabytes. Windows Forum’s writeup made this point plainly: “single GPU” in the headline does not mean a conventional one-card desktop, since the largest configuration still depends on 512GB of system RAM, a capable CPU plus fast host-to-GPU connectivity.

For the configuration most people care about, a 35B MoE, the requirement is far more ordinary. 64GB of system RAM plus a modern card. That is a reasonable gaming build rather than a workstation.

Downloads are large. The Qwen3.6-35B NVFP4 build is roughly 23.5GB.

Where the Concerns Are

Two criticisms have surfaced, both fair.

The first is first-token latency. MoE offloading historically suffers during prefill, when the system has to pull experts layer by layer to process your prompt. Users have raised this specifically for 8GB GPUs. The paper reports tail time-to-first-token under 44 seconds, which is honest reporting of a real cost. Forty-four seconds before the first word appears is a different experience from a cloud API.

FreeToken addresses this with full-layer double-buffered prefill streaming plus semantic anchor checkpoints, letting agentic context edits avoid redundant recomputation. Whether that holds on modest hardware is exactly what independent testing will show.

The second criticism is that on small models the speed gain over existing tools is marginal. That is true and it misses the point. The value is not beating Ollama on a 7B model. It is running a 284B model at all on hardware that cannot hold it.

Worth adding a third thing nobody has raised. Every figure above comes from the paper or from community tests. Those community tests use different hardware from each other. There is no independent benchmark running FreeToken against llama.cpp and Ollama on one machine yet. That will arrive, given the attention this is getting.

Why This Keeps Happening

Step back and this is the third story in two weeks with the same shape.

Somebody wrote a C inference engine over a weekend to run a video model on Apple Silicon that a review had declared impractical. A two-person lab shipped a 2-bit build putting a 27B model on a 12GB card. Now a Berkeley group has published an engine making a 284B model run on a gaming desktop.

None of those are new models. They are all engineering work on top of weights somebody else released, which is only possible because those weights were released.

The paper makes the argument directly. Open weights only solve half the problem, since anyone can download a large model but most people cannot actually run it. There are over 100 million consumer machines with discrete GPUs sitting idle most of the time. FreeToken needs no new hardware, it just uses what people already own better.

That is the answer to every argument that open weights do not matter because the models are too big for normal people. The models were never the constraint. The serving software was. That is a problem a hundred smart people can attack in parallel.

The Part Worth Keeping

A laptop GPU generating 39.3 tokens per second, against a median of 33 in Codex production traces.

The laptop is not faster because it has better hardware. It is faster because somebody spent a year working out how to use hardware nobody thought was enough.

Every headline about this will lead with the 753 billion parameter model on one card. That number is real. But it needs 512GB of system RAM plus a workstation card, which almost nobody has.

The laptop number is the one that changes anything, because roughly a hundred million people already own the machine.

Charts and Blocks

What runs where

Decode speed against the cloud baseline