Key takeaways
- Run very large open-weight AI models on a gaming PC, according to the project, instead of renting datacenter time.
- A desktop app for Windows or Linux sets the engine up and gives you a window for chatting and tuning.
- Compatible APIs let coding agents talk to your local model, and agent edits avoid recomputing the whole context.
- Free to use under the Apache 2.0 licence, with a research paper behind it.
- The catches: native support is listed for NVIDIA RTX cards, speed claims are untested, and offloaded experts stream over PCIe.
What it is
FreeToken is an edge-native Mixture-of-Experts serving engine, built to run frontier-scale open-weight models on personal and consumer hardware. It is released under the Apache 2.0 licence and the code is on GitHub at FlashML-org/FreeToken.
The problem it solves
Big open-weight models are normally a datacenter job. Developers who want to run one privately, on a machine they already own, usually run out of graphics memory first.
The project's pitch is datacenter-class intelligence on the hardware you already own, and it says it can run 290B-plus frontier models on a gaming PC. That figure is the project's own claim.
What you get
- Big models on modest hardware: it treats GPUs, CPUs, host memory and interconnects as one pool of resources.
- Memory that moves: graphics memory can be re-allocated between expert caches and context memory without restarting the engine.
- Faster agent loops: agent edits such as tool calls and thinking blocks avoid redundant context recomputation.
- Drop-in for coding agents: Anthropic and OpenAI-compatible APIs let tools you already use point at it.
- A friendlier start: a desktop app for Windows or Linux sets the engine up and offers a window for running, chatting and tuning.
The catches
- Hardware: native support is listed for NVIDIA RTX graphics cards, and the project does not name other vendors.
- Speed depends on your machine: when experts live in host memory, cache misses stream over PCIe, and one mode asks you to calibrate once per machine.
- Model coverage: prebuilt kernels are tuned for a list of known-good checkpoints, though the docs say other checkpoints of the same architectures work too.
- Maturity: it is a young project that builds on ideas and code from other open-source serving engines,.
- The desktop app is downloaded from the project's own website, which we did not inspect.
What we did not test
We did not install FreeToken, load any model or measure any speed. Every claim here comes from a literal quote in the project's README, its documentation or the GitHub API, and the performance statements are the project's own.
Verdict
Editors' view: this is for developers with a recent NVIDIA gaming or workstation card who want to try large open models locally and are comfortable tuning memory settings. If you have no such GPU, wait for independent benchmarks before getting excited.
Sources
Need help adopting this, or have a software project in mind?
Tell us what you are building. The initial consultation is free. Email [email protected] or follow @zolnaltd.
Start a conversation →