Key takeaways

  • Keep one coding-agent session running for days: old conversation is folded into layered summaries instead of piling up.
  • Compression is model-driven and reversible: the model decides when to fold, and can restore a range when exact details matter.
  • The project's own claims: a small context window is enough, and token savings of 5x fewer tokens. We did not verify either.
  • It sits between your agent and its model API, so it works with many coding agents without changing them.
  • The catches: the README calls it early with rough edges, and the licence is MIT plus an attribution term.

What it is

billion-context is an open-source proxy that sits between a coding agent and its model API, rewriting Anthropic and OpenAI streams. Per its README, it compresses consumed conversation into layered summaries so you can run a single session for days. The code is on GitHub at ranxianglei/billion-context.

The problem it solves

The README puts it bluntly: long coding sessions blow up context. Anyone who leaves an agent working on a large task knows the result, as the session degrades, restarts or gets expensive.

This tool aims to keep a single session going rather than starting over, by shrinking what is sent to the model while leaving your own history alone.

What you get

  • Fewer repeated tokens: old conversation is folded into summaries written in small ranges, and the cache prefix stays intact.
  • The model stays in charge: compression is model-driven rather than a hard truncation limit.
  • Nothing lost for good: a compressed range can be restored when exact details are needed again.
  • Your record stays yours: the client's stored history is never rewritten.
  • Low risk of breakage: endpoints the proxy does not recognise are forwarded unchanged.

How it compares

These are the project's own figures, taken from its GitHub description, not our measurements. It says a small context window is enough, and claims token savings of 5x fewer tokens.

The catches

  • Maturity: the README's status section says Early, tells you to expect rough edges, and calls real-model integration testing the next milestone.
  • Licence: the README says MIT plus one additional attribution term, and GitHub's detection returns NOASSERTION. Read the full terms before using it in a product.
  • Privacy and exposure: every model request passes through it. If you expose the port beyond your machine, there is no authentication, so only a trusted network is advised.
  • Side effects: with the Claude Code lane, features bound to an Anthropic account are disabled while traffic is routed through it. Your client's own history file keeps growing every turn.
  • Updates: newer versions are installed in place by default, and the development branch can be broken at any moment.

What we did not test

We did not install or run billion-context, connect an agent, or measure token use, cost or summary quality. The savings and window figures are the project's own claims. Every statement here comes from a literal quote in the README or the GitHub API.

Verdict

Editors' view: this is aimed at people whose coding-agent sessions regularly outgrow the context window, and who are willing to put a proxy in the path of their prompts. Given the project describes itself as early, try it on a disposable project first and check the licence terms.

Repository
ranxianglei/billion-context
TypeScript · NOASSERTION
View on GitHub ↗

Sources

Need help adopting this, or have a software project in mind?

Tell us what you are building. The initial consultation is free. Email [email protected] or follow @zolnaltd.

Start a conversation →