2.78 Trillion Parameters, 29GB RAM, Half a Token Per Second: Kimi K3 Actually Runs on a Laptop

Half a token per second — by the time you finish a coffee, it has read one word. But this is the full 2.78-trillion-parameter, 982GB model, running on a 64GB laptop. No distillation, no pruning, no gutted variant.

When Kimi K3’s weights went open last week, its 1.56TB footprint made one thing obvious to everyone: this monster belongs in a datacenter. Less than a week later, an open-source engine called WASTE flipped that assumption — it converts K3 into a 982GB container that opens with 29.05GB of RAM and runs at 0.49–0.54 tok/s on a 64GB MacBook.

The HN thread (155 points, 61 comments) split into two camps: half were doing electricity math, half were asking “what can you even do at half a token per second?” But the speed isn’t the point. Three facts behind the demo are:

• Lesson 1: An MoE model’s weight lives on disk, not in RAM. K3 activates only ~4% of its expert weights per token. The other 96% is “idle weight” — it doesn’t need to be in memory, it just needs to be reachable in time.
• Lesson 2: Storage speed isn’t a detail — it IS the performance. One token reads 17GB of expert data: 12.78 GB/s on internal NVMe streams smoothly; on a USB enclosure it drops to 0.94 GB/s and the same token takes thirteen seconds.
• Lesson 3: Between “it runs” and “it’s usable” lies an engineering canyon. 29GB is only the floor to open the model; real throughput needs 64GB and a terabyte NVMe. This isn’t magic — it’s a series of byte-precise engineering decisions.

〇、The meta-question: what actually blocks trillion-scale local inference?

For three years, local LLM inference has lived under one default mental model: a model needs as much RAM as it has parameters. 70B needs 140GB of VRAM, 405B needs 800GB, and 2.78T K3 by that logic needs 5.6TB — consumer hardware need not apply.

That mental model turned “can it run” into a pure money question: if you can’t afford the RAM, you can’t afford the model. Cloud APIs became the only path — and the cloud means data leaving your machine, per-token billing, and zero offline availability.

WASTE’s value is dismantling the core assumption of that model. In a mixture-of-experts model, the vast majority of weights are idle at any given instant. K3 activates 16 experts across 92 layers per token — roughly 4% of its parameters participate in any single token’s computation. So the question changes:

Not “can all weights fit in RAM?” but “can the weights on disk be read in fast enough when needed?”

That’s the name: WASTE — “Weight-Aware Streaming Tensor Engine.” It reframes trillion-scale inference from a memory-capacity problem into a storage-bandwidth problem. And bandwidth is an engineering problem — one that can be solved.

一、Background: the day after K3 went open, someone wanted it off the cloud

Kimi K3’s release was contradictory by design: 2.78T parameters, 1.56TB raw weights, 1M context — open, but with a license barring “large enterprises.” At that scale it seemed destined for cloud-only. Until WASTE.

The author, marcobambini (creator of the Gravity programming language), stated the motivation bluntly in the HN thread: every token generated in the cloud is paid for twice — once on the invoice, once in datacenter electricity. Meanwhile a 64GB laptop holds this model “barely, awkwardly, but genuinely.” WASTE is the first concrete step toward ending that waste.

The requirements list is uncompromising:

Note that “RAM floor”: 29GB is the door to open the model, not the threshold for smooth running. A 32GB machine technically opens it and pages terribly; the author says plainly “treat 64GB as the real requirement.” That gap between the headline 29GB and reality is engineering honesty — the distance between marketing and truth.

二、How it works: turning “fast enough reads” into “it fits”

Decision 1: Stream experts on demand — one pread per expert

The model is converted once into a .waste container: a JSON manifest, a resident trunk (27.28GB, 4/8-bit quantized), and one expert bank per layer.

The key layout: each expert record is 4KiB-aligned with gate, up, and down matrices physically adjacent — routing to an expert costs exactly one pread syscall, not three, not one seek per matrix. As the author puts it: “The arithmetic was never the bottleneck.”

Reads deliberately bypass the page cache (F_NOCACHE on macOS, O_DIRECT on Linux): with a container smaller than RAM the kernel would cache everything, but a 982GB model can’t be cached — hit rates measured through the page cache are a fiction that dies on contact with the real model.

Decision 2: Experts at 3 bits — and only experts

Expert weights use residual vector quantization: 3 stages of 256-entry codebooks over 8-dimensional vectors, 3.00 bits per weight, the matrix never materialized. Each token, the engine builds a partial dot-product table (one per codebook entry per vector position), after which each expert row is three table reads and two adds.

The trunk stays at 4/8 bits — K3 was trained with quantization-aware training on the experts only, so it has no trained tolerance for a squeezed trunk. The author actually built and measured a 3-bit trunk: the cache prediction held, throughput didn’t, and output collapsed. That “we tried it and it failed” detail is worth more than the result itself.

Decision 3: The cache floor is one token’s working set

The most predictive number in the project: K3 touches 16 experts in each of 92 layers per token — 17.0GB. Below that, an expert cached for one token is evicted before the next token asks for it. Hit rate isn’t low — it’s zero.

Above the threshold the curve bends sharply, with a counter-intuitive trap:

*The 46GB row now measures 0.51 tok/s with read-ahead shipped.

Why are 52GB and 58GB slower? Because those configs drive the machine into paging hell — and the machine never fully recovers: after sweeping 46→52→58, re-running 46GB yields only 0.22–0.25 tok/s while reporting identical hit/miss counts. The engine is deterministic; the machine is not. The author’s advice: sweep upward.

三、Results: how “real” is it?

This is the part easiest to doubt — an engine that crams a 2.78T model into a laptop, is the output trustworthy? The author’s verification:

• Every layer validated against a PyTorch reference
• Final logits agree to 3.6e-06
• The vision tower matches its own oracle to 2.3e-06
• Neither the format nor the engine is K3-specific: “a model that streams at 2.78T streams comfortably at 48B”

There’s also an on-ramp: Kimi-Linear-48B-A3B runs from a 19GB container with a 1.87GB floor at 10.7 tok/s — you can try WASTE without donating a terabyte of disk first.

The cost math was done in the comments: at ~42W sustained and 20¢/kWh, roughly $5 per million tokens, hardware excluded. Compare a GPU cluster’s ~80k tok/Wh: this laptop is 1000–2000x worse on energy efficiency. That’s the physical price of moving a model off the cloud, and no engineering removes it.

四、Two fights in the comments: is it useful? who wrote it?

Fight 1: half a token per second — what for?

Most skepticism targets utility. The most honest arithmetic: at this speed, a 1M-token output takes 23 days; even asynchronously, an answer takes 30 seconds to half an hour. One suggestion: communicate with it like email — send the question, come back later. This is not for real-time interaction.

But flip it around: some scenarios never needed real-time. Companies that can’t let data leave the machine (healthcare, finance, government), compliance environments that need a full model on-prem, offline settings that still want frontier capability. For them, a complete K3 at 0.5 tok/s beats a gutted model at 100 tok/s. WASTE’s selling point isn’t speed — it’s erasing the line between “you may not send that data to an API” and “run it here.”

Fight 2: the README was written by AI?

The top comment called it: “This README hits all my ‘authored by an LLM’ instincts,” and the git log literally credits commits to claude. The author owns it openly: “I’m using my skills to orchestrate LLMs and agents, and I can write better code much faster.” The thread split — some called it lazy, some called it honest (simonw: “Claude writes better commit messages than most people”), some begged “at least read the README before shipping it.”

The whole thing is a footnote to the era: an AI-written project running an AI-written inference engine that lets AI models run on your laptop. The toolchain’s self-reference is moving faster than anyone can process.

五、Summary: it changes possibility, not speed

WASTE changes the default association of the phrase “trillion-parameter model.” It used to mean datacenter, per-token APIs, data leaving your machine. Now there’s another option: a 64GB laptop, a terabyte NVMe, half a token per second — but the model is yours, the data is yours, and the electricity bill is yours too.

Half a token per second is obviously not the destination. But running frontier models on consumer hardware went from “impossible” to “slow” — and “slow” is a much better starting point than “impossible.”

🤖 Following local LLM deployment?

• Do you have scenarios where data can’t leave the machine? WASTE proves the road isn’t dead.
• Full model at 0.5 tok/s vs a small quantized model at 100 tok/s — which fits your use case?
• Is your cloud API bill quietly eating your margin?

📬 Follow 梦兽编程 for more AI model landscape and local deployment analysis.

Sources:

• GitHub: WASTE — Weight-Aware Streaming Tensor Engine External link
• HN: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s External link
• Moonshot AI: Kimi K3 open-weights release External link