A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory.

The trick: leave the experts on the SSD

The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction, and the system is named Edge0. It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment.

Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter: a tiny trained head per layer that predicts, one token ahead, which experts the next layer will need. That prediction becomes the routing, so there are no "misses", and reading from the SSD can run in parallel with the computation instead of stalling it.

To make up for the quality lost to int4 and the replaced routing, the authors add a small recovery LoRA, distilled from the full-precision model. A nice detail: they keep it as a separate adapter, because merging it into the weights and re-quantizing throws away most of its benefit.

The numbers

On a Mac mini M4 Pro with 24 GB:

  • Edge0, 35B tier: 20.4 tokens/s decode with a peak of 2.9 GiB of memory.
  • Plain mlx-lm with everything loaded in memory: 3.9 tokens/s at 18.2 GiB.

There's also an 8B tier that does 28 tokens/s in 1.5 GiB.

It isn't free. On the OpenCompass average the 35B tier scores 79.2 versus 83.2 for the full-precision model, and most of that gap is on AIME competition math. For everyday assistant and agent work that's a very reasonable trade for a 6x smaller memory footprint.

What I like most: the honesty

The paper is refreshingly clear about where the speedup actually comes from. On the 24 GB Mac mini, turning the prerouter off still gives 19.9 tokens/s, so on that machine the win mostly comes from the streaming engine itself. The prerouter really shines on a 16 GB MacBook M2, where the model can't fit in memory at all: there it adds roughly 80% more speed. That's the kind of ablation that makes me trust the rest of the results.

The limitations are spelled out too: one request at a time, a floor of about 44 ms per step for CPU-side graph building, and for now only an MLX backend, so Apple Silicon only. A CUDA version is described as an architecture question rather than a code question, which makes me hopeful for Linux homelab boxes with fast NVMe drives. 🌌

Why this matters

For years, "how big a model can I run?" was mostly answered by "how much RAM or VRAM do you have?" Edge0 shows that for MoE models, a fast SSD plus smart prediction can stand in for a lot of that memory. The code is open and the models are on Hugging Face under Apache-2.0, so anyone with a recent Mac can try it today.

Big models on small machines, with an honest write-up: that's my favorite kind of progress.

Sources

Reacties

Een reactie posten

Populaire posts van deze blog

SpaceX just asked to fly Starship hundreds of times a year 🚀

Last week the U.S. Federal Aviation Administration took a quiet but important step: it opened a formal review of SpaceX's request to launch Starship from Starbase in South Texas up to 456 times per year . Today the site is cleared for only 25 flights a year. That jump is the clearest signal yet that Starship is meant to become routine hardware, not a rare test rocket. What the FAA is reviewing On 14 October 2026 the FAA published a Notice of Intent to prepare a Programmatic Environmental Impact Statement (PEIS) for SpaceX's Starbase plans. A PEIS looks at a whole program of operations rather than a single launch, which is exactly what you need if you want a reusable rocket to fly the way airlines fly planes. SpaceX's proposal comes in two phases, according to the Federal Register notice: Phase 1: up to 100 launches and 200 landings per year (100 Ship, 100 Super Heavy), any day or time of week. Phase 2: up to 456 launches and 912 landings per year at Starbase. La...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

Google's Deep Review wants to be the second pair of eyes on every long document 🔍

AI has made writing fast. Checking has not kept up. That gap is exactly what Google Research is going after with Deep Review, which went public on 14 October. It's an agentic system that reads a long technical document and tries to find what's wrong with it: broken proofs, logic gaps, unsupported claims. For anyone working on legal and document AI, like me, this is a launch worth a close look. From conference pilot to public tool Deep Review is the grown-up version of Google's Paper Assistant Tool (PAT). Over the past year, authors of more than 16,000 manuscripts at STOC 2026, ICML 2026, NeurIPS 2026 and ICLR 2027 opted in for private feedback before the submission deadline, fully separate from official peer review. The need is real: Google counts over 74,000 submissions to ICLR, NeurIPS and ICML this year, up from about 23,000 in 2023. Now anyone can use it. Every user gets two free reviews per week, and you can connect a Gemini API key for higher limits and programmati...
Meer lezen