Experts don't work alone: Amazon's HOPE prunes half of a MoE model and keeps the teamwork intact 🧩

Mixture-of-Experts models are a gift for anyone who wants big-model quality at small-model speed. There's one catch that every homelab owner knows: only a few experts fire per token, but all of them still have to sit in memory. On 9 November, researchers from Amazon published a Hugging Face post on a method called HOPE (Higher-Order Pruning of Experts), and the core idea is simple enough that I keep thinking about it: experts work in teams, so you should prune them like teams.

The memory problem

The post gives a striking example. Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and about 95% of its roughly 250 GB parameter footprint is expert MLPs. Yet for any single token, only 8 experts per layer are used. Expert pruning, permanently deleting the least useful experts, attacks this directly. It needs no training, avoids the latency of loading experts on and off, and works together with quantization.

Why "second-order" matters

Most existing pruning methods give each expert its own importance score and cut the lowest ones, layer by layer. The authors call these first-order methods, because they treat every expert as independent. But experts form stable cooperative groups that are activated together. Remove expert A because it looks unimportant on its own, and you might quietly break a partnership with expert B, which really matters.

HOPE captures those pairwise interactions in an interaction matrix collected during the usual calibration pass. The diagonal holds individual importance, and the off-diagonal entries measure how much pairs of experts contribute together. Choosing which experts to remove then becomes a quadratic program, which the authors say a standard solver handles in one to two seconds per layer. A nice detail: that diagonal reproduces the rankings of REAP, a strong first-order method, almost perfectly (Spearman 0.988), so any gain over REAP comes from the teamwork terms.

The results

The team tested Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air at pruning rates from 10% to 50%, against four first-order baselines. HOPE had the best average rank overall, and its lead grew as pruning got more aggressive. The highlight for me is agentic coding: on Qwen3.5-35B-A3B with half the experts removed, HOPE scored 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8, while REAP dropped to 36.2.

They're also honest about the limits. At 10 to 20% pruning, first-order methods are sometimes just as good, each layer is pruned independently, and interactions beyond pairs aren't modeled yet.

You can try it

The code is open. A small hope package runs three steps, calibrate, solve and prune, and outputs a standard Hugging Face checkpoint that is smaller on disk and in memory, with no changes needed in vLLM or SGLang. It has been tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5.

What I love here is the mindset. Instead of buying more memory, look closer at how the model actually organizes itself, and respect that structure. For people running open models on their own hardware, that could mean a bigger model fits where it didn't before.

Sources

Smaller models that remember who works with whom. Good news for every GPU at home. 🏠

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

SpaceX just asked to fly Starship hundreds of times a year 🚀

Last week the U.S. Federal Aviation Administration took a quiet but important step: it opened a formal review of SpaceX's request to launch Starship from Starbase in South Texas up to 456 times per year . Today the site is cleared for only 25 flights a year. That jump is the clearest signal yet that Starship is meant to become routine hardware, not a rare test rocket. What the FAA is reviewing On 14 October 2026 the FAA published a Notice of Intent to prepare a Programmatic Environmental Impact Statement (PEIS) for SpaceX's Starbase plans. A PEIS looks at a whole program of operations rather than a single launch, which is exactly what you need if you want a reusable rocket to fly the way airlines fly planes. SpaceX's proposal comes in two phases, according to the Federal Register notice: Phase 1: up to 100 launches and 200 landings per year (100 Ship, 100 Super Heavy), any day or time of week. Phase 2: up to 456 launches and 912 landings per year at Starbase. La...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen