An LLM edited another model's weights: MIT's Weight-Space Programming results 🔧

Most of the time, when we "improve" a language model, we mean we train it again. More data, more gradients, more compute. This week a team at MIT CSAIL showed something stranger and, to me, more interesting. They asked a frontier model to read and rewrite another model's weights directly, with no gradient descent and no training loop. They call it Weight-Space Programming.

What they actually did

On 2 November, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab published a long research blog with full experiment logs. The headline result for people who run small open models at home is Experiment 8. They gave GPT-6 Astra a Python session over the weights of Qwen3.5-0.8B, practice problems from GSM8K and MATH (not the test sets), and a budget of forward passes. Training and gradients were off limits.

Astra spotted a failure mode that Qwen's own model card already warns about: thinking-mode loops. The base model kept repeating itself and often hit the 32,768-token limit before answering. Astra traced that loop to a direction written by the MLPs in layers 15 and 19, then subtracted it from 64 output columns in each layer. That is about 131,072 weights, or 0.016% of the model.

Under their thinking-on settings, MATH-500 rose from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval from 21.0% to 62.3%. The share of MATH-500 answers that hit the token limit fell from 93% to 33.1%. The edited checkpoint and the full logs are public.

Be honest about what improved

This is not magic math ability appearing from nowhere. An independent Hugging Face rebuild of the same edit reproduces the numbers closely, and it makes the mechanism clear: with thinking on, almost all of the gain comes from answers that no longer loop. When thinking is off, the edit costs a point or two. On Gemma 3 270M, which rarely loops, Astra could not raise MATH-500 by more than about a point. So the honest framing is: Weight-Space Programming can remove a concrete obstacle that was hiding ability the model already had. It is not yet a general way to add new skills.

That still matters. Targeted surgical edits that leave 99.984% of the weights alone are exactly the kind of fix that fine-tuning often messes up through catastrophic forgetting.

Why I care

If you fine-tune small open models for agents, tutoring tools or a homelab stack, this is a new mental model. Sometimes the right move is not another epoch. Sometimes it is finding the bad direction and deleting it. The authors are building a proper benchmark and harness so this stops looking like black magic and becomes something you can measure. I will be watching that closely.

Sources

Would you rather spend your next GPU-hour on fine-tuning, or on hunting one bad direction in the weights?

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen