Can your model find the ball under the cups? Ai2's StateBench tests how LLMs track changes over long documents 🧠

Here's a skill we rarely talk about but use all the time: keeping track of how things change. A variable gets reassigned in code. A character in a novel hands someone a knife. A spreadsheet cell gets edited five times. To get the final answer right, a model has to follow every one of those updates. Ai2 just released StateBench, a benchmark built to measure exactly that, and I really like how cleanly it's designed.

Four levels, explained with variables

StateBench splits state tracking into four levels, and the team explains them in programmer terms:

  • R-trivial: immutable assignment, like let x = 4. You only need the first value.
  • Aperiodic: reassignment by value, like x = 4 and later x = 2. You only need the latest value, like a thermostat.
  • Soluble: self-referential updates, like x = x + 5. You have to carry every change forward, like a game score.
  • General finite: assignment by reference, like x = y or swapping two variables. Think of a magician swapping a ball between five cups.

Under the hood, these levels come from Krohn–Rhodes theory, a branch of algebra that breaks state-tracking problems into basic building blocks. The clever part: Ai2 starts from one hard sequence of assignments and derives three easier versions from it, with the same variables and the same positions. Only the type of assignment changes, so the comparison across levels is fair.

What they found

The frontier models tested (Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna) did well on the first two levels. On the soluble and general finite levels, accuracy dropped sharply as inputs got longer, down to 13% or lower on the longest general finite problems.

Open pretrained models, including the 7B Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), mostly handled the easy level, lost accuracy on aperiodic problems as inputs grew, and stayed at chance on the hardest level. Format mattered too: small models that aced a RULER-style version of a problem were near chance on the same problem framed as spreadsheet edits.

The really interesting experiment

Then the team trained small 275M-parameter models from scratch directly on StateBench, on sequences of up to 256 assignments, and tested them on up to 64,000 assignments. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels all the way out. Transformers learned the easier levels but degraded on long inputs, and they never learned the hardest level. A Gated DeltaNet variant with a more flexible memory update stayed near perfect on the hardest level out to roughly 10 times its training length.

So for recurrent and hybrid models, the gap looks like a training problem, not an architecture limit. That's a hopeful and very practical finding.

Why this makes me happy

For anyone building AI agents, this matters a lot. Agents live in long contexts full of changing state: files edited, tool results overwritten, plans updated. A benchmark that shows exactly where tracking breaks down gives us a way to build better training data and better architectures. Ai2 is also honest about what's still open, such as settings where new entities appear over time, like people joining a long conversation.

Sources

Which kind of state does your agent lose track of first?

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen