SenseTime's Loop-9B runs its own layers twice, and gets a deeper model for free 🔁

Here's a question I keep coming back to when I pick models for my homelab: do you really need more parameters to get more thinking, or can you get extra depth out of the weights you already have? On October 30, SenseTime's OpenSenseNova team released an open model that bets on the second answer, and I love how simple the core idea is. It's called SenseNova-Loop-9B-Preview.

The trick: run the middle of the network twice

A normal transformer pushes your input through each layer once. Loop-9B has 32 distinct decoder layers, but it treats them differently. Layers 0 to 3 run once, then the block of layers 4 to 27 is applied twice in a row with the same shared weights, and finally layers 28 to 31 and the output head finish the job. The model card writes it nicely as B(B(h)): the second pass takes the first pass's output as its input.

The result is 56 decoder-layer executions per forward pass while only 32 layers' worth of weights are stored. SenseTime describes it as roughly 9B total parameters, but about 14.6B "inference parameters" because of the repeated computation. In other words, you pay for depth with compute, not with memory.

Keeping two streams in balance

Reusing layers sounds easy, but feeding a block its own output can get unstable. To manage that, the looped section uses what SenseTime calls two-stream manifold-constrained hyper-connections (mHC). Instead of a single residual stream, there are two, and learned mappings decide how information moves and mixes between them, with those mappings constrained using Sinkhorn normalization. The decoder also interleaves linear attention and full attention in a repeating four-layer pattern, which is a familiar recipe for keeping long contexts affordable.

Built for long documents and agents

This isn't just an architecture demo. The model takes text and images and produces text, and it's aimed at document understanding, coding and tool use. The documented training recipes mix long sequences, images, code and tool trajectories, with training sequences up to 262,144 tokens. Multi-token prediction (MTP) is already available for faster inference, and two speculative-decoding variants, Dspark and Dflash2, are still in training. Weights are on Hugging Face under the Apache 2.0 license, and the model card shows how to load it with Transformers on a single GPU.

Honest about the limits

This is clearly labelled a preview. The technical report is still marked "TBD", benchmark results are published as charts by the team itself, and independent tests don't exist yet. Running the looped block twice also means more compute per token than a plain 9B model, so "free depth" is really "free in memory, not in time". I'd want to see real numbers on my own hardware before drawing big conclusions.

Why I like it

Looped and recurrent-depth models are one of the most interesting directions for anyone who runs AI at home or in a classroom. Memory is usually the hard wall, and this design spends compute instead. If ideas like this mature, a small open model could quietly "think deeper" on hardware schools and hobbyists already own.

Sources

I'm curious whether three or four passes would keep helping, or whether two is the sweet spot.

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen