Kimi K3.1 Pro is out in the open: 2.8 trillion parameters, 1M context, and only 104B awake per token 🧠

Every few months an open-weight release lands that makes me stop and reread the model card twice. This week it's Kimi K3.1 Pro from Moonshot AI. The weights are now public on Hugging Face under moonshotai/Kimi-K3.1-Pro, and the numbers are wild: 2.8 trillion total parameters, 104 billion activated per token, text and image input, and a context window of 1,048,576 tokens.

What's inside

Kimi K3.1 Pro is a Mixture-of-Experts model, post-trained from Kimi K3 with large-scale joint reinforcement learning across hundreds of downstream domains. The model card lists some details I really enjoy as an architecture nerd:

  • 896 experts, of which 16 are selected per token, plus 2 shared experts.
  • 93 layers, mixing 69 Kimi Delta Attention (KDA) layers with 24 Gated DeepSeek Sparse Attention (DSA) layers. Compared with Kimi K3, Moonshot replaced the full-attention layers with sparse attention to make long contexts more efficient.
  • A 401M-parameter vision encoder (MoonViT-V2).
  • MXFP4 weights and MXFP8 activations, trained with quantization-aware training, so the low-precision format is part of the design rather than an afterthought.

Even in 4-bit form, the full set of weights is about 1.56 TB across 96 safetensors files. This is not a model for my homelab, and that's fine. Moonshot worked with vLLM, SGLang and TokenSpeed so serious deployments can start right away, and there's a hosted API too.

Why I like it

First, it's a genuinely agentic model shipped in the open. On Moonshot's own evaluation table it scores 65.7 on Terminal-Bench 4.0, 51.7 on SWE Marathon v1.1 and 92.1 on BrowseComp, with most coding runs done in a Claude Code harness. Those are self-reported numbers, and the same table shows frontier closed models ahead on several benchmarks, such as HLE-Diamond and Terminal-Bench-Science. I appreciate that honesty: it's competitive, not magic.

Second, the details for builders are thoughtful. Reasoning effort is a simple request field with low, high and max. For multi-turn tool use, the card asks you to pass back the full assistant message, including reasoning_content and tool_calls, so the model keeps its own chain of thought between turns. That's a small note that saves agent developers a lot of debugging.

Third, the license is fairly open. It's a modified MIT-style license that allows use, modification, fine-tuning and redistribution. Extra conditions only kick in at large scale: a separate agreement for model-as-a-service businesses above 20 million US dollars in revenue over 12 months, and visible "Kimi K3.1-Pro" branding for products above 100 million monthly users or 20 million dollars in monthly revenue. Internal use is exempt.

For researchers, a 2.8T open checkpoint is also a fantastic teacher. Distilling knowledge from a model this size into small models you can run at home is exactly where I expect the most interesting community work to happen.

Sources

Which small model would you distill from a 2.8T teacher first?

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen