Baidu just released ERNIE 5.5 Flash as a raw base model, and that's the interesting part 🧱

Most open model launches hand you a finished assistant. Baidu went the other way this weekend: it put ERNIE-5.5-Flash-Base on Hugging Face, the checkpoint taken at the very end of pre-training, before any mid-training, instruction tuning or reinforcement learning. In other words, the foundation, not the house.

What was released

According to the model card, ERNIE 5.5 Flash is a sparse Mixture-of-Experts model with 280B total parameters, of which only 14B are active per token. It was pre-trained from scratch on 33.5 trillion tokens with the Muon optimizer, at a training context of up to 128K, and the released config supports fine-tuning and inference up to 1M tokens. It is text-only, and Baidu publishes both FP8 and BF16 weights, with serving instructions for SGLang and vLLM (via a vLLM-ERNIE plugin).

Why I like this

A base checkpoint is the most useful thing a lab can give researchers and builders. Every chat model makes choices about tone, safety style and reasoning format. A base model leaves those choices to you. Baidu says exactly that: different domains and agentic applications need different data and behaviour, so they let developers decide where to take it next. If you want to study instruction tuning, try your own RL recipe, or adapt a model to a niche domain like education, this is a clean starting point.

The architecture is also a nice tour of where efficiency research is heading. Of the 43 layers, 37 use what Baidu calls Heavily Compressed Attention, which caches a single entry per 128 tokens, while 6 layers keep full-context access through Multi-head Latent Attention with sparse attention. That is a very deliberate trade between KV-cache size and long-range recall. Each MoE layer has 512 routed experts (top-10) plus one shared expert, picked by several independent routers instead of one. And instead of a separate multi-token-prediction module, a technique called ECHO reuses the model's own final layer to predict future tokens, which doubles as a built-in draft model for speculative decoding. There's even a small tokenizer detail I enjoyed: numbers are split right-to-left in groups of up to three digits.

On Baidu's own unified evaluation, the base model scores 93.3 on MMLU, 78.4 on MMLU-Pro and 82.3 on HumanEval+, ahead of the other ~300B base models in their table such as DeepSeek-V4-Flash-Base. Those are the publisher's numbers, so I'd treat them as a promising signal and wait for independent fine-tunes to show what it really does.

The honest fine print

This is not something you run on a homelab GPU: the SGLang example uses four-way tensor parallelism. It is a base model, so it completes text rather than following instructions. And the license is Baidu's own ERNIE 5.5 Flash Base Model License Agreement, not Apache or MIT, so read it before building on it commercially.

Sources

Still, I love seeing a big lab share the raw pre-trained weights. That's how the community gets to experiment, and I'm curious which fine-tunes show up first.

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen