Who said what, in 40 languages: NVIDIA opens up Nemotron 3.5 Transcribe and its data recipe 🎙️

Speech-to-text has quietly become one of the most useful AI tools I use. But a plain transcript often isn't enough. In a meeting you want to know who said what, in a class you want to jump to the moment something was explained, and in a doctor's office you want "HbA1c" spelled correctly. On 28 October, NVIDIA released Nemotron 3.5 Transcribe on Hugging Face and ModelScope, together with Granary v2, the open data pipeline behind it. I really like how much of the recipe they shared.

One model, several jobs

Transcribe is a single promptable checkpoint for transcription, speaker attribution, translation and audio question answering. According to NVIDIA's announcement, it transcribes 40 languages (Dutch is one of them), translates 31 of them to or from English, attributes speakers in 13, handles recordings up to an hour long and labels up to eight speakers.

You steer it with plain language, like "Transcribe this recording and label each speaker" or "What decisions were made in this meeting?" You can also give it a short list of possible terms, with the instruction to use them only when the audio supports them.

Under the hood, a speech encoder feeds a hybrid Mamba-Transformer mixture-of-experts language model. It has about 30B parameters in total, but only about 4B are active per token. Speaker cues from NVIDIA's Sortformer reach the language model together with the audio features, so it can tag each turn while it writes, instead of attaching speakers afterwards.

The numbers

NVIDIA reports 5.33% average word error rate on the English track of the Hugging Face Open ASR Leaderboard, first among open-weight models. On Artificial Analysis's independent speech-to-text index it ties for first among open models at 2.8%, with roughly a sixth of the active parameters of the model it ties. It also posts the lowest diarization error rate (18.26% across 21 languages) on David AI's multilingual leaderboard. In NVIDIA's own benchmarking, the upcoming NVFP4 version processed 100 hours of audio in under three minutes on a single RTX PRO 6000 Blackwell GPU.

Why I find this exciting

For me, the best part is Granary v2. Its public manifest covers 463,062 hours of speech across 48 languages and 16 source corpora, up from 25 languages in the first version. Every recording gets three transcript styles, from verbatim with "uh" to clean, written-form text, plus quality flags and provenance. That's the kind of openness that lets others actually learn how a strong speech model gets built.

The adaptation story is just as practical. In a medical case study, a better prompt cut word error rate from 30.8% to 23.8%, and fine-tuning brought it down to 11.8%. On held-out speakers, the error rate dropped from 22.44% to 12.17%, and the whole adaptation took about 14 GPU-hours on one GB300.

And as someone who cares about education: Agora used the word-level timestamps to put a recorded class next to its transcript, so students can see what was said and when.

Honest notes

Transcribe works on recorded audio, not live streams. The licence is NVIDIA's OpenMDW-1.1, so read it before you build on it. Quantized versions and local deployment through NeMo-Speech.cpp are announced as coming soon, and audio question answering is trained in English only. NVIDIA itself recommends spot-checking names and speaker changes against the recording.

Sources

An open model is great. An open model plus the data pipeline and the fine-tuning recipe is even better, because it invites everyone to make it fit their own language and their own field.

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen

A 35B model on a Mac mini, using just 2.9 GiB of memory 🚀

There's something special about running a capable language model on hardware you own: no API bill, no data leaving the house, just a small box on your desk doing the thinking. A paper that's currently #2 on Hugging Face's trending list takes that idea further than I expected: it runs a 35-billion-parameter model on a Mac mini at about 20 tokens per second while using under 3 GiB of memory. The trick: leave the experts on the SSD The paper is called The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction , and the system is named Edge0 . It builds on a Mixture-of-Experts (MoE) model, Qwen3.6-35B-A3B. In an MoE, each token only uses a few "experts" per layer (here 4 out of 256 in each of 40 layers), so most of the weights sit idle at any given moment. Edge0 takes that literally. The expert weights (quantized to int4) stay on the SSD and are streamed in only when needed. The clever part is a small prerouter : a tiny trained he...
Meer lezen