You can now talk to a robot while it works: LeRobot v0.7.0 puts language in the loop 🤖

Most robot demos I've watched follow the same script: you give the robot one instruction, press go, and hope. If you want it to do something else, you stop it, change the prompt and start over. Hugging Face's new LeRobot v0.7.0, released on 5 November, changes that loop in a way I find genuinely exciting. You can now type to a robot while it's moving, ask it what it sees, or hand it a goal and let it plan its own steps.

Talking to a running robot

The headline feature is interactive rollouts. Add --interactive=true to lerobot-rollout and the robot connection and policy stay loaded between instructions. From your terminal you can send /subtask pick up the red cube to change course mid-run, /vqa how many cubes are left on the table? to ask the policy what the cameras show, or /autosteer clear the table to let the policy choose its own subtasks until you take control back. You can even record episodes while doing it.

The questions and autosteering need a policy that can also generate text, such as FineART-VLA or Wall-OSS-0.5. If yours can't, LeRobot can point at any vision-language model behind an OpenAI-compatible API, served locally with something like vLLM, and use it as an external planner that turns "tidy up" into concrete instructions the action model understands. That split between a slow thinker and a fast doer is a pattern I keep seeing in agent design, and it's nice to see it this accessible for robots.

Big models, light robots

The part that speaks to my homelab heart is remote inference. With lerobot-policy-server, the computer next to the robot no longer has to fit the model in memory. A laptop or single-board computer handles cameras, motors and recording, while a GPU workstation across the room computes the next chunk of actions. The robot keeps following buffered actions while the server works, which hides much of the latency, and the client blends overlapping chunks so motion stays smooth.

Training on data you never download

There's also a lot here for people who train. The Hub now holds almost 300 TB of open Physical AI data in LeRobot format, and v0.7.0 can stream it straight from the Hub, a Storage Bucket or LanceDB, fetching only the video byte ranges training needs. The training stack was rebuilt with FSDP2, HSDP and distributed checkpoints for world action models in the 5 to 20 billion parameter range. On an 8×H200 node with a 6B model, the team reports checkpoints saving 3x faster and resume memory dropping from 143 GB to 42 GB per GPU. Four new world action models (including NVIDIA's Cosmos 3 and Black Forest Labs' FLUX 3 Action) and four new VLAs join the library too.

One small detail I really appreciate: lerobot-eval now reports a 95% Wilson confidence interval with every success rate, so "9 out of 10 episodes" shows up honestly as somewhere between about 60% and 98%. That's the kind of humility benchmarks need.

The honest fine print

Not every model in the zoo is fully open: the release notes flag that G0.5 ships under a non-commercial community license, so check each checkpoint before building on it. And the impressive LIBERO scores are simulation results; real kitchens are messier.

Sources

Being able to simply ask a robot "what do you see?" while it works feels like a small step that changes how we'll build with them. I'm curious what the community does with it first.

Reacties

Een reactie posten

Populaire posts van deze blog

Meta just shared the full recipe for a 30B agent model you can run at home 🧠

When Meta released Muse Glimmer in August, the pitch was simple: an open-weight, Apache 2.0 model built for AI agents that run on your own Mac or PC. What was missing was the "how". On 19 October, Meta Superintelligence Labs published the full technical report, and for anyone who cares about distillation and local agents, it is one of the most useful reads in a while. What the model is Muse Glimmer is a dense model of about 29.6 billion parameters, including a 1.8B perception encoder, so it reads text and images and writes text. It has a 128K+ context window and was trained on data from more than a hundred languages. The design goal was strict: everything has to fit in 32 GB of GPU memory, and a smaller 4-bit variant targets 24 GB cards. The parts I found most interesting Dense beat mixture-of-experts here. That surprised me. The team trained scaling ladders for a dense model and two MoE designs with the same memory footprint. Under a fixed memory ceiling and their compu...
Meer lezen

SpaceX just asked to fly Starship hundreds of times a year 🚀

Last week the U.S. Federal Aviation Administration took a quiet but important step: it opened a formal review of SpaceX's request to launch Starship from Starbase in South Texas up to 456 times per year . Today the site is cleared for only 25 flights a year. That jump is the clearest signal yet that Starship is meant to become routine hardware, not a rare test rocket. What the FAA is reviewing On 14 October 2026 the FAA published a Notice of Intent to prepare a Programmatic Environmental Impact Statement (PEIS) for SpaceX's Starbase plans. A PEIS looks at a whole program of operations rather than a single launch, which is exactly what you need if you want a reusable rocket to fly the way airlines fly planes. SpaceX's proposal comes in two phases, according to the Federal Register notice: Phase 1: up to 100 launches and 200 landings per year (100 Ship, 100 Super Heavy), any day or time of week. Phase 2: up to 456 launches and 912 landings per year at Starbase. La...
Meer lezen

AI that can actually "see" a crystal 🔬

Most of the solid stuff around us, from battery materials to chips, is made of crystals: atoms stacked in neat, repeating 3D patterns. Large language models are great at reading papers about those materials, but they have never really looked at the structures themselves. A team at MIT just showed a clever way around that, and the trick is one I keep coming back to: good synthetic data. What they built The model is called CLaC, short for Contrastive Language-Crystal model, and it is described in a paper in Nature Communications by Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. MIT News covered it on 18 October 2026. If you know CLIP, the model that taught image generators to connect pictures and words, you already get the idea. CLaC has two branches: a graph neural network that reads the 3D crystal structure, and a text encoder that reads a description. Contrastive training pulls matching structure-text pairs together in one shared space and pushes mismatche...
Meer lezen