Robots Learn to Predict the Future with LingBot-VA: A Leap in Autonomous Control Technology

Breaking News! Robots have learned to predict the future! This groundbreaking development comes from Ant Group’s LingBot, which has now released the world’s first causal video-action world model for general robot control, dubbed LingBot-VA. This is the fourth consecutive day of unveiling exciting advancements.

So, how does this work? Traditionally, robots, especially those based on VLA (Visual-Language-Action), operate primarily on a reflex basis: they see something and immediately react. This is referred to as an “observe-react” model. However, LingBot-VA takes a different approach. It leverages autoregressive video prediction to break away from this conventional thinking, allowing the robot to visualize future scenarios before executing actions. This imaginative decision-making process is quite novel in the realm of robotic control.

LingBot-VA boasts several impressive features:

• Memory Retention: When performing long sequences of tasks (like making breakfast), it retains knowledge of previous actions, showcasing strong situational awareness.
• Efficient Generalization: With only a few dozen demonstration samples, it can adapt to new tasks; it can also operate with different robot bodies without issues.

Equipped with LingBot-VA, robots can now easily handle high-precision tasks, such as cleaning small transparent test tubes.

As mentioned, today marks the fourth consecutive day of open-source releases from Ant Group’s LingBot. While previous releases have enhanced the robot’s vision (LingBot-Depth), brain (LingBot-VLA), and world simulator (LingBot-World), today’s release of LingBot-VA truly gives these entities a “soul”—a world model in action, translating imagination into execution. This advancement elevates the ceiling for general robots, as one observer noted: “From prediction to execution; this is a significant leap.”

LingBot-VA has taken a more advanced architectural approach. In the traditional VLA paradigm, models typically handle complex tasks like visual comprehension, physical reasoning, and low-level action control all within the same neural network, a phenomenon known as Representation Entanglement. To achieve higher sample efficiency and enhanced generalization, LingBot-VA unravels this complexity, proposing a new method: first envision the world, then derive actions. The Ant Group team employs a two-step strategy:

• Video World Model: Predict future visual states (what will happen next).
• Inverse Dynamics: Based on visual changes, deduce the necessary actions (how to move to achieve the desired visuals).

This stands in stark contrast to traditional VLA, which jumps directly from “now” to “action” without considering “future.” How is this achieved? The Ant Group team focuses on three key architectural breakthroughs.

First, the autoregressive interleaving of video and actions within LingBot-VA’s model incorporates video tokens and action tokens into a single temporal sequence. To ensure logical coherence, the team employs Causal Attention, which enforces a strict rule: only past information can be used—future data is off-limits. Additionally, with KV-cache technology, the model possesses remarkable long-term memory, clearly recalling actions from three steps prior without losing track.

Next, the Mixture-of-Transformers (MoT) strategy addresses the previously mentioned representation entanglement. This process can be likened to a synchronized “fight,” where:

• Video Stream: Wide and deep, responsible for heavy visual reasoning.
• Action Stream: Light and fast, focusing on precise motion control.

These two streams share an attention mechanism, allowing for information exchange while maintaining independence within their respective representation spaces. This way, the complexity of vision does not interfere with the precision of actions, and the simplicity of actions does not diminish the richness of visual input.

The final aspect involves engineering design. Theory alone is insufficient; “practice is the only criterion for testing truth.” Key engineering advancements include:

• Partial Denoising: For action prediction, the model doesn’t need to render future visuals in perfect clarity every time. It has learned to extract key information from noisy intermediate states, significantly improving computational efficiency.
• Asynchronous Inference: While the robot executes a current action, the model is simultaneously calculating the next steps in the background. This parallel reasoning and execution nearly eliminate delays.
• Grounding: To keep the model’s imagination grounded in reality, the system continuously calibrates its imaginings with real observational data, preventing hallucinations.

Now, let’s explore the experimental results that validate these capabilities. The Ant Group team has conducted comprehensive tests on LingBot-VA using both real machines and simulation benchmarks, covering three challenging task categories:

• Long-sequence tasks: Tasks like preparing breakfast (toasting bread, pouring water, plating) and unboxing (using a knife, cutting boxes, opening lids). These tasks are intricate, and any misstep can lead to failure. However, even if a mistake occurs, LingBot-VA retains its progress and attempts to retry.
• High-precision tasks: Tasks such as wiping test tubes and screwing bolts require millimeter-level precision. Thanks to the MoT architecture, the action stream remains unaffected by visual noise, resulting in extremely stable movements.
• Tasks involving deformable objects: Actions like folding clothes and pants are challenging due to the constantly changing state of the objects. However, LingBot-VA predicts the fabric’s deformation through video simulation, performing operations smoothly.

Moreover, LingBot-VA has excelled in the RoboTwin 2.0 and LIBERO hardcore simulation benchmarks. In the dual-arm collaboration tasks of RoboTwin 2.0, whether in simple fixed scenes (Easy) or complex random environments (Hard), LingBot-VA has demonstrated impressive capabilities:

• RoboTwin 2.0 (Easy): Success rate of 92.93%, surpassing the second place by 4.2%.
• RoboTwin 2.0 (Hard): Success rate of 91.55%, ahead of the second place by 4.6%.

A notable trend is that the harder the task and the longer the sequence (increasing horizon), the greater LingBot-VA’s advantage becomes, expanding to over 9% in long tasks with a horizon of 3. In the LIBERO benchmark test, LingBot-VA achieved an average success rate of 98.5%, setting a new state-of-the-art record.

In summary, these experiments clearly highlight three core attributes of LingBot-VA:

• Long-term memory: In a task involving repeatedly wiping plates, a typical VLA model may forget the number of wipes and start going off course; however, LingBot-VA accurately counts and stops after finishing.
• Few-shot adaptation: It requires only about 50 demonstration data points to learn new tasks, significantly more efficient than models that need thousands of data points.
• Generalization capability: Trained on one type of cup, it can still accurately recognize and manipulate cups of different shapes, colors, or positions.

Reflecting on the last four days of continuous releases, it is evident that Ant Group has laid out a strategic plan. When combined, these four open-source projects create a clear technological narrative:

• Day 1: LingBot-Depth – Addresses the issue of “seeing clearly” for enhanced perception.
• Day 2: LingBot-VLA – Solves the “connection” problem, linking language, vision, and action.
• Day 3: LingBot-World – Tackles the “understanding” problem by building a predictable and imaginable world model.
• Day 4: LingBot-VA – Resolves the “action” problem, integrating the world model into a control loop that allows imagination to guide actions.

These four components together emit a powerful signal: general robots are entering a video era. Video is no longer just a training data source; it is evolving into a medium for reasoning and a unified representation that connects perception, memory, physics, and action. This holds immense value for the entire industry. For general robots, long tasks, complex scenes, and unstructured environments—previously significant challenges—now have systematic solutions.

From the perspective of embodied intelligence, a world model is no longer optional; it has become a central capability for robots, evolving from mere action to thoughtful action. Moreover, Ant Group’s continuous open-source initiatives provide not just code and models but a reproducible and scalable technological paradigm. The ripple effects are already becoming evident in the industry. Recently, Google announced its Project Genie to allow more users to experience Genie 3, while Yushu Technology unveiled UnifoLM-VLA-0 as open-source. Furthermore, international media has taken notice of Ant Group’s open-source moves, commenting on the high-quality robot AI simulation environment named LingBot-World released by this Chinese fintech company, which has established a comprehensive open-source toolkit for developing physical AI systems. This represents a strategic initiative in the global competition for leadership in robotics.

In conclusion, the emergence of LingBot-VA signifies the first time a world model has truly taken center stage in robot control.

Project Address: https://technology.robbyant.com/lingbot-va

GitHub Address: https://github.com/robbyant/lingbot-va

Project Weight: https://huggingface.co/robbyant/lingbot-va, https://www.modelscope.cn/collections/Robbyant/LingBot-va