Robots Learn to “Dream”: Hong Kong Universities Develop Intelligent Robots with Imagination能力

Robots Have Learned to “Dream”: A Collaborative Effort by Top Universities in Hong Kong and Beyond to Develop Intelligent Robots with Imagination

This research, jointly conducted by the University of Hong Kong, Hong Kong University, Tsinghua University, and several other prestigious institutions, successfully developed the RISE system, enabling robots to “dream” and practice in virtual environments for the first time. By utilizing a combinatorial world model, robots can safely train their skills in various challenging tasks, achieving success rates of 85%, 85%, and 95% in dynamic block sorting, backpack packing, and box sealing, respectively. This marks a significant breakthrough compared to traditional methods and opens up new pathways for the advancement of robotic intelligence.

The Dreaming Mind of Robots: The Secret of the Combinatorial World Model

The core of the RISE system is an intelligent brain known as the “combinatorial world model,” which functions like a dedicated dream factory for robots. This factory consists of two sophisticated departments, each responsible for different tasks.

The first department, called the “Dynamic Prediction Department,” operates like a skilled animator. When a robot wants to attempt a certain action, this department quickly sketches potential future scenarios based on the current environment and planned actions. For instance, when a robot is preparing to grab a moving block, the dynamic prediction department simulates the block’s trajectory, the movement path of the robotic arm, and various possible outcomes of the attempt.

This prediction system is highly efficient. While traditional simulation systems may take 10 minutes to generate 25 frames of multi-view predictions, RISE’s dynamic prediction department accomplishes the same task in under 2 seconds, enhancing efficiency by 300 times. This remarkable speed allows robots to engage in a significant number of virtual practice sessions, similar to an athlete performing thousands of mental rehearsals in a short period.

The second department is called the “Value Assessment Department,” which functions like an experienced coach. After the dynamic prediction department generates various potential future scenarios, the value assessment department carefully analyzes each one to determine which actions are beneficial and which are not. This department provides not only simple success or failure judgments but also detailed scoring, much like judges in a gymnastics competition.

The training process for the value assessment department is particularly interesting. The research team taught it two different evaluation methods. The first is called “Progress Estimation,” which gradually increases scores over time, similar to observing a student’s homework progress. The second is “Temporal Difference Learning,” which captures subtle errors during the action process. By combining these two methods, the value assessment department maintains scoring stability while being sensitive to issues.

What’s fascinating is that these two departments do not operate independently; they work closely together. The dynamic prediction department generates imagined scenarios, the value assessment department scores them, and the results are fed back into the robot’s decision-making system. This interaction creates a complete learning cycle: imagine, evaluate, improve, and then repeat, continually enhancing the robot’s skills.

From “Novice” to “Expert”: The Robot’s Learning Progression

The training process of the RISE system resembles the development of a skill master from amateur to professional, divided into two crucial stages, each with its unique learning approach and objectives.

The first stage is known as “Policy Warm-Up,” akin to a basic training camp for robots. During this phase, robots learn fundamental operational skills, much like a person learning to drive must first become familiar with the steering wheel and brakes. The research team provides robots with numerous real-world operation videos, including expert demonstrations, success cases, and failure cases, as well as human correction processes.

In this stage, robots acquire a special skill: adjusting their behavior based on “advantage cues.” This is like giving robots an internal voice that tells them, “This action is good, keep it up,” or “This could lead to problems, be cautious.” When robots receive high advantage signals, they execute actions with greater confidence; when they receive low advantage signals, they become more cautious or choose alternative strategies.

The second stage is when the RISE system truly shines, known as the “Self-Improvement Cycle.” During this phase, robots begin extensive “dream training” in virtual worlds. The process resembles an unending learning loop that includes two alternating steps.

In the “Virtual Practice” step, robots start from real-world states and then imagine various actions. The system provides the robot’s virtual avatar with optimized behavior instructions, allowing it to perform at its best in the dream state. The combinatorial world model generates potential future scenarios from these actions and rates each one in detail. This process is akin to allowing the robot to mentally rehearse thousands of times, receiving detailed feedback each time.

To ensure diversity in training, the system uses these imagined scenarios as new starting points for deeper virtual practice. However, the research team found that consecutive virtual practices should not exceed two rounds, as accumulating prediction errors could impact training effectiveness, similar to how a game of telephone can distort messages over multiple relays.

In the “Skill Upgrade” step, the robots analyze all these virtual experiences to learn how to execute tasks better in the real world. The system uses high-scoring actions from virtual practice as learning targets, enabling robots to make better choices when faced with similar situations. Additionally, to prevent robots from forgetting previously learned basic skills, the system regularly reviews real-world operational experiences.

The advantages of this learning method are evident. Traditional robot training requires thousands of attempts in real-world environments, with each failure posing risks of equipment damage or safety issues. The RISE system allows robots to practice extensively in a safe virtual environment, avoiding real-world risks while significantly improving learning efficiency.

Real-World Testing: Outstanding Performance in Three Challenging Tasks

To validate the practical effectiveness of the RISE system, the research team designed three challenging real-world tasks, each testing different aspects of the robot’s capabilities. The complexity of these tasks was daunting for traditional robotic systems, yet the RISE system demonstrated remarkable improvements across all of them.

The first task was “Dynamic Block Sorting,” akin to having robots play a high-difficulty grabbing game on a fast-moving conveyor belt. Blocks move at varying speeds, and the robot must accurately identify the color of each block and place them into corresponding colored bins. This task tests not only the robot’s visual recognition abilities but also its precision in tracking and grabbing dynamic targets. In this task, the RISE system achieved an 85% success rate, a significant improvement from the basic system’s 35%, raising the success rate by 50 percentage points.

The second task was “Backpack Packing,” simulating the process of organizing luggage in daily life. The robot must open a soft backpack, stuff clothes inside, lift the backpack to allow the clothes to settle at the bottom, and finally zip it closed. This task particularly challenges the robot’s capability to handle soft, deformable objects since backpacks and clothes are not rigid and their shapes change during the operation. The RISE system excelled in this task, with a success rate rising from 30% using traditional methods to 85%, an increase of 55 percentage points.

The third task was “Box Sealing,” requiring precise coordination of both robotic arms. The robot needs to place a cup into a box, fold down the side flaps sequentially, and accurately insert the locking mechanism into the slot to complete the sealing process. The entire operation requires precise coordination between the two robotic arms, where any deviation at any step could lead to failure. The RISE system showcased an astonishing 95% success rate in this most challenging task, achieving a significant increase of 60 percentage points compared to the basic method’s 35% success rate.

The implications of these experimental results extend beyond mere numbers. The research team discovered that the RISE system not only significantly improved success rates but also demonstrated better adaptability and robustness. Traditional robotic systems often fail in the face of slight environmental changes or unexpected situations, while robots trained with the RISE system are better equipped to handle such uncertainties.

In-Depth Analysis: Key Elements Behind the Success of the RISE System

The outstanding performance of the RISE system is attributed to several meticulously designed key elements, each validated and optimized through extensive experimentation.

The first is the “Task-Centric Batch Processing” strategy. During the training of the combinatorial world model, the research team identified a crucial issue: mixing data from different tasks and environments for training significantly reduces the model’s learning effectiveness. This is akin to having a student study math, language, and history simultaneously, which can scatter attention and reduce learning efficiency.

To address this, the research team employed a clever training strategy. They had the model focus on the same type of task data in each training batch, ensuring that this data included various combinations of actions relevant to that task. This method is comparable to allowing a student to concentrate on math but encounter a variety of math problems, thus maintaining focus while ensuring comprehensive learning. Experiments showed that this strategy not only sped up the model’s convergence but also significantly enhanced the operational capabilities of the trained robots.

The second key element is the precise tuning of the “Offline Data Mixing Ratio.” In the robot’s learning process, it is crucial to balance two different types of experiences: offline data collected from the real world and online data generated in virtual environments. The research team found that the mixing ratio of these two data types significantly impacts the final outcome.

Through extensive experiments, they determined that the optimal mixing ratio is 60% offline data combined with 40% online data. When the offline data ratio is too low (e.g., only 10%), robots are prone to forgetting basic operational skills, leading to a significant drop in success rates for simple tasks. Conversely, when the offline data ratio is too high (e.g., reaching 90%), robots become overly cautious and fail to fully utilize new skills learned from virtual practice. This 60:40 ratio ensures that robots maintain stable foundational abilities while continually learning and improving.

The third crucial element is the design of the “Advantage Conditioning” mechanism. This mechanism allows robots to adjust their behavioral strategies based on the quality of the current situation. The research team categorized possible advantage values into ten levels, akin to providing robots with a “confidence index” ranging from 1 to 10. When robots receive high-level advantage signals, they proactively execute complex operations; when they receive low-level signals, they adopt more conservative strategies.

This design cleverly mimics the decision-making process of human experts. Experienced operators choose different operational strategies based on the complexity of the current situation and their level of confidence when facing difficult tasks. The RISE system successfully incorporates this human wisdom into the robot’s decision-making process.

The final key element is the “Dual Value Learning” method. Traditional robot training usually employs only one evaluation method, either focusing on overall task progress or specific successes and failures. The RISE system innovatively combines two complementary learning methods: progress estimation and temporal difference learning.

The progress estimation method enables robots to grasp the overall context and timing of tasks, essentially providing them with an internal schedule. Temporal difference learning keeps robots sensitive to subtle changes during the operational process, allowing them to promptly identify and correct minor deviations. The combination of these methods ensures that robots maintain a clear understanding of larger goals while making precise adjustments during execution.

Pushing Boundaries: Technical Innovations and Limitations of the RISE System

While the RISE system has achieved significant technical breakthroughs, the research team also candidly acknowledged some limitations of the current system, providing direction for future improvements.

In terms of technical innovation, the RISE system’s greatest achievement lies in successfully introducing imaginative capabilities into robot learning. Traditional robot training relies on repeated trial and error in real-world environments—a method that is not only costly but also poses safety risks. The RISE system constructs high-quality virtual environments that allow robots to practice extensively in a safe imagined space, akin to providing them with a dedicated training simulator.

The effectiveness of this imaginative training is significant. For instance, in the dynamic block sorting task, a robot equipped with the RISE system completed the equivalent of tens of thousands of grabbing practices in a virtual environment. Conducting such a volume of practice in the real world would take months and result in substantial costs. Through virtual training, the entire process can be completed in just a few days.

Another important innovation is the design concept of a “Hierarchical Architecture.” The RISE system decomposes complex robot learning problems into two relatively independent yet closely connected modules: a dynamic model responsible for predicting future scenarios, and a value model that evaluates behaviors. This decomposition not only simplifies understanding and debugging of the system but, more importantly, allows each module to adopt the most suitable technological solution.

The dynamic model employs advanced video generation technology to quickly produce high-quality future scenario predictions, while the value model is based on the architecture of large-scale language models, benefiting from their advantages in complex reasoning. This specialization in design ensures the efficient operation of the entire system.

However, the RISE system also faces some current limitations that cannot be entirely overcome. The primary constraint arises from the gap between the virtual and real worlds. Although the RISE world model is quite advanced, the virtual scenarios it generates cannot fully replicate the complexity of reality. This is particularly true when dealing with rare or extreme situations, where the effectiveness of virtual training may diminish.

The research team found that when encountering scenes that appear infrequently in training data, the world model occasionally generates predictions that are physically implausible. For example, when simulating the deformation of soft objects, the model may sometimes produce results that violate the laws of physics. While such occurrences are not common, they do limit the system’s reliability in certain extreme cases.

Another limitation is the demand for computational resources. While the RISE system significantly reduces costs compared to direct real-world training, it still requires high computational capabilities. Training a complete RISE system necessitates multiple high-performance GPUs over several days to a week, posing a challenge for some resource-limited research institutions or companies.

Additionally, there is room for improvement in the RISE system’s handling of tasks requiring long-sequence reasoning. Although the system has excelled in current testing tasks, these tasks have relatively short time spans. The system’s performance in complex tasks requiring several minutes or longer to complete still needs further validation.

Lastly, the RISE system is primarily optimized for operational tasks, and its applicability to tasks requiring complex language understanding or multimodal reasoning remains to be tested. While the foundational architecture of the system supports these expansions, specific implementations and optimizations will require significant additional research efforts.

Despite these limitations, the research team remains confident about the future of the RISE system. They believe that as computational technologies continue to advance and the accuracy of world models improves, these current limitations will gradually be addressed. More importantly, the RISE system has demonstrated the feasibility of training robots through imagination, paving a new developmental path for the entire field.

Ultimately, the RISE system represents a significant breakthrough in the realm of robotic learning. It not only achieves multiple innovations on a technical level but also presents a new possibility for robots to acquire human-like intelligence. By enabling robots to learn to “dream” and “imagine,” we are witnessing artificial intelligence move towards greater intelligence and autonomy. The significance of this research lies not only in its current achievements but also in its guidance for the future development of robotic technology. As technology continues to improve, we can anticipate that intelligent robots equipped with imaginative capabilities will soon become indispensable partners and assistants in our lives.

Q&A

Q1: How does the RISE system enable robots to learn imagination?
A: The RISE system equips robots with imaginative capabilities by constructing a “combinatorial world model.” This model consists of two core components: the dynamic prediction department generates potential future scenarios based on the current environment and planned actions, while the value assessment department scores these imagined scenarios. Robots learn and improve their skills by repeatedly practicing in these virtual scenarios.

Q2: How much has the training efficiency of the RISE system improved compared to traditional methods?
A: The RISE system has achieved a tremendous boost in training efficiency. In generating predictions, RISE completes tasks in just 2 seconds that would take traditional systems 10 minutes, leading to a 300-fold increase in efficiency. In practical task performance, RISE achieved success rates of 85%, 85%, and 95% in dynamic block sorting, backpack packing, and box sealing tasks, respectively, representing increases of 50, 55, and 60 percentage points compared to basic methods.

Q3: What complex tasks can robots trained with the RISE system handle?
A: Robots trained with the RISE system can tackle various high-difficulty operational tasks, including accurately grabbing and sorting colored blocks on a moving conveyor belt, packing operations involving soft deformable backpacks and clothing, and box sealing tasks requiring precise coordination of both arms. These tasks necessitate dynamic adaptability, precise operation, and complex reasoning abilities that far exceed the capacities of traditional robots.