The Challenges of Robot Data Collection: Bridging the Gap to Intelligent Automation

Robots that are taking the internet by storm are still trapped in the “data pipeline.”

In the past year, captivating robot action clips featuring backflips, dancing, boxing, and even kicking watermelons have gone viral. The industry is electrified, capital is flowing in rapidly, and public expectations have soared: mature robotic products seem to be swiftly transitioning from the laboratory to reality. However, within the so-called “schools” for robots known as data training centers, the atmosphere is much quieter. Data collectors, armed with operational devices, guide nearby robots to complete seemingly simple tasks, such as picking up parts from a table, placing them in a toolbox, and closing the lid—all at a slow pace, occasionally pausing.

This is merely the first step of “learning.” Each time a robot completes a set of actions, it generates structured data. Feeding this data into large models for training requires a substantial amount to give robots a “brain,” enabling them to move from passive programming to active understanding and decision-making. According to industry insiders, this transition represents the “difference between monkeys and humans.”

The logic of “data + computing power + algorithms” is familiar to us. Large language models, represented by ChatGPT and DeepSeek, have validated its feasibility and established a relatively mature ecosystem of computing resources and algorithms. However, the challenge for robots lies in shifting this intelligence from the digital realm to the physical world, where data acts as the greatest barrier. The language and image data used by large language models exist in a two-dimensional digital world, making them easy to acquire and replicate. In contrast, robots operate in a three-dimensional physical world, which is a high-dimensional, continuous, multimodal spatiotemporal flow that includes a variety of sensory signals such as vision, sound, force, torque, and body posture, significantly increasing the complexity of data processing.

If the vast amounts of data accumulated from years on the internet have provided ample resources for the development of large language models, the data collection and accumulation in the physical world often require starting from scratch. Within this context, “real machine data” that captures the original operational data of robots in real physical environments has become a recognized rarity and treasure in the industry.

Over the past year, robot data collection centers have flourished, positioned as “infrastructure.” The rather dry, yet crucial, data collection scenes described earlier are being enacted across the country. However, true machine training demands immense time and capital investment. The construction of data centers is a one-way street; amidst the fervor, it is essential to think critically about what constitutes “high-quality” data, how to efficiently circulate and reuse the training data, and how to pragmatically advance before bridging the data gap. Before the comprehensive launch of the “new infrastructure” of the robotic era, exploring and resolving these issues will determine whether “embodied intelligence” becomes a solid industrial upgrade or yet another overhyped concept.

1. Data Collection: Craftsmanship in Detail

At the centralized training area of the Beijing Humanoid Robot Data Training Center, visitors can witness how robots “learn” through transparent glass. Data collectors wear gloves on collection devices, allowing their hand movements to be transmitted to adjacent robots, guiding them to pick up pliers from a table, place them in a toolbox, and repeat the process. Simple tasks such as grabbing, taking, and placing are conducted in this tabletop training environment.

As one looks further, the scene becomes more complex. A self-service supermarket filled with products, a living room scattered with books, and a bedroom and bathroom piled with clothes and towels replicate a high-fidelity environment where humans can move freely while robots must complete tasks like arranging items and folding laundry in this multifaceted and realistic setting. The goal is singular: to collect high-quality true machine data in bulk, which is the core objective of all data centers.

Currently, there is no unified data standard in the robotics industry; different data collection centers often have their own methods of data representation and format requirements, which can lead to diverging paths even from the inception of data center construction. The operational entity of the Beijing Humanoid Robot Data Training Center is Ruiman Intelligent Technology (Beijing) Co., Ltd., a company focused on robotic arm development. Ruiman places particular emphasis on hardware requirements across various dimensions of data evaluation. A representative from Ruiman explained that the data center requires high-precision calibration for each hardware component, including absolute motion accuracy and camera-related parameters. All robots are equipped with high-precision sensors capable of collecting data across 57 different dimensions.

Another significant hardware challenge arises from temporal alignment. Specifically, the cameras used for data collection typically operate at a sampling frequency of 30Hz, capturing 30 images per second, with each frame taken approximately 33 milliseconds apart. If the timing is not aligned, a discrepancy of 33 milliseconds can lead to joint encoders, cameras, and force sensors capturing “fragments of the world” from different moments. Since model training relies on strict causal relationships, even millisecond-level desynchronization can lead to severe misalignment, resulting in significant errors. Ruiman employs a hardware synchronization strategy during data collection, ensuring that sensor data and camera data are captured according to actual physical timing, achieving an error margin of less than 1 millisecond.

Based on high-precision hardware calibration and temporal alignment, a diverse matrix system is implemented to ensure the variety of scene items and the generalizability of robot positioning and posture, preventing data overfitting that could negatively impact model performance. After rigorous verification of data credibility, a high-quality true machine data set is deemed complete. A representative from Ruiman stated, “For robots that truly enter households, the physical joints must be sufficiently stable and reliable, easy to use, and able to maximize load capacity in the smallest size possible. In terms of AI, data dimensions are crucial. We believe that true machine data is the final hurdle for robots to enter homes, so we are committed to providing such data assets from the very end.” Currently, the Beijing Humanoid Robot Data Training Center has achieved large-scale output, generating approximately 60,000 data entries daily, covering 16 sub-scenarios across four major fields: industrial intelligence, smart homes, elder care services, and 5G integration.

2. The Gap in Data and the Chasm of Data Heterogeneity

According to technical market research firm Interact Analysis, by the end of 2025, more than 50 humanoid robot data collection and training centers at the national or provincial level in China will be in operation or under construction, with over 50% set to be officially operational by 2025. Taking the Beijing Humanoid Robot Data Training Center as a reference, its annual production capacity of true machine data has reached millions. By rough estimation, if all data centers are fully operational, the annual data collection for robots could reach billions of entries. However, even this seemingly massive data supply falls short of the requirements for the “intelligence” that robots need.

A conservative estimate by robotics data service provider Mite Technology suggests that under optimal conditions, where embodied intelligent large models are sufficiently advanced and data quality is high, training a robot to learn a single action requires approximately 1,000 to 5,000 data entries; training a robot to learn a task composed of multiple actions requires about 10,000 to 20,000 entries; teaching a robot to perform 80% of human tasks in a specific industry necessitates at least 100 million entries; and scaling embodied intelligence to general applications across all industries demands data volumes in the hundreds of billions, creating a gap of four to five orders of magnitude.

An even larger divide lies in data heterogeneity. Different manufacturers and robot types differ in hardware design, sensor configuration, and software protocols, leading to “language barriers” among collected motion, tactile, and visual data. Data generated from one type of robot may not function properly on another. This means that the training outcomes of various data centers may struggle to achieve a simple additive effect of 1+1=2. Until a common industry standard emerges, data centers are exploring various solutions. One approach is to “mask differences” by utilizing widely adopted robotic models for data training, thereby avoiding compatibility issues at the hardware level, as seen in the previously mentioned Beijing Humanoid Robot Data Training Center. Another approach is to “embrace differences,” actively conducting heterogeneous training.

In Zhangjiang, Shanghai, the National-Local Joint Innovation Center for Humanoid Robots has pioneered a method for constructing heterogeneous humanoid robot embodied intelligence data sets, aiming to create the largest-scale heterogeneous humanoid robot data set. Here, robots from different manufacturers operate collaboratively within the same physical space. Chief Scientist Jiang Lei stated in a media interview, “By placing heterogeneous robots from different manufacturers in the same environment, AI can recognize that it exists in a diverse physical world, thereby establishing objective awareness and developing the ability to discern right from wrong.”

A third technical route is to directly “bypass differences” by seeking broader and more universal data. Unlike hardware-collected data from joint sensors, human video data is relatively universal for robots, as it can extract human poses from videos and map them as robot motion trajectories, bypassing intrinsic barriers to train large models. The Beijing Humanoid Robot Data Training Center’s visual motion capture project adopts an even more radical approach by entering the realm of simulation. In a virtual digital environment, massive data can be generated cost-effectively through physical engines and program simulations, which can then be applied to real machines, achieving Sim2Real. However, the extreme complexity of the physical world fundamentally determines that simulated data struggles to achieve ideal levels of accuracy and generalization.

“We hope to find a balance between reality and simulation, benefiting from both,” the CEO of Mite Technology explained, introducing their Real2Sim2Real data collection model: “By adding ‘Human Doing Video’ before the virtual environment, we use 2D video data from real-world human operations to perform 3D reconstruction, simulating human 3D poses, and retargeting these poses to robots, hence the name Real2Sim2Real.” The goal of this method is to reduce the cost of individual data entries from several dozen yuan for true machine data to mere cents and quickly distribute affordable collection devices across various industries to obtain massive data.

3. Working While Optimizing

While various technical paths such as the integration of reality and simulation are still being explored, one undeniable fact remains: no matter the proportion, true machine data is the “last mile” for aligning robots with the physical world. Therefore, the core challenge facing data training centers is not merely pursuing data scale but producing high-quality data that meets current industrial application demands. In Wuxi, this logic is being materialized through the “Jiangsu Provincial Embodied Intelligent Robotics Industrial Data Collection and Training Center,” led by Tianqi Automation Engineering Co., Ltd. This center has moved away from the “model room” approach, closely replicating seven major training scenarios including automotive manufacturing, new energy production lines, and industrial logistics handling.

Tianqi’s chief algorithm scientist, Tong Suibing, noted that automotive assembly is a traditional business for Tianqi, with a vast customer base and profound industry understanding of automotive production lines. In automotive manufacturing, body painting is one of the core processes. After completing the electrophoretic primer, the surface paint application is critical since the uniformity and integrity of the paint directly affect vehicle quality. Traditionally, this quality inspection process heavily relies on human observation, but the spray painting environment is filled with volatile chemicals, posing health risks for workers over prolonged exposure. Automating inspection and flaw detection with robots not only liberates workers from harmful exposure but also allows for more stable and traceable quality inspections.

Tong Suibing believes that a more reasonable approach for embodied intelligent robots is not to design a universal robot for all industries and job types but to create robots tailored to individual needs. In this context, the Jiangsu Provincial Embodied Intelligent Robotics Industrial Data Collection and Training Center has constructed a closed-loop system of “scenario-data-model-application,” focusing on existing business scenarios, accurately collecting robot data from those scenarios, training proprietary embodied intelligent large models using the collected data, and deploying the trained models back into the corresponding production environments for real-world verification and iteration. The real-world scenarios not only serve as a “test stone” for data and model effectiveness but also hold promise as sources of high-quality data.

At CES 2026, Ruiman completed a cross-ocean real-time operational demonstration from “Beijing to Las Vegas.” By building a remote labor network, embodied trainers in Beijing could control the RealBOT wheeled folding robot stationed at the CES venue to execute tasks like “delivering items” and “passing fruits” in a real operational context. This not only addresses specific labor needs in particular scenarios but is crucial for allowing robots to accumulate data directly within real workflows. Each remote operation generates data encompassing environmental interactions, human decision-making, and task results, achieving “work as data collection.” This suggests that future data factories might not need to completely replicate scenarios but can directly connect to global production lines and service terminals, enabling data to naturally settle during real operations.

4. A More Complex Endurance Race

Similar to its “infrastructure” role, the humanoid robot data training center is far more complex than a simple intelligent computing center. It cannot rely on mere “stacking” but must be a new type of infrastructure driven by data, integrated hardware and software, and closed-loop scenarios. According to Dr. Zhang Xiaoyu, an industry expert, evaluating the future potential of a data center hinges on its “heterogeneous data closed-loop capability.” This can be broken down into three key questions: first, can physical scenarios like factories, warehouses, and laboratories be easily connected to the data center through standardized interfaces to form continuous “data pulses”? Second, can a complete technical pipeline be established that encompasses multimodal data collection, cloud-based labeling and training, and model deployment back to robots, allowing data to be reused across different robotic bodies like software? Third, does it possess a robust simulation platform capable of generating vast synthetic data based on limited real data for safe, low-cost “million times testing” to accelerate iteration?

Beyond technology, Zhang Xiaoyu asserts that data centers also depend on the industrial soil in which they are rooted, requiring clear leading industries to act as demand engines to generate valuable data. “The importance of high-quality data sets for model training is self-evident, but from the economic perspective of data collection fields, it is unreasonable to build a separate data collection site for each robot brand. The most rational approach would be to establish one data collection site in a city with a concentration of industrial or academic institutions, where the collected, labeled, and cleaned high-quality data sets can be used by multiple robot manufacturers, thus achieving a ‘one-time investment for repeated use’ leverage effect.”

The establishment of robot data centers resembles large-scale ecological construction, requiring policy support, regulatory environments, and talent development to be effective. Beyond data training, it is hoped to attract enterprise clusters, facilitate industry model iterations, and accelerate the collective development of robotics companies. Ultimately, the significance of this infrastructure lies in enabling high-quality robot data to flow smoothly to every algorithm and robotics company that needs it. To this end, the industry has begun exploring diversified data transaction and application models. In August 2025, the Pasini embodied intelligence super data factory product “OmniSharing DB Pasini Fully Modal Embodied Intelligence Data Set” was officially listed at the Beijing International Big Data Exchange. In October, Pasini entered a strategic partnership with Tencent Cloud to jointly build an embodied intelligence “Data Cloud Mall.” Tianqi will also construct a data platform based on the “Jiangsu Provincial Embodied Intelligent Robotics Industrial Data Collection and Training Center,” allowing robot data to function as a foundational resource for the entire industry, much like today’s cloud resources.

Within the context of understanding robot data collection and industry implementation, a frequently mentioned reference case is intelligent driving. There is a clear consensus within the industry: the path for intelligent driving is relatively straightforward—following established road networks and relying on highly mature automotive and sensor hardware, with the core task simplified to reliable perception and decision-making within structured environments, primarily focusing on “avoiding collisions.” Even so, this technology has taken over a decade to develop, and commercialization has only recently reached the threshold of L3-level assisted driving, beginning limited testing. In contrast, the deployment difficulty of embodied intelligent robots increases exponentially. The rapid construction of data centers addresses more the issue of scaling “training materials,” but the completeness of this “textbook” and whether the robot’s “brain” and “body” can learn and apply efficiently remain unresolved questions. The story of intelligent driving underscores that the journey from lab demonstrations to stable, reliable, and economically viable commercial products traverses a “valley of death” far longer than anticipated. For robotics, this endurance race across the cycle has only just begun.