The quest to unlock the full potential of physical artificial intelligence is unfolding in a nondescript warehouse in San Leandro, California, where the intricate act of pulling wooden blocks from a tottering Jenga tower symbolizes a monumental challenge. This is the operational hub of Encord, a company that has strategically pivoted from merely providing data tooling for AI models to actively manufacturing the scarce, high-fidelity training data essential for the next generation of humanoid and warehouse robotics. The scene, while seemingly mundane, represents the bleeding edge of robotics development, where human dexterity and even neural activity are being harnessed to teach machines how to interact with the physical world.
The Data Bottleneck for Physical AI
The rapid advancements witnessed in large language models (LLMs) have been predicated on the availability of vast, easily accessible textual data — essentially, the entire internet. However, translating this success to the physical realm, where robots must navigate, perceive, and manipulate objects in dynamic, unpredictable environments, has consistently hit a fundamental wall: the severe scarcity of relevant, real-world training data. Unlike digital text, physical interaction data cannot be simply scraped from the web. It must be meticulously collected, often through laborious, real-world demonstrations or complex simulations, making it an incredibly expensive and time-consuming endeavor.
Industry experts widely concur that the primary constraint on the evolution of advanced robotics, particularly for general-purpose humanoid and sophisticated warehouse automation systems, is no longer solely about breakthroughs in model architecture. Instead, it is the sheer volume and quality of physical training data that dictates progress. While self-driving car companies have invested heavily in collecting their own proprietary datasets, this approach is difficult to scale across the myriad of tasks and environments that modern robots are expected to master. Training from video, while a viable initial step, often lacks the granular fidelity and multi-modal context necessary for robust physical manipulation. Vineeth Velmurugan, Encord’s head of robot learning and a veteran of OpenAI’s robot lab and warehouse automation firm Berkshire Grey, estimates that a breakthrough in physical AI might require a dataset five times the size of YouTube’s entire video corpus. This staggering scale underscores why data generation itself has evolved from a research problem into a burgeoning industry.
Encord’s Strategic Pivot: From Management to Manufacturing
Founded initially to assist companies with machine-vision data annotation and model evaluation, Encord observed a critical shift among its clientele. As leading robotics firms began exploring end-to-end learning for complex robotic manipulation tasks, they encountered the stark reality that the necessary training data simply did not exist. This realization prompted Encord to undergo a strategic transformation, moving beyond data management to actively creating the data its customers desperately needed. Velmurugan, who joined Encord specifically to build this internal data-creation team, articulates the problem succinctly: "The data simply does not exist." This pivot positions Encord at the forefront of a new wave of AI infrastructure providers, addressing a foundational need that is pivotal for the widespread adoption of advanced robotics.
The company’s San Leandro facility serves as a dynamic laboratory for experimenting with novel data modalities and collecting specialized datasets. Here, a diverse range of tasks, from the delicate act of stacking poker chips to the complex challenge of pouring coffee without spillage, are meticulously executed by human "pilots" — Encord’s term for its robotic trainers. These tasks are designed to capture the nuances of human dexterity and problem-solving, which are then translated into structured data for training AI models. The facility is equipped with an array of props, from fake flowers and plastic vegetables to kitty litter trays and bundles of wires, simulating the varied objects robots might encounter in household or industrial settings.
The Bleeding Edge: Brainwave-Tagged Data with Zander Labs
One of Encord’s most innovative collaborations involves Zander Labs, a German neuroscience startup. This partnership explores the use of brainwave headsets to capture mental states during human demonstrations. Andrew Ceja, an Encord pilot, exemplifies this pioneering approach. As he carefully disassembles a Jenga tower, his actions are captured by a headset camera, a common practice for collecting robot training data. However, Ceja’s headset is also equipped with sensors that measure his brain waves. Zander Labs’ hypothesis is that by measuring brain activity, they can deduce crucial mental states such as error, intent, and surprise, thereby enriching the training dataset.
Lucas Gehrke, a neuroscientist supervising the work for Zander Labs, explains that the intensity of brain activity during a task can offer vital clues for model builders. For instance, moments of high cognitive load or perceived error could signal to a robot’s AI when to deploy its highest-effort models or when to exercise extreme caution. This trial run with Zander Labs is currently focused on building an initial brainwave-tagged dataset. The next critical step involves running this data through customer robotics models to evaluate whether it genuinely improves performance before any decision is made to scale up the initiative. This careful, evidence-based approach underscores the scientific rigor applied to these novel data collection methods. The potential implications are profound: if successful, understanding human mental states during task execution could significantly accelerate robot learning, enabling more intuitive and adaptable robotic behaviors.
Expanding the Modality Spectrum: Egocentric and Somatic Data
Beyond brainwave analysis, Encord is exploring several other advanced data modalities to overcome the limitations of traditional video capture. One key method involves "egocentric" video, collected by workers wearing cameras that capture their perspective during tasks. This data is often augmented with additional camera angles and other metrics to provide a comprehensive view of the interaction. Encord draws egocentric data from factories globally, complementing its San Leandro experiments.
Another sophisticated technique involves "leader-follower rigs." These setups feature paired robotic arms, where one arm is directly controlled by a human operator, and the other precisely mimics its movements. This allows for the collection of high-fidelity data on complex manipulation tasks, directly translating human motor skills into robot-understandable trajectories. Sofia Infante, another Encord pilot, demonstrates this by maneuvering robotic arms to plug and unplug Ethernet cables from the back of a server. This task, critical for data center automation, highlights the current limitations of robotic dexterity; as a TechCrunch visitor noted, robot pincers lack the precision and degrees of freedom inherent in human fingers.
Encord is also developing a new data modality using sensors strapped to the forearm to detect electrical signals in muscles. Traditional video of human hands manipulating objects often fails to capture the full complexity of hand movements. By leveraging arm sensors, Velmurugan hopes to construct a 3D depiction of the hand’s position and orientation at any given moment, creating a far more robust understanding of human manipulation for AI models. This somatic data could provide an unprecedented level of detail regarding grip force, subtle adjustments, and overall motor control, which are crucial for delicate or precise robotic tasks.
The Economics of Data Manufacturing
All of Encord’s datasets are meticulously annotated with rich, semantic descriptions. For instance, a video might be tagged with "right hand tightens bolt," providing LLM-based models with a deep contextual understanding of the action. Velmurugan estimates that this kind of dense, high-quality annotation is exponentially more valuable—potentially 100 times as much—than raw, unrefined "junky ego data" for training specific tasks. Crucially, he also notes that while it is more expensive to produce, it only costs approximately 20 times more, presenting a highly favorable trade-off on paper.
However, this "20 times more" still represents a significant financial investment, underscoring a fundamental divergence between the development pathways of LLMs and physical AI. The foundational data for LLMs, scraped from the internet, came at virtually no cost to frontier labs. In stark contrast, generating physical training data is an inherently costly endeavor. It requires specialized equipment, controlled environments, and skilled human operators. This changes the entire economic landscape of building physical AI models, moving from a model of free data acquisition to one of active, resource-intensive data manufacturing. This economic reality is a critical factor influencing the pace and direction of robotics development.
The Evolving Workforce: Human Pilots at the Forefront
The dozen or so pilots at Encord’s San Leandro facility represent a burgeoning workforce at the nexus of human expertise and artificial intelligence. Both Sofia Infante and Andrew Ceja, who previously worked at another AI data annotation firm, Scale, before joining Encord, embody this new class of skilled labor. Their roles require not only meticulous execution of tasks but also an understanding of how their actions translate into valuable training data for robots.
Ceja, who previously managed a robotic trash sorter at a waste management company, finds the challenges of solving training tasks for robots deeply engaging. He enjoys the dynamic nature of the work, stating, "It’s something new every day!" This enthusiasm highlights the unique appeal of contributing directly to the foundational building blocks of future neural networks. These pilots are not just operators; they are integral contributors to the iterative process of teaching robots, providing the critical human insight and dexterity that machines currently lack. Their work ensures that the data is not only technically sound but also captures the subtle complexities of human interaction with the physical world.
Broader Implications and Future Outlook
Encord’s unique vantage point, collaborating with numerous leading robotics companies, offers unparalleled visibility into industry-wide trends. Velmurugan notes that this position allows Encord to identify which data techniques are gaining traction and proving effective across various robotics programs, providing a strategic advantage that no single customer could achieve independently. This collective learning accelerates the entire industry’s progress towards more capable and autonomous physical AI systems.
The implications of successfully addressing the physical AI data bottleneck are far-reaching. Overcoming this challenge could unlock exponential growth in sectors reliant on robotic manipulation, including advanced manufacturing, logistics, healthcare, and even domestic assistance. Imagine warehouses where robots autonomously handle complex sorting and packing tasks with human-like dexterity, or service robots that can perform intricate procedures with precision and adaptability. The integration of brainwave and somatic data could lead to robots that not only mimic human actions but also anticipate human intent or react to perceived errors in a more intelligent, human-centric manner.
As the demand for increasingly sophisticated and versatile robots grows, the need for robust, real-world training data will only intensify. Companies like Encord are not just providing a service; they are laying the groundwork for a future where physical AI can truly integrate into and augment human environments, moving beyond specialized, narrow tasks to become truly versatile and intelligent partners. The San Leandro warehouse, with its Jenga towers and brainwave-sensing headsets, is more than just a data collection facility; it is a crucible where the fundamental challenges of physical AI are being confronted and, slowly but surely, overcome.
