On Embodied AI

This summer, after being awarded Northwestern’s Levy Inspiration Grant, I had the privilege of visiting the Bay Area and Seattle to learn a little more about what many experts consider the next rotation in the AI wave, Embodied AI - artificial intelligence that learns and acts in the physical world, using sensory feedback to interact with its environment.

My interest in this space has developed both fast and slow. On a personal note, I’ve admired my brother’s conviction in humanoid robotics from a distance for some time. After years of him sharing observations, predictions, and demo videos from his time in the field, I’ve inched toward the robotics-optimist camp. That said, my urge to understand it has picked up pace for a few reasons.

A fascination with the world’s most prescient technologists

It’s hard not to respect those willing to wager their reputations on contrarian, creative, or outright outlandish ideas about where technology is heading. Respect turns into admiration when those bets pay off often enough to form a track record, particularly in the face of public doubt. You see this frequently in film, deep tech, and investing, among other areas. Embodied AI is one of the places where a number of these thinkers are now spending their calories. I’d love to understand what they see, in hopes of developing some of that discipline and conviction myself.

Reflecting on professional purpose as a graduate student

My early career as a consultant gave me a sense of what well-run businesses tend to get right and where they routinely get stuck. But where it excelled in exposure, it lacked in purpose. As a gun-for-hire, I found it difficult to consistently align my project work with my values and interests. That alignment has become table stakes for my next career shift, and I think I could find it in embodied AI. How much we benefit from AI as a society depends in large part on what it can actually do in the physical world. I’d love to play a part in figuring that out.

Existential dread, exacerbated by the doomerism you see in the media

I find it unsettling to watch researchers at frontier labs ring alarm bells while continuing model development at a breakneck pace, with few constraints from industry or government. Human psychology explains at least some of this. These are brilliant people facing institutional capture, to say nothing of the compensation and IPO incentives that reward moving quickly and making noise. That said, the effective altruists of the bunch have begun to raise hell in an effort to take out a moral and ethical insurance policy or adopt a saviorist complex, or both. Either way, I’m not entirely confident the current class of techno-maximalists are ready to shepherd AI into the physical world - where the stakes are even higher.

With all that in mind, here’s where I heard the most agreement on the trip, and where I’m still forming my own views.

Where the industry has rough consensus:

The biggest constraint on commercial deployment is a shortage of diverse, multimodal data.

LLMs benefited from internet-scale training data, and we’re already seeing some of that progress carry over to robotics with vision-language-action models (VLAs). Even so, It is unlikely visual data alone will get robots deployed as quickly or reliably as we need them to. In short, the diversity of the data seems to matter, and matter greatly. There is demand for vision, force, touch, and position signals together, with plain-language annotations describing tasks being performed. Tactile data specifically has shown immense potential in lab settings, but the question is how to collect it reliability and cost efficiently at scale.

As with LLMs, scaling laws appear to be holding for robotics.

If more data continues to improve how robots perform, it’s easy to see why so many teams are trying to make simulation work. Once you have the compute, the marginal cost of running another simulation is not only low, but significantly lower than other data capture methods. With cost not a concern, the harder part becomes getting what a robot learns in simulation to hold up in the real world. This is often referred to as the sim-to-real gap. The industry consensus seems to be that specialized real-world datasets will retain their value for much the same reason proprietary, domain-specific data remains valuable to frontier LLM labs. This seems especially true for dexterous tasks, where tactile feedback tells you something about the physics of an interaction and whether the robot actually pulled it off.

Getting robots to perform longer tasks takes work on hardware, software, and service design in parallel.

In the US, we tend to overlook the hardware constraints in commercial robotics. At today’s volumes, what component suppliers need to charge is deeply at odds with what robotics companies are charging customers to encourage early deployments. The math doesn’t math, though you could say the same about what frontier labs charge consumers for GenAI relative to the cost of compute. Maybe robotics companies can hold out long enough for economies of scale to kick in, but maybe not.

Where there are still unknowns:

What’s the right mix of data across the “data pyramid”?

In the near term, I expect teleoperation to lose ground to simulation. With relatively few robots out in the world collecting deployment data, I see a case for visuo-tactile gloves, particularly for high-value tasks that require dexterity. A hand can block the camera’s view at precisely the moment of contact (visual occlusion), and force and shear remain difficult to model accurately in simulation.

Longer term, simulation at scale and better models could give robotics its GPT moment. If robots get fast and reliable enough to put to work in greater numbers, those deployments will generate data that improves the next version release. I’d still expect skill-capture gloves or devices to be useful at that point, though each additional hour of glove data may be worth less to a lab, causing the business case to get shaky over time.

Which robotic hands will work best in the real world?

I suspect the companies that capture the most value will be the ones that collect proprietary data in a form the robotic hands they sell can use with minimal translation. There’s an advantage to owning both ends of that process. If you design the data collection around your own hand, you can sidestep much of the work of transferring what one hand learned to another. This is often referred to as the cross-embodiment problem.

From a design standpoint, which hand wins is less clear. There are two fiercely loyal camps today: direct-drive and tendon driven designs, The modular direct-drive design might let you replace a finger instead of the whole hand more easily, though durability and how readily the joints move under an external force (backdrivability) are still concerns. Maybe direct drive wins, maybe tendon-driven hands do, or maybe an entirely different design makes the debate moot.

How do you collect multimodal data at a scale and price labs can live with?

For any data collection device, changing how someone naturally performs a task is a non-starter, both for the quality of the data and for adoption of the tool itself. That’s why I don’t see UMI working at scale. I think gloves are the way to go, provided they’re ergonomic enough for everyday use and the sensors can hold up to repeated use without sacrificing data quality. Barring government intervention (more on that in a future post), I’d expect the value of collecting data this way to diminish over time as simulation and models improve.

Where the opportunity lies

More on this in subsequent posts! Hope you enjoyed this breakdown - please reach out with any feedback or builds!