Back to Blog 
92
“Bring me the red mug on the table.”
When you say this to a robot, its camera activates and begins observing its surroundings. Knowing already what “red” and “mug” mean, the robot relies on state-of-the-art vision-language models to identify the correct object with remarkably high accuracy, even when several similar-looking items are scattered nearby.
At the next moment, however, as the robotic arm reaches toward the cup, it stutters awkwardly, its wrist angle slightly off. In that fraction of a second, it misses the handle entirely or applies incorrect force, pushing the cup away instead. The robot knows what to pick up, but it still struggles with how to pick it up.
This scene captures a fascinating paradox facing AI and robotics today.
Over recent years, Vision-Language Models (VLMs) have advanced at a breathtaking pace, building a rich repository of semantic knowledge by perceiving images and text simultaneously. Today, these models can describe a scene from a single photograph, infer the function of objects, and even make common-sense judgments such as “this object could be used in place of a hammer.”
However, the core issue is that this high-level knowledge does not easily translate into physical action. Understanding a world of pixels on a screen is fundamentally different from moving accurately in a physical world dictated by friction, center of gravity, and latency.
Right at that junction, where understanding turns into action, an unbridged last mile remains. The defining challenge for Physical AI today is precisely bridging this last mile.
The Unseen Gap Between Meaning and Action
A critical gap persists between an AI understanding the world and actually acting upon it. While this transition is so seamless for humans that we rarely even register it, it remains one of the hardest problems in robotics. Just as humans naturally move after processing meaning, a robot must convert perceived information into physical action. Yet this exact translation process, turning meaning into action, remains an unresolved bottleneck.
Historically, robotics bypassed solving this gap head-on. Instead, it developed Cognition, Planning, and Control as separate, modular components linked via interfaces. While this modular structure effectively secured stability and reliability, it erected an invisible barrier between meaning and action.
Consequently, even state-of-the-art AI models have been restricted to playing high-level Planners in robotics, issuing macro-level instructions rather than executing them directly. Why does an AI trained on vast world knowledge struggle to execute physical actions directly?
Two decisive technical bottlenecks account for this:
- Frequency Mismatch: Low vs. High Frequency
The first challenge lies in the gap between the speed of thought and the speed of action.
– Massive VLM Computation (around a few Hz): An AI takes hundreds of milliseconds to several seconds to visually evaluate its environment and plan the next move.
– Real-Time Motor Control (100 Hz to 1 kHz): Conversely, for a robotic arm to grip a cup stably without shaking, motor torque control loops must execute at ultra-high speeds, running 100 to 1,000 times per second to process real-time physical feedback.
Simply put, the computation speed of the brain cannot keep pace with the real-time reaction speed of the muscles. No matter how clever the plan, without real-time joint corrections, it becomes useless in the physical world. - Absence of Physical Modalities: Beyond Vision to Touch
The second challenge stems from the disconnect between web data and the physical world.
Traditional VLMs matured by learning purely visual (Image) and linguistic (Text) data available on the internet. However, regulating force, such as turning a doorknob smoothly without breaking it or picking up an egg without crushing it, depends on tactile pressure at the fingertips and motor current feedback (proprioception), not vision alone.
Current web-scale datasets severely lack the physical sensory data required for robots to interact with the physical world. Vision and language alone cannot calculate real-world friction, weight, and physical resistance.
A Paradigm Shift to Bridge the Gap Between Thought and Action: The Rise of VLA Models
This exact bottleneck is where the frontier of robotic AI is rapidly shifting today.
Moving beyond traditional VLMs that functioned merely as high-level Planners, a profound evolution is underway toward Vision-Language-Action (VLA) models. These models directly output changes in a robot’s joint angles and end-effector coordinates, treating them exactly like text tokens. This represents a head-on attempt to integrate high-dimensional semantic understanding and low-dimensional physical control into a single, massive neural network.
If this mechanism for translating the language of massive intelligence into robotic movement functions perfectly, robots will no longer be machines restricted to repeating pre-programmed routines. They will become autonomous agents capable of interpreting novel situations and adapting flexibly on the fly.
For example:
- Traditional Robot: Encounters a critical error if a designated object is not at its exact pre-programmed location.
- VLA-Powered Robot: “Understands unstructured instructions like, ‘If there are no tongs, find a similar elongated object nearby and use it instead.’ It visually analyzes surrounding objects, selects an alternative, grips it, and creatively improvises a solution.
In this moment, robotics transcends basic factory automation and enters the realm of true Physical AI, capable of real-time, dynamic interaction with the real world.
Of course, directly translating a massive model’s common-sense knowledge into action vectors to narrow the gap between thought and action does not instantly solve every problem. The physical world we live in is filled with unpredictable disturbances and endless frictional variables.
- This raises a critical challenge for industrial environments: In manufacturing, logistics, or even daily life, a single mistake or malfunction can lead to catastrophic losses. Can a probability-based learning model truly guarantee the near-perfect stability and robustness required in these scenarios?
Humanity’s challenge to merge massive intelligence with physical bodies, effectively bridging the gap between mind and control, has now reached its most critical gateway. From here, we will dive deeper into the specific internal mechanisms driving VLA models and explore the immediate challenges that lie ahead.
Why VLMs Hit the Wall of Being Just a ‘Thinking Model’
To simplify how traditional robotic systems operated, they generally relied on a three-stage pipeline:
- Perception Module: Interprets the surrounding environment via cameras and sensors.
- Planning Module: Determines high-level strategy regarding what needs to be done.
- Control Module: Computes precise physical values to actuate the motors.
At first glance, this modular design appears entirely logical. For humans, it mirrors looking with our eyes, planning with our brains, and moving with our limbs. When Vision-Language Models (VLMs) were first integrated into robotics, they were naturally assigned the role of the brain, acting as the high-level Planner.
Google’s SayCan, introduced in 2022, opened the door to this paradigm, relying fundamentally on Large Language Model (LLM) intelligence:
- Say (Does it make semantic sense?): The language model interprets ambiguous user instructions to curate candidate steps that are logically sound.
- Can (Is it physically executable?): A separate Affordance Value Function evaluates whether the robot can actually execute those candidate actions in its current environment.
In short, it determined the next action by multiplying what the semantic brain knew (Say) by what the physical body could do (Can).
Consider a high-level instruction like: “Get the water bottle from the fridge and place it on the table.”
Given this task, a VLM decomposes the high-level plan into surprisingly plausible sub-goals:
- Navigate to the fridge ➔ Open the door ➔ Locate the water bottle ➔ Pick it up ➔ Close the door ➔ Transport the bottle ➔ Place it on the table
Looking strictly at this step-by-step breakdown, a VLM appears to function as a fully capable real-world intelligence.
However, the sharp intelligence of VLMs blurs dramatically the moment it descends into the lower control layers. This is because the real, complex challenges encountered in actual deployment are practical and granular:
- At what exact angle should the robot grip the refrigerator door handle?
- How should the robotic arm maneuver inside if the bottle is partially obstructed near the back?
- How much force should be applied depending on whether the bottle is made of plastic or glass?
These questions belong to the domain of low-level control, requiring high-speed calculation of physical values. Right at this junction, the rich contextual information held by the model at the high level fails to transfer down to the lower layers, suffering significant loss along the way. In literature, this phenomenon is referred to as the Semantic Bottleneck.
The high-level model certainly understood the context that ‘this object is a fragile glass bottle and must be handled with care.’ Yet the low-level control system responsible for generating raw motor commands lacks the capability to consume or interpret such semantic knowledge directly. Instead, it relies strictly on geometric coordinates derived from 3D point cloud data or rigid, hardcoded rules.
Ultimately, while the robot does move, its movement is less an intuitive behavior flowing naturally from semantic understanding and more a disconnected execution operating in isolation from high-level intelligence.
The Gap Between ‘What to Use’ and ‘How to Grasp’
This disconnect is far more critical than it appears. Suppose a small hammer is missing from the table, but the user issues the command: ‘Lightly tap the lid to open it.’ A human can easily look around and identify a solid metal cup or a heavy flashlight as a temporary substitute tool.
Under this scenario, a traditional Planner-based architecture can successfully perform common-sense reasoning, inferring that ‘Since there is no hammer, use the metal cup as a substitute.’ However, it fails to translate this inference directly into low-level execution.
- Traditional Robotic Constraint: The cup-gripping algorithm stored in the robot’s skill library is programmed solely for upright grasping suited for drinking.
- Actual Required Action: To use the cup as a hammer, the robot requires a functional grasp, gripping the bottom or handle securely upside down.
Because high-level common sense like ‘grasp it upside down like a hammer’ fails to cascade down to the fingertip controller, the robot politely grips the cup upright as if preparing to drink water, swings it clumsily, and ultimately drops it on the floor.
Does the Coach Need to Stay on the Sidelines?
The advantage of traditional architecture is clear. Strict modular separation between perception, planning, and control allows each module to be tested and refined independently, securing the critical stability required in industrial applications. However, this compartmentalized structure inevitably dilutes and dissipates semantic context as it trickles down to motor execution. The VLM understood the world deeply, yet its intelligence ultimately failed to reach the fingertips.
To borrow an analogy: early VLM-based robotic systems were much like brilliant football managers.
Sitting on the bench, the manager analyzes the field, devises a flawless strategy, and barks out instructions: ‘Grab that object next, then move over there.’ Yet the player physically running on the field and kicking the ball is someone else entirely. The manager’s tactical vision and nuanced feel for the game never quite transfer fully down to the player’s boots.
Right at this inflection point, a radical question emerged:
“Does the manager really need to stay on the bench? Can a massive model that understands semantics step onto the pitch as a playing coach, opening up pass lanes alongside the players in real time?”
A model that goes beyond issuing tactical plays to directly generate motor trajectories. RT-2, introduced in 2023, confronted this exact question head-on.
RT-2: The Moment Action Was Reframed as Language Rather Than Control Signals
The underlying core concept behind Google’s RT-2, introduced in 2023, was strikingly simple yet ingenious: instead of treating robotic movement as complex engineering calculations, why not handle it as words, the very domain where AI excels?
To appreciate why this represented a monumental paradigm shift, it helps to glance at how traditional robots operated. Classical robot control always relied on computing precise floating-point numbers, such as three-dimensional spatial coordinates or joint rotation angles.
- Traditional Robotic Approach (Mathematical Computation): Moving a robotic arm required real-time calculations to output continuous floating-point values, such as moving 3.14 cm to the right or bending a joint angle by 45.2 degrees.
While such mathematical formulations are second nature to traditional computers, continuous numerical data felt alien and awkward to massive AI models whose intelligence matured on human text and language.
Reframing Numbers into Words: Action Tokenization
RT-2 fundamentally flipped this paradigm on its head.
By quantizing low-level robot control commands (such as positional displacement and rotation angles) into discrete bins, RT-2 mapped them one-to-one onto previously unassigned tokens within the VLM’s existing vocabulary dictionary. In essence, fine-grained movement distances and wrist angles were assigned unique tokens, effectively treating physical actions like letters of an alphabet.
- Traditional Approach: Processing raw floating point data like
[X:+0.032, Y:-0.015, Z:+0.102] - RT-2 Approach: Outputting discrete text tokens like
[Action_Token_142],[Action_Token_89]
Consequently, from the model’s architectural perspective, generating the word ‘mug’ and emitting control tokens for ‘move +3 along the X-axis’ became identical computational operations.
In literature, this concept is known as Action Tokenization. RT-2 leveraged state-of-the-art vision-language backbones of its time, such as PaLI-X and PaLM-E, layering these action tokens directly onto their vocabulary space to train perception, reasoning, and physical control jointly.
The Magic of Co-training
The disruptive power of this concept exceeded expectations. As the model gained the ability to receive images and text instructions as inputs and directly output physical movement as action tokens, long-standing boundaries in robotics began to crumble. Planning and control were no longer compartmentalized; they merged into a single, continuous process within a massive neural network.
However, an important engineering context must not be overlooked. The AI did not magically learn to operate a robotic arm purely by consuming text and images off the internet. The true magic demonstrated by RT-2 stemmed from Co-training. Google DeepMind combined web-scale common-sense data with precise physical trajectory datasets collected by physical robots, baking them together in a single computational furnace.
As a result, a seamless knowledge transfer occurred, allowing high-level human common sense stored in web data to cascade directly into actual physical trajectories.
Consider a scenario where the robot receives the instruction: ‘Pick up the trash and throw it in the bin.’
- Traditional Robot: Sees the environment strictly through geometric properties, evaluating only the physical dimensions and spatial location of objects. Relying solely on geometry, it might make the critical error of tossing a wallet into the trash alongside empty packaging.
- Co-trained Robot: Leverages common-sense categorization capabilities acquired from web-scale data, instantly recognizing that an empty snack wrapper is trash to be discarded, whereas a wallet placed nearby is a personal asset to be left untouched.
This common-sense reasoning serves as a direct input to fingertip trajectory generation, enabling the robot to precisely target and pick up only the snack wrapper.
When Common Sense Becomes the Raw Material for Action: Tool Use and Action Generalization
The most vivid demonstration of this shift lies in tool use.
- Traditional Robotics: When commanded to ‘use a hammer,’ engineers had to explicitly hardcode mathematical equations specifying precise grasping angles and trajectory skills directly into the system.
- VLA Paradigm: Conversely, RT-2 class models already embody broad contextual common sense from web-scale data, understanding what a hammer is, how it functions, and which objects can serve as functional substitutes when a hammer is absent.
When this high-level common sense merged with physical action datasets, a remarkable breakthrough occurred.
Faced with an unexpected scenario where a hammer was unavailable, the robot evaluated its surroundings, selected a solid metal cup as an improvised hammer, and gripped it upside down at an optimal angle for striking. This represented true Action Generalization. It was no longer the helpless robot operating under disconnected control layers, politely grasping the cup upright only to swing clumsy air and drop it to the floor.
This marked the precise moment where a deep understanding of the world ceased to remain trapped inside static manuals, evolving into adaptive real-time physical actions.
Proof of Paradigm and Physical Limitations
RT-2 did not immediately achieve human-level precision or dexterity in robotic manipulation. Its foundational significance lies in providing a Proof of Concept: demonstrating that the reasoning intelligence of vision-language models can directly integrate with low-level control layers.
If traditional VLM-based systems remained sidelines managers issuing instructions from the bench, RT-2 proved that massive models can step onto the pitch as playing coaches, directly generating physical action trajectories on the field.
Yet this unified architecture confronts distinct limitations when colliding with the realities of the physical world:
Precision Limits Driven by Quantization Error: Discretizing infinite, continuous physical control signals into hundreds of discrete action tokens inherently introduces error. The tokenized domain of words remains too coarse to regulate sub-millimeter frictional forces and fine physical resistance.
Computational Overhead and Latency Bottlenecks: The inference latency of massive models spanning billions to tens of billions of parameters struggles to satisfy the stringent loop-time demands of high-frequency closed-loop control, which requires hundreds of real-time interactions per second.
Ultimately, RT-2 illuminated the blueprint for connecting massive intelligence to a robot’s fingertips, yet it fell short of fully securing production-grade precision and real-time execution speeds.
How, then, are AI robotics researchers and practitioners addressing these coarseness and latency bottlenecks to grant robots agile, real-time physical responsiveness? In the next article, we will examine emerging breakthroughs, focusing on Diffusion Policies that smoothly generate continuous control signals and hierarchical VLA architectures.
