Joint Embedding Predictive Architecture (JEPA)

Overview

Recently, I came across a paper on the "Joint Embedding Predictive Architecture" (JEPA), co-authored by Yann LeCun, which may change how we think about robotic perception.

JEPA (Joint Embedding Predictive Architecture) is a method for training machines to understand the world by learning what can be predicted rather than what can be named. The goal is not to describe what is seen, but to grasp the meaning of a situation well enough to anticipate what will happen next.

At its core, JEPA trains a system to understand the world's affordances: what is stable, what can change, and which actions lead to which outcomes. It does this without putting that understanding into any particular language or visual image. Instead, the system learns an internal sense of “what makes sense” by being asked to infer missing pieces of its experience from what remains visible.

This approach is especially natural for 3D perception. Geometry carries constraints that appearance alone cannot provide. Surfaces support or block motion, empty space allows motion, and spatial relationships persist even when parts of the scene are occluded. By learning to predict what must be present but is not directly observed, the system internalizes these constraints as part of its understanding.

The representations learned in this way are not optimized for classification or reconstruction. They are optimized for coherence, or meaning over time. They capture the structure of the environment in a form that remains meaningful as the viewpoint changes or as actions unfold.

This makes JEPA a foundation for robotics. It provides a way to build perception grounded in the physical world, in which meaning emerges from what can happen rather than from how things are labeled. Subsequent chapters will build on this idea, one concept at a time.

The Real World

We all experience the world before we can describe it.
Long before words or drawings, there is experience. The body moves, the environment responds, and patterns slowly become familiar.

A hand presses against a table and feels resistance. An object released from the hand falls. A path allows movement; a wall does not. These are not lessons delivered through instruction. They are discovered through contact, repeated until they no longer surprise. Understanding begins as a sense of what tends to happen.

There is no clear boundary between perception and action. To function at all, a learning system must stay oriented within this flow, forming expectations that are stable enough to guide the next move.

Not every detail can be predicted. Surfaces vary. Shadows change. Yet most of this variation can be ignored. What matters are constraints that hold. Objects occupy space. Support prevents falling. Push transfers force. These regularities persist even as appearances change.

From these regularities, intuition forms. A cup is understood not as an image but as something that can be grasped, filled, tilted, or dropped. The meaning lies in what can be done and what will follow. These are affordances. They are learned through action, not description.

This kind of understanding is wordless and implicit.  It manifests as confidence in movement and restraint, with fewer surprises over time. Prediction improves, and with it, the ability to act without hesitation.

Only later do we begin to name what we already know. Words arrive as mental shortcuts. They point to patterns that existed long before they were labeled. Language facilitates communication, but it rests on a foundation that would exist without it.

The real world teaches first by enforcing consequences. It rewards correct predictions and corrects incorrect ones. From this process, a stable internal sense of space, time, and possibility emerges. That sense precedes abstraction and underpins it.

Text Is Not Enough

Language appears to be intelligent because it is visible. We can read, hear, and test it. When a system produces fluent text, it conveys an understanding. But language is not where understanding begins. It is where it ends up after something quieter and more physical has already occurred.

Text is a record of experience, not the experience itself. It assumes a shared world. When we say that a cup fell, we do not explain gravity, support, or collision. We assume them. A reader already knows what falling means because they have lived in a world where objects drop when released. Language works by leaning on that prior knowledge, not by creating it.

This is the limit of language-only systems. They learn correlations between symbols that already compress the world. They never see the constraints that produced those symbols. They do not feel resistance, continuity, or consequence. As a result, they can describe situations fluently while lacking an internal sense of what can actually happen next.

Real understanding is predictive. It is the ability to anticipate outcomes without narrating them. When you reach for a door handle, you already know how it will turn. When you step onto a surface, you trust it will support your weight. This knowledge is not verbal. It is a lived model of the world, shaped by repeated exposure to structure and constraint.

To move beyond language, we must return to those foundations. Systems must learn from signals that carry the world’s structure directly: space, motion, geometry, and time. They must form internal representations that stay coherent as viewpoints change and actions unfold. The goal is not to predict words or images, but to predict what remains stable and what can change.

This is the shift ahead. From describing the world to modeling it. From naming outcomes to anticipating them. Language will remain useful, but it will sit on top of something deeper: an internal sense of how the world works, learned from the world itself.

Affordance is Play

Affordance as a Function of Intent and Action?

As a person passionate about Behavioral Sciences, I found myself unable to shake the impression that the paper “A Theory of Affordances in Reinforcement Learning” somewhat artificially imposes a mathematical frame on the idea. The authors, following Gibson in 1977, define “affordance” as a relationship between intent and action success within a Markov Decision Process (MDP). 

In their formulation, an affordance is something that fulfills an intended goal with a certain probability. 

The paper remains valid but inspires us to think more deeply about the meaning of affordance, especially in the context of Machine Learning.

Markov Decision Process (MDP) is a loop:

1. You (or agent) in some situation (personal and world state).
2. You (or agent) choose an action.
3. The world reacts (new state + reward).
4. Wash and repeat.

MDP fits neatly into the machinery of reinforcement learning, where an agent seeks to maximize rewards by executing optimal actions.

Gibson (1977) defined affordances as different possibilities of action that the environment affords to an agent.

I do not have an issue with Gibson's ecological definition. 

Affordance is Play

To me, affordance has less to do with intent and everything to do with play.

In the real world, we discover affordances not by planning but by playing with possibilities. 

A child learns what can be done by climbing, falling, touching, and testing the boundaries of safety. The best moments of childhood play are those when we barely escaped unscathed; when the heart raced, when something almost went wrong, but didn’t. That boundary between freedom and danger is where learning is deepest.

Affordance, in this sense, is not about achieving an outcome. It is about staying alive while experimenting with the world. 

It defines the space within which we can change the game's parameters without getting hurt. 

In reinforcement learning (RL) terms, it is the manifold of safe exploration, or the zone where feedback is rich but failure is not fatal.

source: Atlantic

 

Simulation and Safe Play

This form of learning is ancient. We simulate dangerous situations in imagination, running them over and over in the mind’s eye. Or we externalize them through ritualized, controlled play, just like in karate kata, where potentially lethal movements are practiced in a form that is safe and repeatable. Then there is karate kumite, or sparring, where we step closer to real risk but still within agreed limits. Both are forms of play. Both train us to recognize the edges of danger without crossing them.

It is the Most Serious Matter

Play, then, is not the opposite of seriousness; it is the method by which organisms map the terrain of survival. Through play, we learn about the world’s resistance and our own capabilities. The outcome of play is not a single intent achieved, but a widening of understanding; an embodied sense of what is possible.

The Definition

If I were to define affordance, I would not say it is “the subset of state–action pairs that fulfill intent.” I would say:

Affordance is the set of game parameters within which an agent can safely play and explore the outcomes that bring meaningful experiences.

It is the field of viable transformation, the space where curiosity meets consequence and both survive the encounter.

Mathematics

For those people who do not consider writing worth reading unless it contains at least 20% of LaTeX.

Let’s assume hazard H is a function that indicates how risky it is to take a particular action a in a given state s.

$$ H(s,a) \le h_{max} $$

If the hazard is below some acceptable threshold \( h_{max} \), then this action is considered safe enough to play with.
 In everyday terms: “Don’t jump higher than you can land safely.”

Now we introduce a second measure, feedback richness F. It describes how much change or learning happens when you take an action a. We can compute it by looking at the expected difference between the current state s and the next state s' after performing action a.

$$ F(s,a) = \mathrm{E}_{s' \sim P(\cdot \mid s,a)} \left[ D(s,s') \right] $$

where:

Symbol	Meaning
\( s \)	The current state of the agent.
\( a \)	The action taken in that state.
\( s' \)	Read “s-prime” — the next state that results after performing the action.
\( P(s' \mid s,a) \)	The transition probability: how likely it is to reach each possible next state \( s' \) when taking action \( a \) in state \( s \).
\( D(s,s') \)	A difference function measuring how much the world or the agent’s perception changes between \( s \) and \( s' \); it could represent distance, novelty, or information gain.
\( \mathbb{E}_{s' \sim P(s' \mid s,a)}[\cdot] \)	The expected value — the average of that difference over all possible next states following from \( s,a \).

The formula asks: “On average, how different is the world after I act?”
If nothing changes (like bouncing on a log that won’t move), feedback is low. If something new happens (the world reacts in an interesting way), feedback is high.
So \( F \) measures how informative or “fun” the action is: how much it teaches the agent about the world.

The affordance of play is the set of all state–action pairs that are both safe and interesting:

$$ A_{play} = \{ (s,a) \in S \times A \mid H(s,a) \le h_{max},\ F(s,a) \ge f_{min} \} $$

In plain language: An action is “playable” if it won’t hurt you and it’s worth doing because you might learn something new. It defines the zone of viable exploration; the sweet spot between boredom and danger.

Next, we define the play policy: the way an agent chooses its actions to keep playing safely and learning effectively.

$$ \pi_{play}^{*} = \arg\max_{\pi} E\!\left[ \sum_{t=0}^{T} \gamma^{t} F(s_t,a_t) \right] \quad \text{s.t.} \quad H(s_t,a_t) \le h_{max},\ \forall t $$

where:

Symbol	Meaning
\( \pi \)	A policy — a rule or function that tells the agent which action to take in each state.
\( \pi_{\text{play}}^{*} \)	The best (optimal) play policy that satisfies the safety rule.
\( \arg\max_{\pi} \)	Find the policy that gives the maximum possible value.
\( \mathbb{E}[\cdot] \)	The expected value, i.e., the average over all possible outcomes.
\( \sum_{t=0}^{T} \)	A sum over time steps from the beginning (\(t=0\)) until some horizon \(T\).
\( \gamma^{t} \)	A discount factor (0 < \( \gamma \) < 1) that slightly reduces the importance of far-future experiences; it models our natural tendency to value the present more than the distant future.
\( F(s_t,a_t) \)	Feedback richness — how much the agent learns from taking action \(a_t\) in state \(s_t\).
\( H(s_t,a_t) \)	Hazard — the expected risk or potential harm of taking that action.
\( h_{\max} \)	The maximum acceptable hazard, the safety limit.
\( H(s_t,a_t) \le h_{\max} \)	The safety constraint: every action must remain below the danger threshold.

This formula means the agent tries to maximize total feedback over time (it wants the richest learning experience possible) while keeping every action within the safety boundary. 

In other words: “Explore as much as you can without getting hurt.”

The factor \( \gamma \) (gamma) discounts future feedback slightly, just as we naturally prefer immediate experiences to distant ones.

For comparison, the classical reinforcement learning policy focuses purely on reward:

$$ \pi^{*} = \arg\max_{\pi} E\!\left[ \sum_{t=0}^{\infty} \gamma^{t} r(s_t,a_t) \right] $$

Traditional Reinforcement Learning (RL) agents chase rewards \( r \). 

A playful agent instead chases understanding \( F \), which signals curiosity, while respecting a safety boundary \( H \). 

 It is not optimizing for success but for continued participation.

Application for Machine Learning

In practical machine learning, this perspective turns into a guide for building better agents. Instead of rewarding an agent only when it achieves a fixed goal, we can encourage it to discover where it can play safely. The agent learns the boundaries of its world and the regions that respond richly to its actions, rather than blindly optimizing for external rewards.

In reinforcement learning terms, the agent can maintain two learned functions: a hazard model \( H(s,a) \) estimating the probability of failure or damage, and a feedback model \( F(s,a) \) estimating the expected amount of new information or improvement gained from an action. By combining them, the agent can create its own playground of exploration \( A_{play} \) and focus on safe and informative actions.

Such agents do not merely chase rewards; they evolve through curiosity. 

They learn the structure of their environment by interacting with it, not just memorizing outcomes. This approach aligns machine learning more closely with how humans and animals learn: through playful experimentation, through testing what can be changed without harm. It is learning as a continuous dance with uncertainty; an art of exploring the edge of safety where discovery happens.

References

• “A Theory of Affordances in Reinforcement Learning”, Khimya Khetarpal et al. https://arxiv.org/pdf/2006.15085