Meta’s Silent AI Will Replace LLMs

Yann LeCun has spent decades trying to replace how machines learn to see and think. The Turing Award winner, who helped invent convolutional neural networks and now serves as Chief AI Scientist at Meta, is once again aiming straight at the field he helped create. His target this time: the large language models that dominate today’s AI hype cycle.

Meta’s FAIR lab quietly posted a new paper describing a vision–language system built on LeCun’s Joint Embedding Predictive Architecture (JEPA). Branded as a VL-JEPA or VLJEPA model, it extends earlier V-JEPA work from 2023 by adding language on top of a predictive visual backbone. Instead of predicting pixels or tokens, the model learns to anticipate future or missing content directly in a shared embedding space.

LeCun has argued for years that real intelligence comes from learning a world model, not from auto-completing text. This new JEPA-based system embodies that stance: it operates as a non-generative model that predicts “meaning vectors” and only produces words when prompted. The architecture treats language as an optional interface sitting on top of a richer, silent internal state.

That makes the paper read less like another multimodal benchmark entry and more like a manifesto against the reigning LLM stack. Autoregressive models such as GPT-4, Claude, and Llama 3 generate outputs token by token, left to right, with every step exposed as text. JEPA-style models keep their reasoning internal, updating a latent state over time and emitting language only as a final serialization step.

LeCun has publicly called LLMs “blurry JPEGs of the web” and predicted that current architectures will look primitive within a few years. This work attempts to formalize his alternative: predictive, self-supervised systems that learn from continuous streams of video, audio, and other sensory data. The stakes go beyond chatbots, reaching into robotics, AR glasses, and real-world agents that must plan rather than merely talk.

All of this lands amid reports that LeCun plans to leave Meta to launch a startup built around next-generation JEPA-style AI. Rumors suggest a company focused on large-scale world models trained on video and embodied data, not just text scraped from the internet. If that happens, Meta’s own AI chief may end up leading the charge against the LLM paradigm he never fully embraced.

Generative AI talks its way toward an answer. Models like GPT‑4 or Llama 3 operate as autoregressive engines: they predict the next token, then the next, marching left to right until the sentence ends. Every answer exists only as a growing chain of tokens, so “thinking” and “speaking” are fused into the same slow, compute‑hungry process.

Non‑generative JEPA models split those apart. A Joint Embedding Predictive Architecture first forms an internal representation of what’s happening—across images, video, and text—then sits on that silent understanding. Language becomes an optional translation layer, not the medium of thought itself.

Generative systems behave like someone narrating their reasoning out loud: “Let me explain what I think while I’m still figuring it out.” Each word depends on the last, so the model literally cannot know the final phrasing, or sometimes even the final answer, until the sequence finishes. That token‑by‑token pipeline burns GPU cycles and introduces latency on every query.

JEPA flips the script: “I already know, and I’ll only explain if you ask.” Instead of predicting the next word, it predicts a meaning vector directly in a high‑dimensional semantic space. The core computation produces a single, dense representation that encodes entities, actions, and relationships without ever emitting text.

Because JEPA operates in semantic space rather than token space, it dodges the most expensive part of LLM-style inference. Autoregressive models must: - Run a forward pass for every token - Maintain and update a long context window - Sample from a large vocabulary distribution repeatedly

JEPA runs one forward pass to get a stable embedding and stops. Converting that embedding into a caption, answer, or command becomes a lightweight decoding step instead of the main event. Meta’s VL‑JEPA prototypes already report using roughly half the parameters of comparable generative vision‑language stacks while matching or beating them on benchmarks.

Silent internal state also enables continuous understanding without constant chatter. A VL‑JEPA system can watch a video stream, refine its meaning vector over hundreds of frames, and only emit language when prompted or when an external system needs a symbolic description. Thinking happens continuously; talking becomes a side effect.

Language models like GPT live and die by tokens. They slice the world into discrete word pieces, then grind through them left to right, predicting the next fragment of text. Vision add-ons for LLMs usually just bolt on a classifier that turns each frame into a caption, then hand those labels back to the text engine.

JEPA flips that pipeline. Meta’s VLJ model ingests raw video and builds a dense internal representation—an embedding—that tracks what is happening over time. Instead of narrating every frame, it maintains a silent, continuous meaning vector that only turns into words when you ask for them.

That embedding behaves like a “meaning space” rather than a token stream. Each point in that space encodes objects, actions, and context across multiple frames: hand, canister, motion, intent. When the system finally outputs “picking up a canister,” it is summarizing a trajectory through that space, not stitching together a guessy word-by-word description.

Meta’s researchers claim this buys serious efficiency. Because VLJ predicts in a compressed latent space instead of generating pixels or tokens, it reportedly uses roughly half the parameters of comparable vision-language transformers while matching or beating them on standard benchmarks. Fewer parameters mean lower memory pressure, faster inference, and better scaling on edge hardware like headsets or robots.

Contrast that with a typical LLM vision stack. A standard vision encoder looks at each frame, emits a label—“bottle,” “hand,” “table”—and forgets almost everything between steps. There is no persistent semantic state, only a stream of captions that the language model tries to weave into a story after the fact.

JEPA’s world model runs the other way: persistent understanding first, language second. The VLJ: Vision-Language-Jeopardy (placeholder arXiv entry) paper describes a system that keeps that internal movie of meaning running silently, then surfaces it as text only when humans need a sentence.

Yann LeCun has been hammering the same point for years: intelligence is about building an internal model of the world, not about sounding smart in English. In his view, language sits on top as a convenient “I/O protocol” for humans, the way HDMI is for monitors. Useful, yes, but not where real understanding lives.

That philosophy puts him squarely at odds with the LLM arms race. GPT‑style systems train almost entirely on text scraped from the internet, then generate more text token by token. LeCun argues that this setup confuses eloquence with comprehension and locks research into a dead-end architecture.

He calls the core problem “ungrounded” learning. Text alone never touches friction, gravity, occlusion, or causality; it only reflects how humans talk about those things. Train only on words, he says, and you get a model of culture, not a model of reality.

LeCun’s critique shows up in his favorite comparison: a teenager learns to drive in roughly 20 hours of practice, yet after more than a decade, billions of dollars, and millions of driven miles, we still do not have reliable Level 5 self-driving cars. For him, that gap is not just an engineering lag; it is evidence that current data and architectures are fundamentally misaligned with how humans acquire competence.

Humans learn from continuous, messy sensory streams—vision, sound, proprioception—and only later attach words. LLMs invert that pipeline, starting from captions, manuals, and forum posts. LeCun argues this inversion forces models to fake physics and common sense from statistical patterns in text, which breaks down in edge cases, robotics, and real-time control.

JEPA is his escape hatch from that wall. Joint Embedding Predictive Architecture systems learn by predicting missing or future chunks of a scene in a latent “meaning” space, especially from video. Instead of outputting pixels or tokens, they predict how internal representations should evolve if the world obeys certain physical and causal rules.

World models built this way can, in principle, internalize dynamics like “if the mug tips, liquid spills” without ever reading the word “spill.” Feed JEPA models large-scale video—driving footage, household manipulation, warehouse robots—and they learn the regularities of motion, contact, and consequence directly.

LeCun frames VL‑JEPA and its successors as the route around the LLM plateau. Text becomes an optional interface bolted onto a grounded world model, not the foundation of intelligence itself.

Forget chatty bots; Meta’s new model starts with raw video. A visual encoder ingests a stream of frames and compresses them into dense vectors, a kind of internal movie of what’s happening. No captions, no labels, just compact representations of motion, objects, and context.

Those vectors feed into a predictor network that functions as the model’s “brain.” Its job: given some parts of the video, imagine the missing pieces inside that latent space. Instead of filling in missing pixels, it tries to fill in missing meaning — what should the internal representation of the unseen clip look like if the system truly understands the scene.

On the other side sits a target encoder. It processes the actual withheld video segment into its own latent representation. Training becomes a simple but brutal game: the predictor’s imagined vector must match the target encoder’s real vector as closely as possible, over millions of masked‑and‑predict episodes.

That setup forces V‑JEPA to learn abstract structure rather than surface patterns. To succeed, the model has to internalize concepts like “object permanence,” “occlusion,” and “cause and effect” because those are exactly what let it infer a hidden future frame from a past one. You can’t just memorize textures when half the action is missing.

The video’s simplified diagram helps demystify this. Picture three boxes in a row: “Video In” → “Brain” → “Understanding Cloud.” The first box is the visual encoder, the middle is the predictor, and the cloud is the evolving map of meanings where nearby points correspond to similar events, like “hand reaching” or “object being grasped.”

Training looks like repeatedly erasing chunks of that cloud and asking the brain box to restore them. Sometimes it only sees earlier frames and must guess what comes next; other times it sees the edges of a masked region and must infer what happens in the middle. Every success tightens the mapping between context and consequence.

Over time, that pressure sculpts a world model that tracks continuous events instead of isolated snapshots. Language can later tap into those latent vectors, but the understanding lives underneath, in the geometry of that meaning space.

Robots do not think in sentences. A warehouse arm deciding how to grab a box or a home robot figuring out how to open a fridge needs a continuous, non-linguistic model of the world: where objects are, how they move, what happens if it pushes, pulls, or waits half a second longer.

LLMs, even multimodal ones, bolt language on top of vision. They see a frame, generate a caption, then another caption for the next frame. That token-by-token narration wastes compute and, more importantly, shatters time into disconnected snapshots that are useless when a gripper has to land on a moving canister.

V-JEPA flips that around. Video flows into a visual encoder, which feeds a predictor tasked with forecasting future latent states, not future words. The system maintains a silent, high-dimensional “meaning vector” that evolves smoothly as the scene unfolds, and only surfaces language when a downstream task demands it.

Cheap vision models treat each frame like a separate quiz. They label one image “hand,” the next “bottle,” then “picking up canister,” then back to “hand,” producing jumpy, contradictory outputs with no memory. V-JEPA instead tracks a stable temporal representation of “a hand approaching, grasping, and lifting a canister,” and emits a single, confident label once the action pattern locks in.

That temporal stability comes from JEPA’s predictive objective. The model learns to predict the embedding of masked or future chunks of video, forcing it to encode not just what is visible now, but what is likely to happen next. Cause and effect over time becomes baked into the geometry of its latent space.

For robotics, that difference is existential. A robot that only recognizes “bottle, bottle, bottle” cannot decide when to close its gripper; a robot that internally simulates “this trajectory ends in a successful pick” can time its move, recover from slips, and plan multi-step behaviors. Planning, control, and navigation all hinge on this kind of forward model.

Meta positions JEPA-based systems as the backbone for embodied agents, wearables, and AR devices, and has started publishing technical details through Meta AI Research. If LeCun is right, those quiet, predictive world models—not chatty LLMs—will drive the next generation of physical AI.

Benchmarks are where Meta’s V-JEPA stops sounding like a philosophy lecture and starts looking like a problem for today’s vision–language models. In the video, the model posts state-of-the-art results on zero-shot video classification, beating larger, more complex baselines that rely on full-blown text decoders. It does this while operating purely in that “meaning space” LeCun keeps talking about, not by guessing the next word.

Meta’s numbers show V-JEPA matching or outpacing popular vision–language stacks on action recognition and temporal understanding, even when they get access to labeled examples. On zero-shot splits—where models never see labeled training clips from the target dataset—V-JEPA still tags actions and scenes more accurately, a sign that its internal representations actually generalize across domains.

Efficiency is the other headline. V-JEPA uses roughly half the trainable parameters of comparable vision–language setups because it skips the heavy, autoregressive text decoder during training. No giant language head churning through tokens means less memory, fewer FLOPs, and faster iteration, while the compact latent predictor does the real intellectual work.

“Zero-shot” here means the model receives only a natural-language label space—say, “pouring water,” “opening door,” “cutting vegetables”—and must classify new videos without seeing any labeled examples from that dataset. Strong zero-shot performance implies the model’s embedding space already encodes concepts like motion, intent, and object interaction in a way that transfers. It is a stress test of generalized understanding, not just memorization.

Critics on Reddit have already pointed out that V-JEPA’s predictions are sometimes off, especially in ambiguous frames or weird edge cases. That complaint accidentally underlines the point: this is an early research system, not a polished product, and the fact that it can fail visibly on complex temporal predictions shows Meta is finally attacking the right, hard problem rather than just scaling more tokens.

A quiet but very real fork is opening in AI strategy, and JEPA sits right at the split. On one side, companies like OpenAI and Google double down on LLM-centric, generative systems that treat everything—code, images, video, even action plans—as sequences of tokens to be predicted. On the other, Yann LeCun and Meta’s FAIR lab push Joint Embedding Predictive Architectures that never need to speak to think.

Path one looks familiar: keep scaling GPT-4-style models into multimodal behemoths. OpenAI’s GPT-4o, Google’s Gemini 1.5, and Anthropic’s Claude 3 all follow the same recipe: massive transformer backbones, trillions of tokens of web and proprietary data, and an autoregressive loop that predicts the next symbol, whether that symbol is a word, a pixel token, or an audio chunk.

JEPA represents a hard pivot away from that. Instead of generating pixels or words, V-JEPA and VL-JEPA learn to predict latent representations of future or missing content—what the model believes will happen next in a video, or what concept a region belongs to. Language becomes a thin layer on top of a world model, not the core substrate of intelligence.

That split leads to two optimization targets. LLM-first labs optimize for chat interfaces, code assistants, search, and productivity tools where natural language remains the primary I/O. JEPA-first research optimizes for robots, AR glasses, and autonomous agents that must track objects, intentions, and causality over time without narrating every microstep.

On the LLM path, progress comes from scale and alignment. Bigger context windows (up to 2M tokens), richer tool use, and retrieval-augmented generation push models deeper into workflows like software development, legal drafting, and customer support. The metric is how coherent, safe, and useful the generated text and code look to humans.

On the JEPA path, progress comes from better predictive world models. Benchmarks shift to zero-shot action recognition, temporal localization, and downstream control: can the system anticipate a hand reaching for a canister, or plan a sequence of grasps and pushes for a robot arm, using compact internal state instead of verbose prompts?

Both paths will likely coexist, but they pull the industry’s center of gravity in opposite directions. Either language stays the universal API for intelligence, or it becomes just one optional interface on top of silent, highly structured models that primarily understand and act in the physical world.

Rumors around Yann LeCun’s next move suddenly look less like gossip and more like strategy. Multiple reports say Meta’s chief AI scientist is spinning up a new startup, with Meta likely as anchor partner and funder rather than employer, giving him a separate vehicle to build the kind of AI he has been sketching in talks and papers for a decade.

LeCun has complained for years that frontier AI research moves on decade timelines while Big Tech ships on quarterly ones. A separate venture lets him chase JEPA-style world models and long-horizon learning without having to justify every experiment against Reels engagement or ad targeting.

His stated target is not “AGI” in the OpenAI or Anthropic sense, but Advanced Machine Intelligence (AMI). In LeCun’s definition, AMI means systems that can: - Build predictive world models from raw sensory input - Reason and plan over long horizons - Maintain persistent, grounded memory of the real world

AMI, in this vision, lives in robots, AR glasses, vehicles, and home devices before it lives in chatbots. It needs to track objects, intentions, and physics over time, not just autocomplete sentences. That is exactly the regime where JEPA and V-JEPA-style models, which predict in latent “meaning space” instead of token space, claim a structural advantage.

Meta’s latest V-JEPA and VL-JEPA work already shows non-generative models beating or matching larger generative rivals on zero-shot video classification and temporal understanding with roughly half the parameters. For a founder, those numbers translate into a simple thesis: world-model-centric AMI scales better than ever-larger LLMs that hallucinate and struggle with causality.

LeCun’s startup therefore looks like a clean, high-stakes bet that JEPA will outgrow today’s transformer LLM stack. If OpenAI and Google double down on massive autoregressive models, his camp will push silent, predictive systems that only speak when asked, but think all the time.

Anyone tracking this split should read the 2025 AI Index Report – Stanford HAI, which already flags a shift from pure language benchmarks to multimodal, embodied, and agentic evaluations. If those metrics become the scorecard that matters, LeCun’s gambit stops being contrarian and starts looking like the main event.

Post-LLM sounds apocalyptic, but reality looks more like coexistence than extinction. Large language models already run inside search, productivity suites, code editors, and customer service stacks, and their economics improve every time Nvidia ships a new GPU. Companies have poured tens of billions into LLM infrastructure, and that momentum alone guarantees they will dominate commercial AI interfaces for years.

JEPA-style systems aim at a different layer of the stack. LLMs excel at compressing the internet into autocomplete-on-steroids, but they struggle with tasks that demand grounded perception, long-horizon prediction, or fine-grained control of bodies in space. A robot that must decide where to place a foot on uneven ground cannot wait for a 200-token essay about its options.

Post-LLM, in LeCun’s vocabulary, describes the research frontier, not the product shelf. The frontier is shifting from “predict the next token” to “predict the next state of the world” across images, video, audio, and sensor streams. Language becomes a query and reporting channel, not the substrate of thought.

JEPA models such as V-JEPA and its vision–language cousins try to learn compact “meaning vectors” that evolve over time. Instead of emitting words at every timestep, they maintain a silent internal state that updates as new frames arrive, then expose that state when asked: “What is happening?” or “What should I do next?” That design lines up with control loops in robotics, AR glasses, cars, and factory systems.

Commercially, you can imagine a stack where: - A JEPA-like core tracks the environment and predicts future states - A planning module chooses actions over that latent space - An LLM explains those actions to humans in natural language

That is a post-LLM world: not LLM-free, but LLM-decentered.

If LeCun is right, the historical pivot is not about bigger models, but different thinking primitives. Swapping token-by-token generation for continuous prediction in a learned semantic space could unlock capabilities—agile robots, persistent agents, real-time assistants—that scaling GPT-style systems another 10x still cannot deliver.

What is the JEPA AI architecture?

JEPA, or Joint Embedding Predictive Architecture, is a type of AI model designed by Meta's Yann LeCun. Instead of predicting the next word in a sentence, it learns an internal model of the world by predicting missing or future information in a compressed, abstract 'meaning space'.

How is JEPA different from an LLM like ChatGPT?

LLMs are generative models that produce text token-by-token. JEPA is non-generative at its core; it builds an internal understanding first and only generates language as an optional output. This makes it potentially more efficient and better suited for tasks requiring real-world grounding, like robotics.

Will JEPA models replace LLMs?

Not necessarily replace, but they target different problems. While LLMs excel at language-based tasks, JEPA aims to solve physical world interaction and planning. LeCun believes this 'world model' approach is the path to more advanced AI, potentially making current LLMs obsolete for many future applications.

Why is Yann LeCun critical of today's Large Language Models?

LeCun argues that intelligence is about understanding the world, not just manipulating language. He believes that training models only on text is a fundamental limitation, as they lack the deep, causal understanding of reality that comes from sensory data like video, which is what JEPA is designed to learn from.