Hierarchical World Models with Implicit Dynamics

Gaoyue Zhou^1,2, Yvonne Wu¹, Zichen Jeff Cui¹
Nicolas Ballas², Mido Assran², Lerrel Pinto¹, Yann LeCun^1,3

1. Courant Institute, New York University 2. Meta-FAIR 3. AMI Labs

NeurIPS 2026

arXiv Code (coming soon) BibTeX

ID-WM

Abstract

Planning with world models enables test-time adaptation for embodied agents, offering superior generalization over direct policy learning. However, long-horizon tasks remain hindered by cumulative errors and search complexity. Existing hierarchical methods often use rigid macro-actions or latent skills, leading to brittle representations and limited fine-grained control. We introduce ID-WM (Implicit Dynamics World Models), a hierarchical world model that decouples state-space reachability from local control. Our framework utilizes a high-level model to sample feasible transitions across the state manifold and a low-level world model as an inverse dynamics solver to ground these transitions into precise actions. This allows the planner to identify reachable subgoals without being constrained by low-level execution details. By prioritizing local physical feasibility over global behavioral patterns, ID-WM synthesizes novel long-horizon plans from short segments, transcending the specific behavioral modes seen during training. We demonstrate its effectiveness across three navigation and manipulation suites, achieving a 38.5% improvement over state-of-the-art policy and diffusion-based planners. Crucially, by leveraging this hierarchical structure to mitigate compounding errors, our approach yields a 135% performance gain over single-level world model MPC, particularly in complex, unseen configurations for long-horizon tasks. Videos can be viewed at https://id-wm.github.io.

Architecture of ID-WM

Method

In this paper, we introduce ID-WM, a hierarchical world model with implicit dynamics grounded in the insight that effective planning requires decoupling topological reachability from execution details. ID-WM consists of a high-level diffusion-based world model that only implicitly models the dynamics by outputting states that are within a fixed distance away from the current state, functioning as a neighboring state generator, and a deterministic low-level model conditioned on both current state and action, acting as an inverse dynamics solver. Conditioned on the current state and a Gaussian noise, the high-level model leverages the generative power of diffusion to map a tractable unit Gaussian latent space to the manifold of feasible neighboring states. This formulation allows us to treat the noise as an implicit high-level action, enabling the planner to search over a well-behaved distribution instead of complex, discontinuous skill spaces to reason purely about "where to go," while the low-level model recovers the precise executable actions for "how to get there."

Diverse high-level planning of ID-WM

Long-Horizon Planning with ID-WM

*For all the images and videos below, the top row shows ground truth rollouts in the environment, while the bottom row presents world model-imagined rollouts. The images on the right represent the goal states in both cases.

PushT Random State

PushT Fix Goal

Wall

U-Maze

Medium Maze

Large Maze

Giant Maze

Comparing Planning Performance with Baselines

*For the Wall, Push-T, Giant Maze, Large Maze, Medium Maze, and U-Maze environments, we randomly sample initial and goal states from each environment's state space. For Push-T, the fixed-goal setting samples a random initial state while using the environment's fixed goal location.