AI that imagines whole worlds: how world models like Genie 4 and Cosmos 3 are training robots and self-driving cars

A language model predicts the next word. A world model predicts the next moment of a world: what you would see, and sometimes hear, if you turned left, opened a door or moved a robot arm.

Within nine days in late 2026, Google DeepMind previewed Genie 4 and NVIDIA released new open Cosmos 3 models. Together they show where this is heading: AI that can generate endless practice worlds for robots, cars and software agents.

Why build a world in a neural network?

Traditional simulators are hand-built: engineers model physics, materials, lighting and every object. That is accurate but slow to extend. A learned world model can produce a new scene from a sentence or a single photo, and vary it endlessly: rain, dusk, a crowded platform, a dropped box.

The catch is that a learned world only approximates physics. So the most useful setups combine both: physics engines for ground truth, world models for realistic appearance and variety.

Genie 4: real-time worlds you can steer with words

Google DeepMind's Genie 4 announcement describes four advances:

• Real-time steering: you navigate with keyboard controls and change the world with text prompts, like changing the weather or adding a person, while it runs.
• Physics and people: simulations of aerodynamics and collisions, and characters who move and react naturally.
• Native audio: ambient sound and, for the first time in the series, interactive dialogue generated with the world.
• Gemini in the loop: a Gemini model watches the scene, breaks goals into steps and checks frames to verify each step, which helps keep longer events coherent.

The robotics angle is explicit. DeepMind shows Gemini 4 Argon breaking a task into steps that Genie 4 renders, simulating Apptronik's Apollo humanoid playing checkers. In a separate robotics post, DeepMind says Argon can generate thousands of varied simulated practice runs from a real camera scene and build a realistic digital twin, which helped train Gemini Robotics 2.5.

DeepMind is upfront about limits: details and object permanence can drift in longer sessions or after many edits, and complex physical interactions, legible text and spatial accuracy are still active research. All Genie 4 output carries a SynthID watermark.

Cosmos 3: open world models for cars and robots

NVIDIA's approach is open weights for developers. Its 5 November 2026 release adds four post-trained Cosmos 3 Nano checkpoints, each with 16B trainable parameters, under the OpenMDW 1.1 licence:

• Transfer-Auto: generates synchronised driving video across up to 11 camera views, plus LiDAR, with controllable weather, lighting and time of day.
• Sim-Auto: a faster, streaming version for interactive driving simulation.
• Sim-Transfer: turns a physics simulator's depth, segmentation and edge maps plus a text prompt into photorealistic video, so the physics stays with the simulator while the look comes from the model.
• Sim-Bimanual: predicts video from two-armed robot actions, controlled by a policy or a joystick.

For a self-driving team, that means you can take one recorded drive and generate the same route in fog, at night or in heavy rain, with camera and LiDAR in agreement, to test how a driving system responds.

NVIDIA's own framing is measured: the work ahead is "to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short."

Where this is already showing up

World models are not only research demos. Cosmos models are part of how Siemens and NVIDIA generate realistic training images for circuit-board inspection at Siemens' Erlangen factory, according to Siemens' October 2026 announcement. The general pattern:

1. Capture a small amount of real data.
2. Generate many realistic variations with a world model.
3. Train perception or control models on real and synthetic data together.
4. Validate on held-out real data before deployment.

What can go wrong

• Drift and hallucinated physics: a model can "imagine" that a box floats or a car fits through a gap. Training on that teaches the wrong lesson.
• Hidden bias: if generated scenes mostly look like the data the model was trained on, rare local conditions, say Indian roads with mixed traffic, may still be under-represented.
• Compute cost: NVIDIA tested its published setup on a single GB200 GPU, which is data-centre class hardware.
• Misuse: photorealistic generated video can be misused, which is why watermarking like SynthID matters.

What this means for you: skills to learn

World models sit at the meeting point of generative AI, computer vision and simulation. Useful skills:

• Generative model basics: diffusion and transformers, at the level of knowing what controls what.
• Synthetic data discipline: mixing real and generated data, and always validating on real data.
• Simulation tools such as Isaac Sim, and an understanding of sensors like cameras and LiDAR.
• Agents that plan and verify: the Gemini-in-the-loop pattern of planning steps and checking results is the same pattern used in LLM agents.
• Evaluation: measuring whether synthetic data actually improved the real-world metric.

Our Applied AI Engineer Bootcamp teaches the agent, evaluation and deployment foundations that this pattern relies on. The world models themselves are open enough that a determined learner can try them, starting with the Cosmos notebooks linked below.