Towards Physically Plausible Video Generation via VLM Planning

Xindi Yang1*, Baolu Li2*, Yiming Zhang2, Zhenfei Yin4, 5✉, Lei Bai3✉, Liqian Ma6, Zhiyong Wang5,
Jianfei Cai1, Tien-Tsin Wong1, Huchuan Lu2, Xu Jia2✉
1Monash University,  2Dalian University of Technology,  3Shanghai Artificial Intelligence Laboratory 4Oxford University,  5The University of Sydney,  6ZMO AI
* Equal Contribution   ✉ Corresponding Author  
arXiv Code(Coming Soon)
Showcases Comparisons Overview Abstract

Showcases

Prompt: A wooden ball collides with an iron cube.
Prompt: A rubber ball is initially suspended in midair, then falls to the ground and bounces until it comes to a stop.
Prompt: Pouring water into a clear glass.
Prompt: A clear acrylic sheet placed on a wooden table with a small dollop of red paint. A rotating paintbrush attached to a rotating platform rotates clockwise and goes through the paint. Static shot with no camera movement.
Prompt: A tennis ball attached to a magnet and string is hanging in front of a mirror and creating an illusion of two tennis balls. The string lowers the ball slowly. Static shot with no camera movement.
Prompt: A timelapse of a water-filled soft towel being forcefully squeezed by hand, with the pressure intensifying rapidly over time.
Prompt: A timelapse captures the reaction as concentrated sulfuric acid is poured onto a sponge.
Prompt: On the left side of the desk, there is a turntable with a yellow mug on it. On the right side, a mirror clearly reflects the mug's image, while the turntable rotates clockwise.
◀
▶

 

Prompt: Two small balls on the table move towards each other and collide.
Prompt: Clean water is poured into a transparent glass.
Prompt: A basketball fell onto the clean floor
◀
▶

 

Comparisons with baseline methods

◀
▶

 

OverView

Our pipeline consists of two stages. In the first stage, the VLM generates a coarse-grained, physically plausible motion trajectory based on the provided input conditions. In the second stage, we synthesize a fake motion video using the predicted trajectory. We then extract the optical flow from this video and convert it into structured noise. These conditions are fed into an image-to-video diffusion model, and ultimately generates a physically plausible video.

 

Abstract

Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos and drawing the attention of the community in their potential as world simulators. However, despite their capabilities, VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics, resulting in incorrect dynamics and event sequences. To address this limitation, we propose a novel two-stage image-to-video generation framework that explicitly incorporates physics. In the first stage, we employ a Vision Language Model (VLM) as a coarse-grained motion planner, integrating chain-of-thought and physics-aware reasoning to predict a rough motion trajectories/changes that approximate real-world physical dynamics while ensuring the inter-frame consistency. In the second stage, we use the predicted motion trajectories/changes to guide the video generation of a VDM. As the predicted motion trajectories/changes are rough, noise is added during inference to provide freedom to the VDM in generating motion with more fine details. Extensive experimental results demonstrate that our framework can produce physically plausible motion, and comparative evaluations highlight the notable superiority of our approach over existing methods.