Pandora: Towards General World Model
with Natural Language Actions and Video States

SimWorld

We introduce Pandora, a step towards a General World Model (GWM) that:

• Simulates world states by generating videos across any domains
• Allows any-time control with actions expressed in natural language

On-the-Fly Control with Natural Language

Pandora accepts free-text actions as inputs during video generation to steer the video on the fly. This differs crucially from previous text-to-video models which only allow text prompts at the beginning of the video. The on-the-fly control fulfills the promise of the world model to support interactive content generation and enhance robust reasoning and planning.

Predicting Alternative Futures as You Like

World models simulate alternative futures of the world. Pandora allows you to control the future. Here we show some counterfactual futures – different videos generated from the same initial state but different actions.

Initial state

Future 1

Future 2

Action: Magma erupts from the crater

Action: The sky gets dark

Action: Turn left. There is a white van

Action: Turn right. There is a red car

Action: The man waves his hand

Action: Two men walks towards the microphone

Action: Use spoon to scoop some broccoli

Action: Use spoon to stir the rice

Simulating Worlds across Any Domains

Pandora is capable of generating videos across a wide range of general domains, such as indoor/outdoor, natural/urban, human/robot, 2D/3D, and other scenarios. You may find more videos in the Pandora’s Box gallery.

Learning Actions in One Domain and Using in Another

Instruction tuning with high-quality data allows the model to learn effective action control and transfer to different unseen domains. E.g., Pandora saw the only 2D game Coinrun during training, but can seamlessly apply the learned actions to other 2D games.

Source Domain

Target Domain

Source Domain

Target Domain

Autoregressive Model Yields Longer Videos

Existing diffusion video models typically produce videos of a fixed length. By integrating the video model with the Pandora autoregressive backbone, longer videos with potentially unlimited duration can be generated. We show 8-second videos generated by Pandora, even though our training videos are up to 5 seconds.

Limitation

Pandora as a prelimiary step towards GWM is still limited. It can fail in generating consistent videos, simulating complex scenarios, understanding commonsense and physical laws, and following instructions/actions.

Note

We processed the videos on this website with FLAVR for frame interpolation to make them smoother. No other post-processing used.

If you have concern about the copyright of any image/video on this website, please contact us.

Pandora Team

1
2
3
4
5
6
@article{xiang2024pandora,
  title={Pandora: Towards General World Model with Natural Language Actions and Video States},
  author={Jiannan Xiang and Guangyi Liu and Yi Gu and Qiyue Gao and Yuting Ning and Yuheng Zha and Zeyu Feng and Tianhua Tao and Shibo Hao and Yemin Shi and Zhengzhong Liu and Eric P. Xing and Zhiting Hu},
  journal={arXiv preprint arXiv:2406.09455},
  year={2024}
}