Skip to content

Latest commit

 

History

History
56 lines (36 loc) · 3.08 KB

README.md

File metadata and controls

56 lines (36 loc) · 3.08 KB

Perception-as-Control

Official implementation of "Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation"

Project page | Paper | Video | Online Demo

Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation
Yingjie Chen, Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo

💡 Abstract

Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via the same 2D motion representations or different control signals, while they still struggle in supporting collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user intentions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive, consistent visual changes. Then, the proposed framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed method.

🔥 Updates

(2025-01-09) The project page, demo video and technical report are released. The full paper version with more details is in process.

🎥 Demo

camera-only_github.mp4
object-only_github.mp4
collaborative_github.mp4
applications_github.mp4

For more details, please refer to our project page.

📑 TODO List

  • Release inference code and checkpoints
  • Provide a Gradio demo
  • Release training code

🔗 Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{chen2025perception,
  title={Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation},
  author={Chen, Yingjie and Men, Yifang and Yao, Yuan and Cui, Miaomiao and Bo, Liefeng},
  journal={arXiv preprint arXiv:2501.05020},
  website={https://chen-yingjie.github.io/projects/Perception-as-Control/index.html},
  year={2025}}