Audio-driven video-to-video support

[xuyangcao]

Our model is able to conduct audio-driven video editing, see this zero shot inference demo :

demo.mp4

The video has been blurred due to copyright reasons.

Code will be released later.

[xuyangcao]

video2video-demo2.mp4

source video: https://civitai.com/images/42887071

[xuyangcao]

video2video-demo2.mp4
source video: https://civitai.com/images/42887071

and the paste-back result:

video-2-video-demo_03.mp4

[xuyangcao]

this looks legit! does it currently only support Chinese audio as motion_generator_hubert_chinese.pt is the only motion generator weights downloadable from hf right now? would it somehow work for non Chinese audio and how accurate those are? thanks!

The term 'motion_generator_hubert_chinese' means the motion generator is trained using the hubert-base-Chinese audio-encoder, and it support non-Chinese audio. Currently, the lip-sync performs better in Chinese and English audios, as our training dataset mainly contains Chinese and English videos.