Reproducing Video Generation from RealEstate10K as in Fig. 3

[sixiaozheng]

I would like to express my sincere appreciation for your impressive work. The approach and results presented in your paper are inspiring, especially the generated videos that align well with the input sequences.

I have a question regarding reproducing the video generation process using RealEstate10K as depicted in Fig. 3 of your paper. Specifically, I would like to know how I can take the first frame of a RealEstate10K video and the corresponding camera pose sequence as input, render the sequence of frames, and then use the diffusion model to generate the final video.

Could you provide some guidance or example code on how to proceed with this pipeline?

[Drexubery]

Hi, thanks for your interest in our work.

We use DUSt3R to process a video clip of 25 frames, then the camera pose and point cloud of every frame can be obtained.

For your test video, you can pass the video frame (must be 25 frames) folder into run_sparse.sh and delete this line

ViewCrafter/utils/pvd_utils.py

Line 236 in f55d64b

c2ws = interp_traj(c2ws,n_inserts= ns,device=device)

Then select the frame you want through a simple index operation here

ViewCrafter/viewcrafter.py

Line 64 in f55d64b

pts3d = to_numpy(pts3d)

Then it should produce a render result align with your test video.

[vidit98]

Hi, thanks for your work. If we want to evaluate model on single view task then using Dust3r to estimate point cloud using all the 25 frames might be unfair right? Because it would be easy to estimate point cloud from 25 frames than just one input frame. I understand the part you need to run Dust3r to get the reference camera trajectory.

Here my assumption is that Fig 3 and Table 1 report results for single image conditioned novel view synthesis