Frames are not smooth

[sumansid]

Is there any way to smooth out the frames? When I run it, the change of frames is very visible and the cutoff from one expression to another is harsh.

[xuyangcao]

Is there any way to smooth out the frames? When I run it, the change of frames is very visible and the cutoff from one expression to another is harsh.

Yes, in many cases, the continuity between frames is not smooth enough. Currently, we have added some smoothing loss during training and applied EMA smoothing to the inferred motion sequence during post-processing, but the results are not yet optimal. We are considering releasing the training code to facilitate community collaboration for further optimization.

[sumansid]

cool, thanks for the reply. Are the frames not smooth because of too many motions in every frame generated by your model before running it through liveportrait? Have you tried lip-sync only ? or lip-sync and eyes only ?

[xuyangcao]

cool, thanks for the reply. Are the frames not smooth because of too many motions in every frame generated by your model before running it through liveportrait? Have you tried lip-sync only ? or lip-sync and eyes only ?

Good question. We smoothed the head motions before running the LivePortrait model.

I think another possible reason for residual inconsistencies might be that, during training and inference, we treated different dimensions of the motion sequence as independent units rather than as integrated sequences, as they may have correlations with each other.

We have tried lip-sync only and used head motions from real-life videos, and it works better than predicting whole motions.

[wvinzh]

I have also done work similar to JoyVASA, and the results I have obtained may not be smooth enough. Sometimes there may even be mutations in Pose. I have tried smoothing the loss, but almost none of them have effectively improved it. Further experiments revealed that this was not caused by changes in Pose, but rather by the delta of the expression, as even if I used a predefined Pose to only change the delta of the expression, this situation would still occur. I speculate that the motion data extracted by LivePortrait is not very good, and there may be some flaws in certain KeyPoints in LP, which needs further verification. Looking forward to the author solving this problem.

[sumansid]

Have you tried smoothing the keypoints ?

[wvinzh]

Have you tried smoothing the keypoints ?

Yes, during training, I tried several levels of Smooth, including the Coefficients Smooth, the Keypoints Smooth, and the Pose Smooth, but the problem still persists. During inference, I directly reuse the smooth from LivePortrait. videoHere are my results.

[sumansid]

wow, your results actually look much better than mine.

[sumansid]

I think the main issue is due to sudden pose or keypoint changes, I wonder if there's a way to smooth them out without affecting lipsync.

[xuyangcao]

I have also done work similar to JoyVASA, and the results I have obtained may not be smooth enough. Sometimes there may even be mutations in Pose. I have tried smoothing the loss, but almost none of them have effectively improved it. Further experiments revealed that this was not caused by changes in Pose, but rather by the delta of the expression, as even if I used a predefined Pose to only change the delta of the expression, this situation would still occur. I speculate that the motion data extracted by LivePortrait is not very good, and there may be some flaws in certain KeyPoints in LP, which needs further verification. Looking forward to the author solving this problem.

Exactly, I also think the motion data is not good enough, see these issues: 1. KwaiVGI/LivePortrait#439 2. KwaiVGI/LivePortrait#433