Lip Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation

[rittik9]

🚀 Feature

I would like to propose incorporating an essential evaluation metric for 3D talking heads into the TorchMetrics library: Lip Vertex Error (LVE) . This metric is widely used in speech-driven facial animation research to assess lip synchronization accuracy respectively.

Motivation

Current TorchMetrics offerings lack dedicated metrics for evaluating 3D talking heads, particularly in assessing the quality of lip synchronization . I think this metric also fits in multimodal folder of this library.

Pitch

The Lip Vertex Error (LVE) metric evaluates the quality of lip synchronization in 3D facial animations by measuring the maximum Euclidean distance (L2 error) between corresponding lip vertices of the generated and ground truth meshes for each frame. This metric assesses how accurately the animated lip movements align with the intended speech input.

References

Paper: MeshTalk

Additional context

I would like to open a PR for the same.

[SkafteNicki]

@rittik9 sound good to me, feel free to open a PR !