Pose Evaluation

This repository provides tools and metrics for the automatic evaluation of pose sequences, specifically designed for sign language applications.

This includes metrics proposed by Ham2Pose¹ as well as custom-developed metrics unique to our approach. We recognize the distinct challenges in evaluating single signs versus continuous signing, and our methods reflect this differentiation.

Usage

Install the package:

pip install git+https://github.com/sign-language-processing/pose-evaluation.git

Create a metric:

from pose_evaluation.metrics.distance_metric import DistanceMetric
from pose_evaluation.metrics.dtw_metric import DTWDTAIImplementationDistanceMeasure
from pose_evaluation.metrics.pose_processors import *

DTWp = DistanceMetric(
    name="DTWp",
    # Select distance measure
    distance_measure=DTWDTAIImplementationDistanceMeasure(),
    # Provide pose processing pipeline
    pose_preprocessors=[
        TrimMeaninglessFramesPoseProcessor(),
        GetHandsOnlyHolisticPoseProcessor(), # select only the hands
        FillMaskedOrInvalidValuesPoseProcessor(masked_fill_value=10.0), # fill masked values with 10.0
        ReducePosesToCommonComponentsProcessor() # reduce pairs of poses to common components
    ],
)

Evaluate two pose sequences:

from pose_format import Pose

with open("hypothesis.pose", "rb") as f:
    hypothesis = Pose.read(f)

with open("reference.pose", "rb") as f:
    reference = Pose.read(f)

DTWp.score(hypothesis, reference)

Demo Notebook

For a demonstration of how to use the package, see the demo notebook:

Demonstrates:

• How to reconstruct the metrics from our paper.
• How to use them to score poses, with signatures.
• How to score poses with different lengths, missing/undetected keypoints, or different keypoint formats.

Contributing

Please make sure to run make format before submitting a pull request.

Cite

If you use our toolkit in your research or projects, please consider citing the work.

@inproceedings{Jiang2025PoseEvaluation,
  title={Meaningful Pose-Based Sign Language Evaluation},
  author={Zifan Jiang, Colin Leong, Amit Moryossef, Oliver Cory, Maksym Ivashechkin, Neha Tarigopula, Biao Zhang, Anne Göhring, Annette Rios, Rico Sennrich, Sarah Ebling},
  booktitle={Conference on Machine Translation},
  year={2025},
  url={https://github.com/sign-language-processing/pose-evaluation}
}

Contributions

• Zifan, Colin, and Amit developed the evaluation metrics and tools. Zifan did correlation and human evaluations, Colin did automated meta-eval, KNN, etc.
• Colin and Amit developed the library code.
• Zifan, Anne, and Lisa conducted the qualitative and quantitative evaluations.

References

Footnotes

1. Rotem Shalev-Arkushin, Amit Moryossef, and Ohad Fried. 2022. Ham2Pose: Animating Sign Language Notation into Pose Sequences. ↩