Nan in evaluation

[miraymen]

Hi

When I use your code for evaluation, [vald] Control L2 dist: nan

This happens for your pretrained model. Could you please look into this.

==================== Replication 0 ====================
Time: 2024-05-24 23:24:18.863608
========== Evaluating Matching Score ==========

---> [ground truth] Matching Score: nan
---> [ground truth] R_precision: (top 1): 0.5082 (top 2): 0.6905 (top 3): 0.7897 
---> [vald] Matching Score: nan
---> [vald] R_precision: (top 1): 0.3643 (top 2): 0.5586 (top 3): 0.6846 
Time: 2024-05-24 23:24:23.930179
========== Evaluating Control ==========
---> [vald] Control L2 dist: nan
---> [vald] Skating Ratio: 0.0568
---> [vald] Trajectory Error: (traj_fail_20cm): 0.0039 (traj_fail_50cm): 0.0020 (kps_fail_20cm): 0.0029 (kps_fail_50cm): 0.0010 (kps_mean_err(m)): nan 
Time: 2024-05-24 23:24:25.589527

[ymingxie]

Hi I tried to run the evaluation with the pretrained model using the following command:

bash ./eval_omnicontrol.sh ./save/omnicontrol_ckpt/model_humanml3d.pt 0 100

Here is my log:

==================== Replication 0 ====================
Time: 2024-05-27 17:05:05.680433
========== Evaluating Matching Score ==========
---> [ground truth] Matching Score: 2.9838
---> [ground truth] R_precision: (top 1): 0.5069 (top 2): 0.6972 (top 3): 0.7869 
---> [vald] Matching Score: 5.0465
---> [vald] R_precision: (top 1): 0.4043 (top 2): 0.5889 (top 3): 0.6904 
Time: 2024-05-27 17:05:13.171648
========== Evaluating Control ==========
---> [vald] Control L2 dist: 0.0573
---> [vald] Skating Ratio: 0.0528
---> [vald] Trajectory Error: (traj_fail_20cm): 0.2725 (traj_fail_50cm): 0.0605 (kps_fail_20cm): 0.0598 (kps_fail_50cm): 0.0101 (kps_mean_err(m)): 0.0573 
Time: 2024-05-27 17:05:16.169228
========== Evaluating FID ==========
---> [ground truth] FID: 0.0017
---> [vald] FID: 0.3267
Time: 2024-05-27 17:05:20.134575
========== Evaluating Diversity ==========
---> [ground truth] Diversity: 9.4727
---> [vald] Diversity: 9.8427
!!! DONE !!!
========== Matching Score Summary ==========
---> [ground truth] Mean: 2.9838 CInterval: 0.0000
---> [vald] Mean: 5.0465 CInterval: 0.0000
========== R_precision Summary ==========
---> [ground truth](top 1) Mean: 0.5069 CInt: 0.0000;(top 2) Mean: 0.6972 CInt: 0.0000;(top 3) Mean: 0.7869 CInt: 0.0000;
---> [vald](top 1) Mean: 0.4043 CInt: 0.0000;(top 2) Mean: 0.5889 CInt: 0.0000;(top 3) Mean: 0.6904 CInt: 0.0000;
========== FID Summary ==========
---> [ground truth] Mean: 0.0017 CInterval: 0.0000
---> [vald] Mean: 0.3267 CInterval: 0.0000
========== Diversity Summary ==========
---> [ground truth] Mean: 9.4727 CInterval: 0.0000
---> [vald] Mean: 9.8427 CInterval: 0.0000
========== MultiModality Summary ==========
========== Control_l2 Summary ==========
---> [vald] Mean: 0.0573 CInterval: 0.0000
========== Skating Ratio Summary ==========
---> [vald] Mean: 0.0528 CInterval: 0.0000
========== Trajectory Error Summary ==========
---> [vald](traj_fail_20cm): Mean: 0.2725 CInt: 0.0000; (traj_fail_50cm): Mean: 0.0605 CInt: 0.0000; (kps_fail_20cm): Mean: 0.0598 CInt: 0.0000; (kps_fail_50cm): Mean: 0.0101 CInt: 0.0000; (kps_mean_err(m)): Mean: 0.0573 CInt: 0.0000;

[miraymen]

Generated Dataset Loading Completed!!!
==================== Replication 0 ====================
Time: 2024-05-28 12:28:50.902163
========== Evaluating Matching Score ==========
---> [ground truth] Matching Score: nan
---> [ground truth] R_precision: (top 1): 0.5082 (top 2): 0.6903 (top 3): 0.7897
---> [vald] Matching Score: 4.9373
---> [vald] R_precision: (top 1): 0.4062 (top 2): 0.5781 (top 3): 0.6094
Time: 2024-05-28 12:28:57.285724

I use the same command as yours.

if len(batch) == 7:
      word_embeddings, pos_one_hots, _, sent_lens, motions, m_lens, _ = batch
else:
     word_embeddings, pos_one_hots, _, sent_lens, motions, m_lens, _, _ = batch
               
if torch.isnan(torch.sum(motions)):
      

motion is NaN

I assume this is a problem with the data then?

[ymingxie]

Yeah I guess it's the data issue. Could you please check why the motion is NaN here?

[MichaelaQ]

Hi，I tried to use the same command and I met the question when calculating FID.

[ymingxie]

Hi @MichaelaQ , could you please help check where the NaNs come from? Right now I cannot reproduce this error. Maybe it's a data issue or not...

[MichaelaQ]

Hi @MichaelaQ , could you please help check where the NaNs come from? Right now I cannot reproduce this error. Maybe it's a data issue or not...

Hi,I think there is something wrong with the ground_truth.The number of ground_truth is 4640，I want to know what the classification of ground_truth is like.When I evaluate_matching score on groud_truth,there are two Nan in the ground_truth.

[ymingxie]

Thank you for the information! I'm currently working on it and will inform you once I've identified the cause of the NaNs in the ground truth.

[ymingxie]

I suspect that the issue is caused by the broken data in HumanML3D data. Please check similar issues #190, #110, #18, #33, #88, #95 to see if they can help.

[MichaelaQ]

I suspect that the issue is caused by the broken data in HumanML3D data. Please check similar issues #190, #110, #18, #33, #88, #95 to see if they can help.

Thank you，when I remove 007975.npy and M007975.npy,the problem is solved