NAD Metric Implementation in Evaluation Code

[KJW988]

Hello, thank you for sharing your research code.

I noticed that the current evaluation code does not include the calculation of NAD. Do you have any plans to add it?
Also, regarding the NAD results presented in the paper’s result table—were they computed using the Benign trajectory as the ground truth?

Thank you again for the great research.

[KJW988]

(+) If the NAD values in the paper’s result table were indeed calculated using the Benign trajectory as the ground truth, did you construct a separate evaluation dataset for these experiments? If so, how did you handle cases where the Benign trajectory failed?
If possible, could you please provide more detailed information about the NAD metric used in the result table? Thank you again for your great work.

[William-wAng618]

Thank you for your interest in our work, and sorry for this late response.

To address your questions:

1. The implementation can be found in the calculate_relative_distance method within UADA.py. The discrepancy stems from code development preceding the final naming conventions in the paper. While the code retains the earlier label (calculate_relative_distance), it implements the exact NAD metric described in the manuscript.

2. Details about NAD's calculation. You are correct that the NAD was not computed using benign trajectories. Instead, it was calculated using the test dataset (e.g., adversarial patches generated on LIBERO were evaluated on the LIBERO test set, and similarly for Bridge v2). This approach ensures label reliability since the test trajectories are derived from human demonstrations.

If you have further questions, please don’t hesitate to reach out. If I don’t respond within a week, feel free to email me directly at wangtaowen3@gmail.com.

Best,
Taowen

[KJW988]

Hello, thank you for your response.

1. I initially thought that the cal_UAD function implemented in UADA.py was the same as the NAD metric described in the paper. Based on what you mentioned, if the calculate_relative_distance method in UADA.py is the actual implementation of NAD, could you please clarify the differences among (1) the UAD returned by the weighted_loss function, (2) the value returned by the cal_UAD function, and (3) the value returned by the calculate_relative_distance function?

2. I also noticed that there are slight variations in code style and implementation depending on the attack method. Would it be possible to consistently compute NAD across all attack methods using the calculate_relative_distance method in UADA.py as a basis?

3. When you mentioned that NAD was evaluated using the test dataset, should I understand this as meaning that the evaluation was conducted in a fixed testing environment by setting the test seed during testing?

I learned a lot by going through both the code and the paper.
Thank you again for your detailed response.

[William-wAng618]

Sure, let me explain for you.

1. Generally, cal_UAD and calculate_relative_distance are doing the same thing: calculate current prediction to the maximum distance boundary (e.g., [-1, 1], if the action prediction is 0.5, we will calculate this distance with boundary -1 - indicate the worst action adversarial case).

   For implementation, cal_UAD is to speed up NAD calculation with O(1) complexity (roughly), as you may see we use 'torch.where'. For calculate_relative_distance, it is an early slow version calculate NAD point-by-point - we use 2 for loop ~ O(n^2). But we still keep this implementation because it monitors the NAD for each DoF separately.

   Thus, for your question (1)-(2) the UAD returned by cal_UAD is a multi-DoFs averaged NAD. (3) the value returned by calculate_relative_distance is a separate NAD (separate for each DoF). - For different granularity observations, but logically same.

2. Yep, you can use calculate_relative_distance, but it's quite a bit slow, and if you attack more than one DoFs, you need to manually avg them.

3. We consider the val in UADA.py as a fixed test, and we read out the NAD metric in val during training (sorry for the previous ambiguous expression). For the seed, even we try to fix as 42, as you may see in UADA_wrapper.py (line 15-23), somehow it remains some randomness based on our observation.

Thank you for your interest in our work, may these reply can help you.
And if you have further questions, please feel free to ask.