KeyError in evaluate.py on 'map_names' and 'all_other_agents_curr_speed' in Lyft data settings

[jxmmy7777]

Hi team,

Thank you for releasing the code! I am currently testing the evaluate.py script in the Lyft data settings. However, I encountered multiple errors during the execution of evaluate.py. I would like to understand if any modifications are required when using the Lyft data settings or if the configuration needs to be changed.

Steps to Reproduce

I first followed the instructions to train a spatial planner and a multiagent predictor.

To reproduce the error, execute the following command:

python scripts/evaluate.py \
  --results_root_dir results/ \
  --dataset_path /mnt/12TBSeagate/lyft_prediction \
  --env l5kit \
  --ckpt_yaml evaluation/BITS_example.yaml \
  --eval_class HierAgentAware \
  --render

The first error I faced is:

File "/mnt/12TBSeagate/***/traffic-behavior-simulation/scripts/evaluate.py", line 427, in <module>
    run_evaluation(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/scripts/evaluate.py", line 147, in run_evaluation
    stats, info, renderings, adjust_plans, trace = rollout_episodes(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/utils/env_utils.py", line 286, in rollout_episodes
    action = policy.get_action(obs_torch, step_index=counter)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 266, in get_action
    ego_action, ego_action_info = self.ego_policy.get_action(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 234, in get_action
    action, action_info = self.policy.get_action(obs, **kwargs)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 175, in get_action
    return self.model.get_action(obs, **self.action_kwargs, **kwargs)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 107, in get_action
    _, action_info = self.sampler.get_action(obs)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 64, in get_action
    actions_tiled, _ = self.controller.get_action(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/algos/multiagent_algos.py", line 132, in get_action
    relevant_obs = {k: obs_dict[k] for k in relevant_keys}
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/algos/multiagent_algos.py", line 132, in <dictcomp>
    relevant_obs = {k: obs_dict[k] for k in relevant_keys}
KeyError: 'all_other_agents_curr_speed'

If I eliminate this keys, then the error will occured in maps.

File "/mnt/12TBSeagate/***/traffic-behavior-simulation/scripts/evaluate.py", line 427, in <module>
    run_evaluation(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/scripts/evaluate.py", line 147, in run
evaluation
    stats, info, renderings, adjust_plans, trace = rollout_episodes(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/utils/env_utils.py", line 286, in rollout_episodes
    action = policy.get_action(obs_torch, step_index=counter)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 266, in get_action
    ego_action, ego_action_info = self.ego_policy.get_action(
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 234, in get_action
    action, action_info = self.policy.get_action(obs, **kwargs)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 175, in get_action
    return self.model.get_action(obs, **self.action_kwargs, **kwargs)
  File "/mnt/12TBSeagate/***/traffic-behavior-simulation/tbsim/policies/wrappers.py", line 123, in get_action
    for map_name in obs["map_names"]:
KeyError: 'map_names'

Please assist me in resolving these errors. Thank you!

[chenyx09]

Hi,
Thanks for pointing this out, there are indeed bugs when running BITS with L5kit. I just pushed a fix and please let me know if the issue is still there. However, it should be noted that L5kit is no longer maintained and we normally don't recommend using it for simulation or training. In the future, almost all the datasets supported by TBSIM will be through trajdata.

[jxmmy7777]

Thanks for your reply. The lyft version works fine now. However, in the evaluate.py of nusc version, the error message indicates a KeyError with the key 'all_other_agents_curr_speed'. The specific traceback is as follows:
Exception has occurred: KeyError 'all_other_agents_curr_speed' File "/mnt/12TBSeagate/*/traffic-behavior-simulation/tbsim/utils/batch_utils.py", line 209, in batch_to_raw_all_agents data_batch["all_other_agents_curr_speed"] ... File "/mnt/12TBSeagate/*/traffic-behavior-simulation/scripts/evaluate.py", line 428, in <module> run_evaluation( KeyError: 'all_other_agents_curr_speed'
This error seems to originate from the "batch_to_raw_all_agents" function in the "batch_utils.py" file. It is encountered while executing the "run_evaluation" function in the "evaluate.py" script. (train.py works fine with nuscenes)

Thank you for your support and assistance!