dataset error

[an1iu]

Traceback (most recent call last):
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 345, in
main()
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 341, in main
valid(args, train_env, val_envs, rank=rank)
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 262, in valid
agent.test(
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent_base.py", line 211, in test
super().test(iters=iters)
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent_base.py", line 67, in test
for traj in self.rollout(**kwargs):
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent.py", line 411, in rollout
llava_a_t=self.infer_llava_nav(obs,traj,nav_inputs)
File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent.py", line 646, in infer_llava_nav
image=Image.open("/home/taizun/Data2/aliu/Taga-VLM/data/mp3d_data/"+image_file).convert("RGB")
File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/PIL/Image.py", line 3513, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/taizun/Data2/aliu/Taga-VLM/data/mp3d_data/JeFG25nYj2p/matterport_skybox_images/b2f31140a9d0482096da4ac481fb8a56_skybox_small_cropped.jpg'

No such file or directory:mp3d_data/JeFG25nYj2p/matterport_skybox_images/b2f31140a9d0482096da4ac481fb8a56_skybox_small_cropped.jpg

May I ask if the dataset is incomplete?

[Nagi-ovo]

Hi, we are working on this!

Taga-VLM/README.md

Line 13 in 76d7aed

> **⚠️ IMPORTANT:** Code cleaning and preprocessed data release are in progress. Full release coming soon!

[36171221]

@an1iu Thank you for your interest in our project. We have fixed this error.

[an1iu]

hi,i meet two new error

1

FileNotFoundError: [Errno 2] No such file or directory: '/home/taizun/Data2/aliu/Taga-VLM/data/view_images_hm3d/JeFG25nYj2p/matterport_skybox_images/b2f31140a9d0482096da4ac481fb8a56_skybox_small.jpg'

Should I replace “hm3d_views_file” with “mp3d_views_file” on line 639 in map_nav_src/r2r_llava/agent.py ？

2

Traceback (most recent call last):                                                                                                                                      
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 345, in <module>
    main()
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 341, in main
    valid(args, train_env, val_envs, rank=rank)
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/main_nav_llava.py", line 262, in valid
    agent.test(
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent_base.py", line 212, in test
    super().test(iters=iters)
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent_base.py", line 67, in test
    for traj in self.rollout(**kwargs):
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent.py", line 411, in rollout
    llava_a_t=self.infer_llava_nav(obs,traj,nav_inputs)
  File "/home/taizun/Data2/aliu/Taga-VLM/map_nav_src/r2r_llava/agent.py", line 683, in infer_llava_nav
    cont = self.llava_model.generate(
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/taizun/Data2/aliu/Taga-VLM/llava/model/language_model/llava_qwen.py", line 148, in generate
    return super().generate(position_ids=position_ids, attention_mask=attention_mask, inputs_embeds=inputs_embeds, **kwargs)
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/transformers/generation/utils.py", line 1527, in generate
    result = self._greedy_search(
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
    outputs = self(
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/taizun/anaconda3/envs/tagavlm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/taizun/Data2/aliu/Taga-VLM/llava/model/language_model/llava_qwen.py", line 110, in forward
    return super().forward(
TypeError: forward() got an unexpected keyword argument 'graph_sprels'

should i remove line 121 in Taga-VLM/llava/model/language_model/llava_qwen.py？

[an1iu]

Perhaps I need to wait for the final version😁

[36171221]

We apologize for the inconvenience, and we are still working on the code checks at full speed.
For Error 1: You do need to replace hm3d_views_file with mp3d_views_file.
For Error 2: We believe this issue is most likely caused by not installing our modified version of transformers-4.40.0. We have made custom changes to the file TagaVLM/transformers-4.40.0/src/transformers/models/qwen2/modeling_qwen2.py in our project. The official transformers library does not include the relevant implementation for graph_sprels, which is the root cause of Error 2.

[Nagi-ovo]

@an1iu should be fixed now!