Question about size when training

[LiuMY13]

RuntimeError: Caught RuntimeError in DataLoader worker process 2.
Original Traceback (most recent call last):
File "/wiz-llm-storage/anaconda3/envs/h2_nemo/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/wiz-llm-storage/anaconda3/envs/h2_nemo/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
return self.collate_fn(data)
File "/wiz-llm-storage/anaconda3/envs/h2_nemo/lib/python3.10/site-packages/speechtokenizer/trainer/dataset.py", line 23, in collate_fn
output = pad_sequence(datum, batch_first=True)
File "/wiz-llm-storage/anaconda3/envs/h2_nemo/lib/python3.10/site-packages/torch/nn/utils/rnn.py", line 400, in pad_sequence
return torch._C._nn.pad_sequence(sequences, batch_first, padding_value)
RuntimeError: The size of tensor a (768) must match the size of tensor b (769) at non-singleton dimension 2

I followed all the steps and used the Hubert, which is the same version of the paper.

[LioYao]

I also encountered the same problem, have you solved it?

[ggiggit]

I also had this error:
RuntimeError: The size of tensor a (716) must match the size of tensor b (772) at non-singleton dimension 1

[alonhelvits]

This comes from the fact that the datum has a batch dimension and the pad_sequence() can't handle this.

in speechtokenizer/trainer/dataset.py,

replace line 22:
output = pad_sequence(datum, batch_first=True)

With:
tensor_list = [d.squeeze(0) for d in datum]
output = pad_sequence(tensor_list, batch_first=True)