Hi, I encountered some issues while trying to reproduce the model training.
Firstly, I completed the environment configuration according to the requirements of the README, but some APIs may have encountered errors due to changes in the torch-lighting version.
Then, I lowered the version of torch-lightning,
pip install pytorch-lightning==1.9.3
At this point, training can run normally on the CPU.
However, when I try to train on multiple GPUs (3090 * 8), whether using the 'ddp' or 'dp' strategy, unexpected errors occur. For the former, error cannot pick' xxx 'object is reported, while for the latter, error TypeError: can't convert CUDA: 0 device type tensor to numpy' is reported
May I ask why this problem occurs? I don't know if this is still caused by environmental issues. Assuming that the code itself has no problems, please help provide key environmental information that can run normally, such as Python and package versions (Except for the torch-lighting version, all other environments are automatically configured according to the README). Thank you very much!
Hi, I encountered some issues while trying to reproduce the model training.
Firstly, I completed the environment configuration according to the requirements of the README, but some APIs may have encountered errors due to changes in the
torch-lightingversion.Then, I lowered the version of
torch-lightning,At this point, training can run normally on the CPU.
However, when I try to train on multiple GPUs (3090 * 8), whether using the 'ddp' or 'dp' strategy, unexpected errors occur. For the former, error
cannot pick' xxx 'objectis reported, while for the latter, errorTypeError: can't convert CUDA: 0 device type tensor to numpy' is reportedMay I ask why this problem occurs? I don't know if this is still caused by environmental issues. Assuming that the code itself has no problems, please help provide key environmental information that can run normally, such as Python and package versions (Except for the
torch-lightingversion, all other environments are automatically configured according to the README). Thank you very much!