"Killed" error in the training process

[leenamx]

Training with a single process on 1 GPUs.
100%█████████████████████████████████████████████████████████| 837/837 [07:11<00:00, 1.94it/s]
train poses bds tensor([-6.9000, 3.9000, 2.0000]) tensor([6.9000, 7.4000, 2.0000])
Killed

I wanna train the block_E of MatrixCity dataset, but error "Killed" come out soon after loaded the dataset.

I see the task manager, the raw still have about 30GB left, so I don't know what cause this problem.

I would be very appreciated if you can help me to slove this problem!

[Ga1ahad]

Training with a single process on 1 GPUs. 100%█████████████████████████████████████████████████████████| 837/837 [07:11<00:00, 1.94it/s] train poses bds tensor([-6.9000, 3.9000, 2.0000]) tensor([6.9000, 7.4000, 2.0000]) Killed

I wanna train the block_E of MatrixCity dataset, but error "Killed" come out soon after loaded the dataset.

I see the task manager, the raw still have about 30GB left, so I don't know what cause this problem.

I would be very appreciated if you can help me to slove this problem!

"Insufficient memory can cause this issue. I completed the training using 256GB of memory, and at least 215GB of memory was occupied during the training process."