Repository navigation
[BUG]This happened after I ran Colossal after building the environment according to the instructions #3175
Description
🐛 Describe the bug
Traceback (most recent call last):
File "/root/anaconda3/bin/colossalai", line 5, in
from colossalai.cli import cli
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/init.py", line 1, in
from .initialize import (
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/initialize.py", line 18, in
from colossalai.amp import AMP_TYPE, convert_to_amp
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/init.py", line 9, in
from .torch_amp import convert_to_torch_amp
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/torch_amp/init.py", line 9, in
from .torch_amp import TorchAMPLoss, TorchAMPModel, TorchAMPOptimizer
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/torch_amp/torch_amp.py", line 10, in
from colossalai.nn.optimizer import ColossalaiOptimizer
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/init.py", line 1, in
from ._ops import *
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/init.py", line 1, in
from .addmm import colo_addmm
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/addmm.py", line 5, in
from ._utils import GeneralTensor, Number, convert_to_colo_tensor
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/_utils.py", line 8, in
from colossalai.nn.layer.utils import divide
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/init.py", line 7, in
from .moe import *
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/moe/init.py", line 1, in
from .experts import Experts, FFNExperts, TPExperts
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/moe/experts.py", line 8, in
from colossalai.zero.init_ctx import no_shard_zero_decrator
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/init.py", line 7, in
from colossalai.zero.sharded_model.sharded_model_v2 import ShardedModelV2
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/sharded_model/init.py", line 1, in
from .sharded_model_v2 import ShardedModelV2
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/sharded_model/sharded_model_v2.py", line 16, in
from colossalai.gemini.memory_tracer import MemStatsCollector, StaticMemStatsCollector
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/init.py", line 1, in
from .chunk import ChunkManager, TensorInfo, TensorState, search_chunk_configuration
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/chunk/init.py", line 3, in
from .search_utils import classify_params_by_dp_degree, search_chunk_configuration
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/chunk/search_utils.py", line 8, in
from colossalai.gemini.memory_tracer import MemStats, OrderedParamGenerator
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/memory_tracer/init.py", line 6, in
from .static_memstats_collector import StaticMemStatsCollector # isort:skip
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/memory_tracer/static_memstats_collector.py", line 7, in
from colossalai.fx.passes.meta_info_prop import MetaInfoProp
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/init.py", line 3, in
from .passes import MetaInfoProp, metainfo_trace
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/passes/init.py", line 2, in
from .concrete_info_prop import ConcreteInfoProp
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/passes/concrete_info_prop.py", line 10, in
from colossalai.fx.profiler import GraphInfo, profile_function, profile_method, profile_module
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/profiler/init.py", line 4, in
from .opcount import flop_mapping
File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/profiler/opcount.py", line 273, in
aten.upsample_nearest2d_backward.vec: elementwise_flop_counter(0, 1),
File "/root/anaconda3/lib/python3.9/site-packages/torch/_ops.py", line 488, in getattr
raise AttributeError(
AttributeError: The underlying op of 'aten.upsample_nearest2d_backward' has no overload name 'vec'
Environment
cuda kit ：12.1 cuda ：11.7 cudnn：cudnn-local-repo-ubuntu2004-8.8.0.121_1.0-1_amd64 Python 3.9 pytorch：pytorch.org/whl/cu117
Activity
binmakeswell commentedon Mar 20, 2023
Hi @dong49 Which step did you execute according to the instructions from where? Could you please provide detailed reproduction steps? Thanks.
Shikamaru5 commentedon Mar 21, 2023
I'm also seeing this issue, I've got the latest version of PyTorch and Cuda, and Cudnn, and all ChatGPT could tell me was it was likely one of those but when testing they do seem to be working. Going through those scripts, I mean it does seem correct that I can't find a method for the vec overload but I'm not sure why? For me at least it pops up when I start trying to train. literally I type python3 train.py and that is what shows up, except the only difference for me is that it starts at import colossalai. I've also tried uninstalling and reinstalling colossalai, and doing it from the github source, including grabbing the requirements. I've tried uninstalling and reinstalling different versions of PyTorch, but that hasn't changed the issue.
super-dainiu commentedon Mar 22, 2023
I think something happens here.
https://github.com/hpcaitech/ColossalAI/blob/main/colossalai/gemini/memory_tracer/static_memstats_collector.py
What is the usage of MetaInfoProp in this file? Can remove that if the problem cannot be solved.
cc @YuliangLiu0306 @DAC-HAO
super-dainiu commentedon Mar 22, 2023
Also the flop count should be in a import-friendly way..
Shikamaru5 commentedon Mar 22, 2023
After doing some major digging into it, I determined the issue is that PyTorch had a script in torch called aten which is no longer there, so I've sent an issue to the PyTorch GitHub page asking where it is, and what I should do about it, but I may have to download the version that isn't found on the get started on the PyTorch site.
super-dainiu commentedon Mar 23, 2023
sorry for this inconvenience @Shikamaru5, I don't know why but bugs in my depreciated code was called by this function.
Might wait some time before maintainers fix this problem. Or u can manually remove the ATen causing errors in flop_count.py, which does not harm other usages.
Shikamaru5 commentedon Mar 23, 2023
It's alright, the pytorch people should've stated that they'd make that kind of change, imo. I uninstalled PyTorch and reinstalled the previous version using:
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
Now it's working for the most part, I'm struggling with a bit of cudnn stuff but that isn't really colossalai related I don't think.
JazyJiang commentedon Oct 22, 2026
Hi, thanks for reporting this, and sorry it went unanswered for so long. We're going through older issues and closing the ones that no longer apply to the current codebase. The code this issue refers to (
colossalai/amp/torch_amp/__init__.py,colossalai/amp/torch_amp/torch_amp.py) has since been refactored or moved tocolossalai/legacy, so the problem as described no longer applies to the current main branch. We're marking it as stale and will close it in 14 days. If you still see this problem with the latest ColossalAI on PyTorch 2.x, please comment here (or open a new issue) with your environment and a minimal reproduction, and we'll take a look.JazyJiang commentedon Nov 5, 2026
Closing this issue since there has been no response in the 14 days after it was marked as stale. If you still run into this problem with the latest ColossalAI on PyTorch 2.x, please open a new issue using the bug report template, with your environment and a minimal reproduction.