Add Qwen2.5, Gemma-1, Phi-2 #369

cglagovichTT · 2024-10-31T13:14:11Z

I'm adding support for a few more language models which are high priority for the models team.

Note that the Gemma model requires a huggingface-cli login. In addition, Qwen2-VL is marked to be skipped since the transformers version is < 4.45.

@kevinwuTT could you help me understand why the Qwen and Gemma tests fail like this? It doesn't look like these tests are any different from Phi-2 and GPT-Neo, which pass.

E                   TypeError: Gemma-1.1-2b-it - Torch run with bypass compilation failed. Please check whether `model` or `model.generate` is passed to `record_property`.

cglagovichTT · 2024-10-31T18:51:14Z

fyi @uaydonat

npetrovic-tenstorrent · 2025-03-18T21:30:26Z

fyi @uaydonat

Dynamo surpression:
import torch._dynamo
torch._dynamo.config.suppress_errors = True

For Qwen2 /2.5 test, it returns many warnings of this type. I also attach full log.

[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] torch._dynamo.exc.BackendCompilerFailed: backend=‘ttnn_backend’ raised:
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] RuntimeError: aten::clone() Expected a value of type ‘Tensor’ for argument ‘self’ but instead found type ‘SymInt’.
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] Position: 0
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] Value: s0
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] Declaration: aten::clone(Tensor self, *, MemoryFormat? memory_format=None) -> Tensor
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING] Cast error details: Unable to cast s0 to Tensor
[2025-03-18 21:20:13,936] torch._dynamo.convert_frame: [WARNING]

To reproduce:
pytest [tests/models/qwen/test_qwen.py -s &> /tmp/qwen2.log
log_qwen2.log

No surpression
log_qwen_no_surpression.log
pytest [tests/models/qwen/test_qwen.py -s &> /tmp/qwen_no_surpression.log

---------------------------- Captured log teardown -----------------------------
INFO root:conftest.py:137 Test run completed successfully in 26174.84 ms.
DEBUG root:conftest.py:143 Processing runtime metrics.
INFO root:conftest.py:158 Runtime metrics saved to metrics/Qwen2-VL-2B/original-run_time_metrics.pickle.
DEBUG root:conftest.py:169 Compiling model with ttnn backend.
ERROR root:conftest.py:223 Compilation failed.
Traceback (most recent call last):
File "/home/ubuntu/pytorch2.0_ttnn/tests/conftest.py", line 182, in compile_and_run
outputs_after = model_tester.test_model(as_ttnn=True, option=option)
File "/home/ubuntu/pytorch2.0_ttnn/tests/utils.py", line 145, in test_model
return self.test_model_eval(as_ttnn, option)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/pytorch2.0_ttnn/tests/utils.py", line 137, in test_model_eval
outputs = self.run_model(model, inputs)
File "/home/ubuntu/pytorch2.0_ttnn/tests/utils.py", line 71, in run_model
return model(inputs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 489, in _fn
return fn(*args, **kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 655, in catch_errors
return callback(frame, cache_entry, hooks, frame_state)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 727, in _convert_frame
result = inner_convert(frame, cache_entry, hooks, frame_state)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
compiled_product = _compile(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 665, in _compile
raise InternalTorchDynamoError(str(e)).with_traceback(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 646, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
r = func(*args, **kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 562, in compile_inner
out_code = transform_code_object(code, transform)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
transformations(instructions, code_options)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
return fn(*args, **kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 527, in transform
tracer.run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2128, in run
super().run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
and self.step()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/lazy.py", line 90, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 294, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2261, in inline_call
return cls.inline_call(parent, func, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2376, in inline_call
tracer.run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
and self.step()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2261, in inline_call
return cls.inline_call(parent, func, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2376, in inline_call
tracer.run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
and self.step()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1213, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 294, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2261, in inline_call
return cls.inline_call(parent, func, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2376, in inline_call
tracer.run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
and self.step()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1213, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2261, in inline_call
return cls.inline_call(parent, func, args, kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/dynamo/symbolic_convert.py", line 2376, in inline_call
tracer.run()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
and self.step()
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
result = BuiltinVariable(getattr).call_function(
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 651, in call_function
result = handler(tx, *args, **kwargs)
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 1252, in call_getattr
member = obj.value.dict[name]
torch._dynamo.exc.InternalTorchDynamoError: is_compiling

npetrovic-tenstorrent · 2025-03-20T11:12:33Z

n with bypass compilation failed. Please check whether model or model.generate is passed to `record_proper

After more detailed investigation - it seems that for both Qwen2 and Qwen2.5 there is some problem with torch.compile itself, as even in case when TTNN backend is not used for compilation, the same error occurs. Looks like torch.compile does not behave properly with either the qwen model or newer versions of transformers. transformers>=4.45.0 .

kevinwuTT · 2025-03-20T16:12:39Z

For this particular error concerning is_compiling, there is a difference in implementation between v4.38.0 and v4.45.0:

v4.38.0
https://github.com/huggingface/transformers/blob/08ab54ada594f8f4cc1e458b1217bf8c53f04dd8/src/transformers/utils/import_utils.py#L529-L537

def is_torchdynamo_compiling():
    if not is_torch_available():
        return False
    try:
        import torch._dynamo as dynamo  # noqa: F401

        return dynamo.is_compiling()
    except Exception:
        return False

v4.45.0
https://github.com/huggingface/transformers/blob/2ef31dec1676249d26044a8aa8abe33dbecf0d10/src/transformers/utils/import_utils.py#L734-L750

def is_torchdynamo_compiling():
    if not is_torch_available():
        return False

    # Importing torch._dynamo causes issues with PyTorch profiler (https://github.com/pytorch/pytorch/issues/130622)
    # hence rather relying on `torch.compiler.is_compiling()` when possible (torch>=2.3)
    try:
        import torch

        return torch.compiler.is_compiling()
    except Exception:
        try:
            import torch._dynamo as dynamo  # noqa: F401

            return dynamo.is_compiling()
        except Exception:
            return False

You can try to investigate why the newer implementation is causing problems with torch.compile.

Add Qwen2.5, Gemma-1, Phi-2

74c15e4

npetrovic-tenstorrent added a commit that referenced this pull request Mar 18, 2025

#369: Add qwen, qwen2, vilt tests

7539730

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Qwen2.5, Gemma-1, Phi-2 #369

Add Qwen2.5, Gemma-1, Phi-2 #369

cglagovichTT commented Oct 31, 2024

cglagovichTT commented Oct 31, 2024

npetrovic-tenstorrent commented Mar 18, 2025 •

edited

Loading

npetrovic-tenstorrent commented Mar 20, 2025

kevinwuTT commented Mar 20, 2025

Add Qwen2.5, Gemma-1, Phi-2 #369

Are you sure you want to change the base?

Add Qwen2.5, Gemma-1, Phi-2 #369

Conversation

cglagovichTT commented Oct 31, 2024

cglagovichTT commented Oct 31, 2024

npetrovic-tenstorrent commented Mar 18, 2025 • edited Loading

npetrovic-tenstorrent commented Mar 20, 2025

kevinwuTT commented Mar 20, 2025

npetrovic-tenstorrent commented Mar 18, 2025 •

edited

Loading