Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend unit tests #1967

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Extend unit tests #1967

wants to merge 1 commit into from

Conversation

qododavid
Copy link

Hi, I used cover-agent to write the tests for this PR. It uses an AI-based workflow to add new test cases that increase code coverage.

All AI-generated tests have met these four essential criteria:

  1. Execute successfully
  2. Pass all assertions
  3. Increase overall code coverage
  4. Test previously uncovered behaviors (as specified in the LLM prompt)

Increased Coverage

Coverage Improvement Report
=========================

File: onnxscript/evaluator.py
Coverage improvement: 73.26% → 80.62% (+7.36%)
Newly covered lines: 19
Remaining uncovered lines: 50


File: onnxscript/onnx_opset/_impl/opset15.py
Coverage improvement: 60.66% → 85.25% (+24.59%)
Newly covered lines: 15
Remaining uncovered lines: 9


File: onnxscript/onnx_types.py
Coverage improvement: 86.67% → 95.00% (+8.33%)
Newly covered lines: 10
Remaining uncovered lines: 6


File: onnxscript/ir/serde.py
Coverage improvement: 54.29% → 57.53% (+3.24%)
Newly covered lines: 25
Remaining uncovered lines: 327


File: onnxscript/_internal/version_utils.py
Coverage improvement: 32.00% → 94.00% (+62.00%)
Newly covered lines: 31
Remaining uncovered lines: 3


File: opgen/pygen.py
Coverage improvement: 60.70% → 64.91% (+4.21%)
Newly covered lines: 43
Remaining uncovered lines: 359


File: onnxscript/values.py
Coverage improvement: 68.79% → 71.02% (+2.23%)
Newly covered lines: 7
Remaining uncovered lines: 91


File: onnxscript/tensor.py
Coverage improvement: 77.78% → 98.52% (+20.74%)
Newly covered lines: 28
Remaining uncovered lines: 2


File: onnxscript/type_annotation.py
Coverage improvement: 63.27% → 82.99% (+19.72%)
Newly covered lines: 29
Remaining uncovered lines: 25


File: onnxscript/_internal/param_manipulation.py
Coverage improvement: 48.33% → 100.00% (+51.67%)
Newly covered lines: 31
Remaining uncovered lines: 0


File: onnxscript/rewriter/broadcast_to_matmul.py
Coverage improvement: 90.70% → 100.00% (+9.30%)
Newly covered lines: 8
Remaining uncovered lines: 0


File: onnxscript/rewriter/llama_rule_sets.py
Coverage improvement: 78.33% → 78.89% (+0.56%)
Newly covered lines: 1
Remaining uncovered lines: 38


File: onnxscript/rewriter/collapse_slices.py
Coverage improvement: 63.38% → 97.18% (+33.80%)
Newly covered lines: 24
Remaining uncovered lines: 2


File: onnxscript/rewriter/generic_pattern.py
Coverage improvement: 77.70% → 79.09% (+1.39%)
Newly covered lines: 4
Remaining uncovered lines: 60


File: onnxscript/rewriter/onnxruntime/fused_matmul_rule_sets.py
Coverage improvement: 96.33% → 98.17% (+1.84%)
Newly covered lines: 2
Remaining uncovered lines: 2


File: onnxscript/rewriter/onnxruntime/softmax.py
Coverage improvement: 92.00% → 100.00% (+8.00%)
Newly covered lines: 2
Remaining uncovered lines: 0


File: onnxscript/rewriter/onnxruntime/bfloat16_utils/bfloat16_converter.py
Coverage improvement: 91.11% → 100.00% (+8.89%)
Newly covered lines: 4
Remaining uncovered lines: 0


File: tests/common/testutils.py
Coverage improvement: 78.69% → 98.36% (+19.67%)
Newly covered lines: 12
Remaining uncovered lines: 1


File: onnxscript/version_converter/_version_converter.py
Coverage improvement: 92.02% → 93.09% (+1.07%)
Newly covered lines: 2
Remaining uncovered lines: 13


File: onnxscript/tools/memory_peak.py
Coverage improvement: 79.69% → 85.94% (+6.25%)
Newly covered lines: 8
Remaining uncovered lines: 18


File: onnxscript/tools/benchmark/benchmark_helpers.py
Coverage improvement: 13.12% → 49.50% (+36.38%)
Newly covered lines: 147
Remaining uncovered lines: 204


File: onnxscript/tools/benchmark/export_model.py
Coverage improvement: 52.70% → 54.05% (+1.35%)
Newly covered lines: 1
Remaining uncovered lines: 34


File: onnxscript/tools/transformers_models/mistral.py
Coverage improvement: 60.00% → 97.14% (+37.14%)
Newly covered lines: 26
Remaining uncovered lines: 2


File: onnxscript/tools/transformers_models/phi.py
Coverage improvement: 57.75% → 94.37% (+36.62%)
Newly covered lines: 26
Remaining uncovered lines: 4


File: onnxscript/tools/transformers_models/phi3.py
Coverage improvement: 58.44% → 93.51% (+35.07%)
Newly covered lines: 27
Remaining uncovered lines: 5


File: onnxscript/optimizer/_inliner.py
Coverage improvement: 79.11% → 81.01% (+1.90%)
Newly covered lines: 3
Remaining uncovered lines: 30


File: onnxscript/optimizer/_legacy/_simple_function_folding.py
Coverage improvement: 96.69% → 97.35% (+0.66%)
Newly covered lines: 1
Remaining uncovered lines: 4


File: onnxscript/_legacy_ir/visitor.py
Coverage improvement: 60.10% → 68.49% (+8.39%)
Newly covered lines: 49
Remaining uncovered lines: 184


File: onnxscript/ir/_core.py
Coverage improvement: 41.19% → 52.61% (+11.42%)
Newly covered lines: 140
Remaining uncovered lines: 581


File: onnxscript/ir/_enums.py
Coverage improvement: 94.20% → 100.00% (+5.80%)
Newly covered lines: 4
Remaining uncovered lines: 0


File: onnxscript/ir/_external_data.py
Coverage improvement: 95.65% → 99.13% (+3.48%)
Newly covered lines: 4
Remaining uncovered lines: 1


File: onnxscript/ir/_type_casting.py
Coverage improvement: 55.56% → 88.89% (+33.33%)
Newly covered lines: 12
Remaining uncovered lines: 4


File: onnxscript/ir/_schemas.py
Coverage improvement: 51.48% → 59.92% (+8.44%)
Newly covered lines: 20
Remaining uncovered lines: 95


File: onnxscript/ir/traversal.py
Coverage improvement: 86.11% → 97.22% (+11.11%)
Newly covered lines: 4
Remaining uncovered lines: 1


File: onnxscript/ir/_linked_list.py
Coverage improvement: 92.59% → 100.00% (+7.41%)
Newly covered lines: 8
Remaining uncovered lines: 0


File: onnxscript/ir/tensor_adapters.py
Coverage improvement: 90.91% → 93.94% (+3.03%)
Newly covered lines: 1
Remaining uncovered lines: 2


File: onnxscript/backend/onnx_backend.py
Coverage improvement: 60.49% → 64.81% (+4.32%)
Newly covered lines: 7
Remaining uncovered lines: 57

Why adding unit tests is important

  • Writing and maintaining unit tests can be tedious, especially as codebases grow larger

  • Despite this challenge, unit tests provide key benefits:

    • Enable confident code changes through reliable test coverage
    • Allow faster development by preventing regressions
    • Make refactoring safer with comprehensive test validation
  • Tests serve as executable documentation by:

    • Clearly demonstrating input/output relationships
    • Providing quick feedback during development
    • Presenting behavior changes in readable format

@justinchuby
Copy link
Collaborator

This is pretty cool. If you can isolate changes to onnxscript/ir in a separate PR that’d be great

type_constraint = deduce_type_constraints.TypeConstraint("test_constraint", {"tensor(float)", "tensor(int64)"})
type_constraint.bind_value(value1)
type_constraint.bind_value(value2)
expected_repr = "TypeConstraint(name=test_constraint, type_strs={'tensor(float)', 'tensor(int64)'}, values=['value1', 'value2'])"

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable expected_repr is not used.
graph.register_initializer(unnamed_initializer)

# Attempt to register an initializer that is produced by a node
node = ir.Node(domain="", op_type="Add", inputs=[], outputs=[initializer])

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable node is not used.
type_constraints = {}
parameter = _schemas._convert_formal_parameter(mock_param, type_constraints)
self.assertEqual(parameter.name, "param")
self.assertTrue(ir.TensorType(ir.DataType.FLOAT) in parameter.type_constraint.allowed_types)

Check notice

Code scanning / CodeQL

Imprecise assert Note

assertTrue(a in b) cannot provide an informative message. Using assertIn(a, b) instead will give more informative messages.
def match_pattern(op, x):
return op.Add(x, x)

def apply_pattern(op, x, **_):

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable apply_pattern is not used.
@@ -4,6 +4,11 @@

import onnxscript.tools.benchmark.benchmark_helpers as bh

import torch

Check notice

Code scanning / CodeQL

Unused import Note test

Import of 'torch' is not used.
@@ -4,6 +4,11 @@

import onnxscript.tools.benchmark.benchmark_helpers as bh

import torch
import onnx

Check notice

Code scanning / CodeQL

Unused import Note test

Import of 'onnx' is not used.
def test_class_getitem_shape_already_specified(self):
from onnxscript.onnx_types import FLOAT
with self.assertRaises(ValueError):
FLOAT[None][None]

Check notice

Code scanning / CodeQL

Statement has no effect Note test

This statement has no effect.
Copy link

codecov bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 98.09069% with 24 lines in your changes missing coverage. Please review.

Project coverage is 77.04%. Comparing base (99cf79f) to head (9a5f69a).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
onnxscript/tools/transformers_models/phi3_test.py 77.08% 10 Missing and 1 partial ⚠️
onnxscript/tools/transformers_models/llama_test.py 82.69% 9 Missing ⚠️
onnxscript/rewriter/generic_pattern_test.py 92.30% 1 Missing ⚠️
...rewriter/onnxruntime/transformers/fastgelu_test.py 93.75% 1 Missing ⚠️
...ewriter/onnxruntime/transformers/layernorm_test.py 83.33% 1 Missing ⚠️
onnxscript/values_test.py 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1967      +/-   ##
==========================================
+ Coverage   75.13%   77.04%   +1.90%     
==========================================
  Files         265      265              
  Lines       28907    30164    +1257     
  Branches     3331     3336       +5     
==========================================
+ Hits        21720    23239    +1519     
+ Misses       6158     5903     -255     
+ Partials     1029     1022       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lintrunner found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@qododavid
Copy link
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

2 participants