Fix CI tests #1993

justinchuby · 2024-12-31T18:35:26Z

Bump ort and onnx versions
Remove dort tests as they are obsolete
Improve constant folding and assert the invariance of const_value being tensors

- Bump ort and onnx versions - Remove dort tests as they are obsolete

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

Files not reviewed (1)

requirements-dev.txt: Language not supported

codecov · 2024-12-31T18:41:34Z

❌ 34 Tests Failed:

Tests completed	Failed	Passed	Skipped
13288	34	13254	2158

View the full list of 3 ❄️ flaky tests

tests.eager_mode_test.TestEagerModeArguments_0_reference_runtime::test_function_input_and_attribute_by_kwargs_out_of_order

Flake rate in main: 39.27% (Passed 12633 times, Failed 8169 times)

Stack Traces | 0.002s run time

..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:91: in run
    res = self._run(x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:139: in _run
    res = (convert_from_ml_dtypes(res[0]),)
..../test_torch_nightly/lib/python3.11.../onnx/reference/custom_element_types.py:50: in convert_from_ml_dtypes
    return array.view(dtype=dtype)
E   ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

The above exception was the direct cause of the following exception:
tests/eager_mode_test.py:115: in test_function_input_and_attribute_by_kwargs_out_of_order
    self.assertEqual(add_with_alpha(alpha=3.0, other=2.0, this=1.0), 7.0)
onnxscript/values.py:576: in __call__
    return evaluator.default().eval_function(self, args, kwargs)
onnxscript/evaluator.py:307: in eval_function
    result = function.function(*adapted_args, **adapted_kwargs)
tests/eager_mode_test.py:59: in add_with_alpha
    other = op.Mul(other, alpha)
.../onnx_opset/_impl/opset14.py:696: in Mul
    return op(*self._prepare_inputs(schema, A, B))
onnxscript/values.py:304: in __call__
    return evaluator.default().eval(schema, args, kwargs)
onnxscript/evaluator.py:194: in eval
    outputs = self._eval(schema, inputs, attributes, closure)
onnxscript/evaluator.py:526: in _eval
    result = session.run(None, session_run_input)
..../test_torch_nightly/lib/python3.11.../onnx/reference/reference_evaluator.py:593: in run
    outputs = node.run(*inputs, **linked_attributes)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:114: in run
    res = OpRunBinary.run(self, x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:93: in run
    raise TypeError(
E   TypeError: Issues with types <class 'numpy.ndarray'>, <class 'numpy.ndarray'> (binary operator 'Mul').

tests.eager_mode_test.TestEagerModeArguments_0_reference_runtime::test_function_some_input_by_kwargs

Flake rate in main: 39.27% (Passed 12633 times, Failed 8169 times)

Stack Traces | 0.002s run time

..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:91: in run
    res = self._run(x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:139: in _run
    res = (convert_from_ml_dtypes(res[0]),)
..../test_torch_nightly/lib/python3.11.../onnx/reference/custom_element_types.py:50: in convert_from_ml_dtypes
    return array.view(dtype=dtype)
E   ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

The above exception was the direct cause of the following exception:
tests/eager_mode_test.py:106: in test_function_some_input_by_kwargs
    self.assertEqual(add_with_alpha(1.0, other=2.0), 3.0)
onnxscript/values.py:576: in __call__
    return evaluator.default().eval_function(self, args, kwargs)
onnxscript/evaluator.py:307: in eval_function
    result = function.function(*adapted_args, **adapted_kwargs)
tests/eager_mode_test.py:59: in add_with_alpha
    other = op.Mul(other, alpha)
.../onnx_opset/_impl/opset14.py:696: in Mul
    return op(*self._prepare_inputs(schema, A, B))
onnxscript/values.py:304: in __call__
    return evaluator.default().eval(schema, args, kwargs)
onnxscript/evaluator.py:194: in eval
    outputs = self._eval(schema, inputs, attributes, closure)
onnxscript/evaluator.py:526: in _eval
    result = session.run(None, session_run_input)
..../test_torch_nightly/lib/python3.11.../onnx/reference/reference_evaluator.py:593: in run
    outputs = node.run(*inputs, **linked_attributes)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:114: in run
    res = OpRunBinary.run(self, x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:93: in run
    raise TypeError(
E   TypeError: Issues with types <class 'numpy.ndarray'>, <class 'numpy.ndarray'> (binary operator 'Mul').

tests.eager_mode_test.TestEagerModeArguments_0_reference_runtime::test_function_all_input_by_kwargs

Flake rate in main: 39.27% (Passed 12633 times, Failed 8169 times)

Stack Traces | 0.002s run time

..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:91: in run
    res = self._run(x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:139: in _run
    res = (convert_from_ml_dtypes(res[0]),)
..../test_torch_nightly/lib/python3.11.../onnx/reference/custom_element_types.py:50: in convert_from_ml_dtypes
    return array.view(dtype=dtype)
E   ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

The above exception was the direct cause of the following exception:
tests/eager_mode_test.py:109: in test_function_all_input_by_kwargs
    self.assertEqual(add_with_alpha(this=1.0, other=2.0), 3.0)
onnxscript/values.py:576: in __call__
    return evaluator.default().eval_function(self, args, kwargs)
onnxscript/evaluator.py:307: in eval_function
    result = function.function(*adapted_args, **adapted_kwargs)
tests/eager_mode_test.py:59: in add_with_alpha
    other = op.Mul(other, alpha)
.../onnx_opset/_impl/opset14.py:696: in Mul
    return op(*self._prepare_inputs(schema, A, B))
onnxscript/values.py:304: in __call__
    return evaluator.default().eval(schema, args, kwargs)
onnxscript/evaluator.py:194: in eval
    outputs = self._eval(schema, inputs, attributes, closure)
onnxscript/evaluator.py:526: in _eval
    result = session.run(None, session_run_input)
..../test_torch_nightly/lib/python3.11.../onnx/reference/reference_evaluator.py:593: in run
    outputs = node.run(*inputs, **linked_attributes)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:114: in run
    res = OpRunBinary.run(self, x, y)
..../test_torch_nightly/lib/python3.11.../reference/ops/_op.py:93: in run
    raise TypeError(
E   TypeError: Issues with types <class 'numpy.ndarray'>, <class 'numpy.ndarray'> (binary operator 'Mul').

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

Copilot reviewed 5 out of 12 changed files in this pull request and generated no comments.

Files not reviewed (7)

requirements-dev.txt: Language not supported
requirements/ci/requirements-ort-nightly.txt: Language not supported
onnxscript/rewriter/broadcast_to_matmul.py: Evaluated as low risk
onnxscript/function_libs/torch_lib/ops/core.py: Evaluated as low risk
tests/models/sequences.py: Evaluated as low risk
.github/workflows/main.yaml: Evaluated as low risk
onnxscript/tools/transformers_models/mistral_test.py: Evaluated as low risk

Comments suppressed due to low confidence (10)

tests/function_libs/torch_lib/ops_test_data.py:552

The test for zero sized inputs is skipped with the reason 'zero sized inputs cannot be compared'. Ensure that this is the intended behavior.

.skip(

tests/function_libs/torch_lib/ops_test_data.py:569

The skip condition for scalar inputs to ReduceMax-18 is removed. Confirm that the issue with ORT has been resolved.

TorchLibOpInfo("amax", core_ops.aten_amax, input_wrangler=_amin_amax_input_wrangler,).skip(

tests/function_libs/torch_lib/ops_test_data.py:701

The skip condition for zero-dim tensors is updated to handle device-specific tensors.

matcher=lambda sample: sample.input[0].equal(torch.tensor([]).to(sample.input[0].device)),

tests/function_libs/torch_lib/ops_test_data.py:726

The skip condition for size 0 inputs is added with the reason 'Size 0 inputs are not handled by design'.

TorchLibOpInfo("clamp_max", core_ops.aten_clamp_max).skip(

tests/function_libs/torch_lib/ops_test_data.py:853

The skip condition for index_put_bool is added with the reason 'FIXME: [torchlib] Improve index_put #1749'.

.skip(reason="FIXME: https://github.com/microsoft/onnxscript/issues/1749"),

tests/function_libs/torch_lib/ops_test_data.py:963

The skip condition for scalar inputs to Reduce*-18 is removed. Confirm that the issue with ORT has been resolved.

TorchLibOpInfo("maximum", core_ops.aten_maximum),

tests/function_libs/torch_lib/ops_test_data.py:1061

The tolerance value for embedding_bag is updated.

tolerance={torch.float16: (1e-2, 5e-2)},

tests/function_libs/torch_lib/ops_test_data.py:1449

The tolerance value for sub is added.

TorchLibOpInfo("sub", core_ops.aten_sub, tolerance={torch.float16: (2e-3, 1e-3)}),

tests/function_libs/torch_lib/ops_test_data.py:1470

The xfail condition for topk is added with the reason 'fixme: result mismatch. [torchlib] topk results do not match when input is int64/int32 #853'.

TorchLibOpInfo("topk", core_ops.aten_topk).xfail(

tests/function_libs/torch_lib/ops_test_data.py:1587

The skip condition for scalar inputs to Reduce*-18 is removed. Confirm that the issue with ORT has been resolved.

TorchLibOpInfo("clamp", core_ops.aten_clamp),

justinchuby added 2 commits December 31, 2024 10:33

Fix CI tests

76f8477

- Bump ort and onnx versions - Remove dort tests as they are obsolete

req

cd6f437

justinchuby requested review from Copilot and gramalingam December 31, 2024 18:35

Copilot AI reviewed Dec 31, 2024

View reviewed changes

Update main.yaml

4c83616

justinchuby added 16 commits January 2, 2025 11:16

Remove test

6156557

nox

e428591

eval

dcb5694

subclass

fe82d4b

nightly

8b69721

eager

ff7a0a2

numpy

06d556e

lint

6ba054b

onnxruntime

37a981c

constant folding

04ca4b5

bool

75a355d

1.21.0.dev20241108002

6b69ec1

fixme

c39fcc8

lint

5e04011

lint

8dfa02e

np

ab63cc9

gramalingam approved these changes Jan 3, 2025

View reviewed changes

justinchuby added 6 commits January 3, 2025 10:32

compare

1f2008a

lint

8b1702d

16492

39c82d6

update

1991546

update

ddd683d

topk

59c2f2f

justinchuby added 3 commits January 3, 2025 19:16

more fixes

811e74c

topk

82eecf3

Merge branch 'main' into justinchu/fix-ci

135d5fd

justinchuby requested a review from Copilot January 4, 2025 03:30

Copilot AI reviewed Jan 4, 2025

View reviewed changes

justinchuby added 9 commits January 3, 2025 19:35

topk

db51048

norm

9724ad9

ort

61b9f18

test

eea9fad

dort

a29db04

torch

26b4973

vision

0967efb

exp

24bdc3f

q

45d0ce5

justinchuby merged commit b064539 into main Jan 4, 2025
21 of 29 checks passed

justinchuby deleted the justinchu/fix-ci branch January 4, 2025 07:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix CI tests #1993

Fix CI tests #1993

justinchuby commented Dec 31, 2024 •

edited

Loading

codecov bot commented Dec 31, 2024 •

edited

Loading

Fix CI tests #1993

Fix CI tests #1993

Conversation

justinchuby commented Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Dec 31, 2024 • edited Loading

❌ 34 Tests Failed:

Choose a reason for hiding this comment

justinchuby commented Dec 31, 2024 •

edited

Loading

codecov bot commented Dec 31, 2024 •

edited

Loading