Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytorch Sweeps tracing - ttnn.mul - Low PCC and other fails #14731

Closed
Tracked by #13521
KalaivaniMCW opened this issue Nov 5, 2024 · 4 comments
Closed
Tracked by #13521

Pytorch Sweeps tracing - ttnn.mul - Low PCC and other fails #14731

KalaivaniMCW opened this issue Nov 5, 2024 · 4 comments

Comments

@KalaivaniMCW
Copy link
Contributor

KalaivaniMCW commented Nov 5, 2024

Following parameters failed during Pytorch Sweeps tracing tests: for ttnn.mul
File: tests/sweep_framework/sweeps/eltwise/binary/multiply/mul_tensor_pytorch2.py
Pytorch sweeps: tracker

Parameter   Error
current parameter {'input_shape': {'self': [4, 12, 64, 64], 'other': [12, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [4, 16, 64, 64], 'other': [16, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [64, 3, 64, 64], 'other': [3, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [64, 4, 64, 64], 'other': [4, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [16, 1], 'other': [1, 1, 32]}   low pcc fail- unequal ranks and broadcast issue
current parameter {'input_shape': {'self': [16, 6, 64, 64], 'other': [6, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [16, 8, 64, 64], 'other': [8, 1, 1]},   low pcc fail- unequal ranks ?
current parameter {'input_shape': {'self': [0], 'other': 0.5},   round_up: multiple must not be 0
current parameter {'input_shape': {'self': [0], 'other': []},   round_up: multiple must not be 0
current parameter {'input_shape': {'self': [0, 1], 'other': [0, 1]},   round_up: multiple must not be 0
current parameter {'input_shape': {'self': [1], 'other': [1]},   message list(expected_pytorch_result.shape)=[1] vs list(actual_pytorch_result.shape)=[1, 1]
current parameter {'input_shape': {'self': [], 'other': [0, 1]},   round_up: multiple must not be 0
current parameter {'input_shape': {'self': [], 'other': [1, 1, 768]}   Layout issue for [ ] shape
current parameter {'input_shape': {'self': [], 'other': [1, 24, 768]},   same as above
current parameter {'input_shape': {'self': [], 'other': [3234, 1]},   same as above
current parameter {'input_shape': {'self': [], 'other': [8732, 1]},   same as above
@KalaivaniMCW
Copy link
Contributor Author

Out of these 7 shapes for low pcc fails:
[[4, 12, 64, 64], [12, 1, 1]], [[4, 16, 64, 64], [16, 1, 1]], [[64, 3, 64, 64], [3, 1, 1]], [[64, 4, 64, 64], [4, 1, 1]], [[16, 1], [1, 1, 32]], [[16, 6, 64, 64], [6, 1, 1]], [[16, 8, 64, 64], [8, 1, 1]],

Six of them pass when the lesser-ranked is unsqueezed to match rank. i.e. following shapes,

        [[4, 12, 64, 64], [1, 12, 1, 1]], 
        [[4, 16, 64, 64], [1, 16, 1, 1]], 
        [[64, 3, 64, 64], [1, 3, 1, 1]], 
        [[64, 4, 64, 64], [1, 4, 1, 1]], 
        [[16, 6, 64, 64], [1, 6, 1, 1]],
        [[16, 8, 64, 64], [1, 8, 1, 1]],

One shape gives a different error on unit test [[16, 1], [1, 1, 32]],
and it fails, even when unsqueezed i.e [[16, 1], [1, 1, 32]], to [[1, 16, 1], [1, 1, 32]],

On sweeps
Image

On unit tests

E   RuntimeError: TT_ASSERT @ ../ttnn/cpp/ttnn/operations/eltwise/binary/device/binary_device_operation.cpp:150: height_a > height_b and height_b == 1
E       info:
E       ttnn::operations::binary::BinaryDeviceOperation: height mismatch

Image

@KalaivaniMCW
Copy link
Contributor Author

PR #14803 has fixes for low PCC of 6 shapes mentioned above

@KalaivaniMCW KalaivaniMCW self-assigned this Nov 8, 2024
KalaivaniMCW added a commit that referenced this issue Nov 12, 2024
### Ticket
Link to Github Issue #14730 
also #14731
### Problem description
Need support for unequal ranked inputs in eltwise binary

### What's changed
Added support using ttnn.reshape when inputs are of different ranks

### Checklist
- [x] Post commit CI passes
https://github.com/tenstorrent/tt-metal/actions/runs/11736842353

https://github.com/tenstorrent/tt-metal/actions/runs/11794021567/attempts/2
- [ ] Nightly FD
https://github.com/tenstorrent/tt-metal/actions/runs/11736844358

https://github.com/tenstorrent/tt-metal/actions/runs/11794025766/job/32854516936
- [ ] Blackhole Post commit (if applicable)
- [ ] Model regression CI testing passes (if applicable)
- [ ] Device performance regression CI testing passes (if applicable)
- [x] New/Existing tests provide coverage for changes
@KalaivaniMCW
Copy link
Contributor Author

Merged #14803
The 6 out 7 pcc fail cases mentioned above are now passing. in this case {'input_shape': {'self': [16, 1], 'other': [1, 1, 32]} , the unequal rank has been fixed but it fails due to broadcast issue which is mentioned in #14710

@KalaivaniMCW
Copy link
Contributor Author

Torch: {'input_shape': {'self': [], 'other': [0, 1]},
Image

Torch: {'input_shape': {'self': [], 'other': [1, 1, 768]}
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants