Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] ttnn::reshape failing to reshape tensor from [1, 1, 1, 65536] to [1, 32, 32, 32] citing L1 allocation error #15075

Open
marty1885 opened this issue Nov 15, 2024 · 5 comments
Assignees
Labels
bug Something isn't working community P1

Comments

@marty1885
Copy link
Contributor

Describe the bug
I'm attempting to eliminate most if not all CPU fallback code paths in my GGML backend. One of which is reshaping a single vector into a 3D tensor - I know this is not efficient on TT hardware and I should eliminate it completely by graph rewrite in the future - This operation fails when performed purely on device.

To Reproduce
The following is the minimal reproducible example. Compile and run it:

#include <cstddef>
#include <ttnn/core.hpp>
#include <ttnn/operations/eltwise/unary/unary.hpp>
#include <ttnn/operations/creation.hpp>
#include <ttnn/device.hpp>
#include <ttnn/operations/data_movement/tilize_with_val_padding/tilize_with_val_padding.hpp>
#include "ttnn/operations/data_movement/reshape_on_device/reshape.hpp"

#include "ttnn/operations/data_movement/reshape_view/reshape.hpp"
#include "common/bfloat16.hpp"

#include <vector>
#include <iostream>

int main()
{
    auto& device = ttnn::open_device(0);
    auto t = ttnn::zeros(ttnn::SimpleShape({1, 1, 1, 65536})).to(&device);
    t = ttnn::tilize_with_zero_padding(t);

    std::cout << "Original shape: " << t.shape() << std::endl;
    auto b = ttnn::reshape(t, ttnn::SimpleShape({1, 32, 32, 32}));
    std::cout << "resulting shaoe: " << b.shape() << std::endl;
}

Observe the error:

                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 0 is:   1000 MHz
                  Metal | INFO     | MMIO Device 0 : Tunnel 0 : Device 0
                  Metal | INFO     | MMIO Device 0 : Tunnel 0 : Device 4
Original shape: ttnn.Shape([1, 1, 1[32], 65536])
                 Always | FATAL    | Statically allocated circular buffers on core range [(x=0,y=0) - (x=0,y=0)] grow to 8487424 B which is beyond max L1 size of 1499136 B
terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_THROW @ /home/marty/Documents/tt/tt-metal/tt_metal/impl/program/program.cpp:761: tt::exception
info:
Statically allocated circular buffers on core range [(x=0,y=0) - (x=0,y=0)] grow to 8487424 B which is beyond max L1 size of 1499136 B

Expected behavior
Reshape from 1D vector should work no matter the vector size. Or at least the API/document should provide some way to indicate the algorithm cannot run if the input exceeds a certain size.

Screenshots
If applicable, add screenshots to help explain your problem.

Please complete the following environment information:

  • OS: Arch Linux
  • Card: N300
  • commit: 008c50a

Additional context
Add any other context about the problem here.

@marty1885 marty1885 added the bug Something isn't working label Nov 15, 2024
@ayerofieiev-tt
Copy link
Member

ayerofieiev-tt commented Nov 15, 2024

Looks similar to
#15032

@dmakoviichuk-tt
Copy link
Contributor

dmakoviichuk-tt commented Nov 15, 2024

@marty1885 could you pass last parameter multicore=true to the tilize_with_zero_padding please?

@marty1885
Copy link
Contributor Author

marty1885 commented Nov 15, 2024

@dmakoviichuk-tt The problem is not tilize_with_zero_padding. It's with ttnn::reshape. The line Original shape: ttnn.Shape([1, 1, 1[32], 65536]) is printed. I assume this means tilize_with_zero_padding passes?

You are right. The problem is indeed in tilize_with_zero_padding.

Update: No,with multicore=true i get the same error.

@dmakoviichuk-tt
Copy link
Contributor

dmakoviichuk-tt commented Nov 15, 2024

ah :( thats sad. btw I've got second idea to make it work.
Out of curiosity (we still need to fix this one )Is it possible to reshape before tilize?

@marty1885
Copy link
Contributor Author

@dmakoviichuk-tt No unfortunately. ttnn::reshape eventually calls ttnn::to_layout and runs into the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community P1
Projects
None yet
Development

No branches or pull requests

4 participants