[BUG] SinusoidalEmbedding rotate_every_two , related to #106

[evelynmitchell]

From colab, the 2nd SinusoidalEmbeddings example

from zeta import apply_rotary_pos_emb
import torch

# Create query and key tensors
q = torch.randn(2, 3, 4)
k = torch.randn(2, 3, 4)

# Generate frequency and scale embeddings using SinusoidalEmbeddings

# Apply rotary positional embeddings
q_emb, k_emb = apply_rotary_pos_emb(q, k, freqs, scale)

Has the error

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

[<ipython-input-39-a72a64e54f5b>](https://localhost:8080/#) in <cell line: 11>()
      9 
     10 # Apply rotary positional embeddings
---> 11 q_emb, k_emb = apply_rotary_pos_emb(q, k, freqs, scale)

[/usr/local/lib/python3.10/dist-packages/zeta/nn/embeddings/xpos_relative_position.py](https://localhost:8080/#) in apply_rotary_pos_emb(x, sin, cos, scale)
     69     """
     70     sin, cos = map(lambda t: duplicate_interleave(t * scale), (sin, cos))
---> 71     return (x * cos) + (rotate_every_two(x) * sin)
     72 
     73 

RuntimeError: The size of tensor a (4) must match the size of tensor b (128) at non-singleton dimension 2

This is another instance of the problem reported in #106 which I'm still testing the fix for.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[github-actions]

Stale issue message