Improving embedding

[EmmaRenauld]

Hi,

Thanks for the code. It gives me a base for mine.
I just tested the following implementation for the position embedding in custom/layers. It works faster so I offer you my solution. The difference in results due to numerical approx is < e^-12.

    pos_emb = np.zeros((max_seq, embedding_dim))
    for index in range(0, d_emb,2):
        pos_emb[:, index] = np.array([m.sin(pos/10000**(index/embedding_dim))
                                      for pos in range(max_seq)])
        pos_emb[:, index+1] = np.array([m.cos(pos/10000**(index/embedding_dim))
                                       for pos in range(max_seq)])
    return pos_emb

[serkansulun]

Does this directly replace the sinusoid function?

[serkansulun]

Looks like the answer is yes, but it doesn't matter much since it's only run one time, before training begins.