Did not multiply embedding weights by sqrt(d_model)

[orena1]

Hi,
In this line:

Transformer/Embed.py

Line 12 in 37bf492

return self.embed(x)

I think you need to multiply the embedding by sqrt(d_model)

[fabrahman]

@orena1 Hi, the implementation also didn't share the embedding weights, right?

[fabrahman]

@orena1 The code actually has * math.sqrt(self.d_model) in the positional embedding class. In forward method.

[zhangxixi0904]

Did somebody know the reason for multiplying embedding weights by sqrt(d_model)？

[wangzelin-em]

@orena1 Hi, the implementation also didn't share the embedding weights, right?

Yes, the implementation didn't share the embedding weights.