Small paper ideas to be added

[RyanKim17920]

Here's some papers I've read that would be nice to have, I'll try to implement them if I can:

https://arxiv.org/pdf/2010.04245

https://arxiv.org/abs/2210.05144
(Probably should add FFN MoE as well)

https://arxiv.org/pdf/2404.02258
(Probably will be hard to make work with other features)

[lucidrains]

@RyanKim17920 so the first paper is already in the repository and even cited

i do like the second paper, and can try it out before adding it

the third paper, i like as well, but may be outside the scope of this repo

[lucidrains]

@RyanKim17920 someone also shared with me https://arxiv.org/abs/2312.07987 which could be an improvement from MoA

[lucidrains]

@RyanKim17920 the switchhead paper is pretty good

will run the experiments tomorrow morning, and if all goes well, it will probably in the repository by week's end

[Baran-phys]

@lucidrains What do you think of https://www.arxiv.org/abs/2408.14915, in particular the DRA activation function for Continuous Transformers?

[Baran-phys]

@lucidrains If you confirm, I can also open a PR for DRA.

[lucidrains]

@Baran-phys hey Baran, thanks for sharing your paper.

it is interesting but i will probably not accept as it is not relevant for this repository. periodic activation functions is something i've been meaning to look into once the right problem presents