Feature request: add local and reformer

[samvanstroud]

Thanks for this repo. Is there a possibility of adding your existing local attention and reformer implementations here?

I'm hoping they may also be able to be updated to take advantage of the upcoming attention mask support for the meff kernel in PT2.1.

[lucidrains]

yeah, I do have plans to make it so one can register custom transformer blocks. probably will be tested with mixture of experts first https://github.com/lucidrains/st-moe-pytorch, but will prob also consider local attention