Layernorm #203
Merged
Layernorm #203
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jvdp1
reviewed
Feb 21, 2025
jvdp1
reviewed
Feb 21, 2025
|
@milancurcic @jvdp1 Ready for review |
Open
milancurcic
reviewed
Feb 24, 2025
milancurcic
reviewed
Feb 24, 2025
milancurcic
reviewed
Feb 24, 2025
milancurcic
reviewed
Feb 24, 2025
milancurcic
reviewed
Feb 24, 2025
|
Great, thank you! I left several comments. |
OneAdder
commented
Feb 25, 2025
| flatten()& | ||
| ]) | ||
|
|
||
| ! Kaiming weights to achieve semblance of convergance |
There was a problem hiding this comment.
If we want LayerNorm used as activation function to converge, we need to at least use correct weights initialization. Otherwise, the net will not be able to consistently converge and we'll get #145 once more
milancurcic
reviewed
Feb 25, 2025
Comment on lines
+37
to
+39
| | Linear (2-d) | `linear2d` | `input2d`, `layernorm`, `linear2d`, `self_attention` | 2 | ✅ | ✅ | | ||
| | Self-attention | `self_attention` | `input2d`, `layernorm`, `linear2d`, `self_attention` | 2 | ✅ | ✅ | | ||
| | Layer Normalization | `layernorm` | `linear2d`, `self_attention` | 2 | ✅ | ✅ | |
There was a problem hiding this comment.
@OneAdder can you please check that I did this correctly?
OneAdder
added a commit
to OneAdder/neural-fortran
that referenced
this pull request
Mar 3, 2025
* layernorm: initial implementation * layernorm: rename source file * layernorm: remove redundant arguments * layernorm: remove stack allocated arrays * layernorm: rearrange into submodule * layernorm: add error to stop in test * layernorm: add gradient updates * layernorm: public api * layernorm: update tests * layernorm: update cmake * layernorm: use mold for temp allocation * layernorm: rename to layernorm * layernorm: allow usage of layernorm at the end * layernorm: integration test for layernorm * layernorm: memory allocation optimization * Tidy up * Bump version * Add layernorm to the table of layers --------- Co-authored-by: milancurcic <caomaco@gmail.com>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Layer Normalization
Layer Normalization applies normalization over a sequence of inputs by the following formula:
Where:
Reference
Lei Ba, J., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. ArXiv e-prints, arXiv-1607.
Status
Ready