Layer Normalization

Last updated Feb 8, 2024 Edit Source

• Hebbian

#hebbian

The technique stabilizes hidden state dynamics in RNNs. It may be an alternative solution to some previous issues I had with my hebbian experiments, where the weight values, activations, and subsequently the loss numbers got increasingly erratic and large over time.

Here’s the quote: Annotations-(1-31-2024,-9-54-17-AM)-2XQN472I