We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent a58a565 commit 3d61801Copy full SHA for 3d61801
torch/nn/modules/normalization.py
@@ -325,7 +325,7 @@ class RMSNorm(Module):
325
the paper `Root Mean Square Layer Normalization <https://arxiv.org/pdf/1910.07467.pdf>`__
326
327
.. math::
328
- y = \frac{x}{\mathrm{RMS}[x]} * \gamma \quad
+ y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad
329
\text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2}
330
331
The RMS is taken over the last ``D`` dimensions, where ``D``
0 commit comments