Skip to content

Commit 3d61801

Browse files
d-kleinepytorchmergebot
authored andcommitted
Fix RMSNorm Notation: Parentheses, Indices, Comma (pytorch#140215)
Fixes pytorch#140165 * fixed mathematical notation for RMSNorm: * changed RMS function from brackets `[x]` to parenthesis `(x)` for consistency and align with mathematical notation standards for functions * added indices (e.g. `y_i`) for element-wise operations for the correctness in the context of tensor operations * added comma `,` before $$\text{where}$$ ![grafik](https://github.com/user-attachments/assets/47368625-d97a-43de-8b90-17b2c01cbe2f) Pull Request resolved: pytorch#140215 Approved by: https://github.com/mikaylagawarecki
1 parent a58a565 commit 3d61801

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

torch/nn/modules/normalization.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@ class RMSNorm(Module):
325325
the paper `Root Mean Square Layer Normalization <https://arxiv.org/pdf/1910.07467.pdf>`__
326326
327327
.. math::
328-
y = \frac{x}{\mathrm{RMS}[x]} * \gamma \quad
328+
y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad
329329
\text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2}
330330
331331
The RMS is taken over the last ``D`` dimensions, where ``D``

0 commit comments

Comments
 (0)