From: AspenStars <AspenStars@outlook.com>
Date: Fri, 13 Aug 2021 13:40:41 +0000 (-0700)
Subject: DOC Improve documentation for LayerNorm (#63144)
X-Git-Tag: accepted/tizen/8.0/unified/20231005.095509~1050
X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=72bc6dc8c31f977036195536292170ac9132751d;p=platform%2Fupstream%2Fpytorch.git

DOC Improve documentation for LayerNorm (#63144)

Summary:
In this [commit](https://github.com/pytorch/pytorch/pull/59178/commits/7026995f3ca253fbc19bf511d53f48f861799a4a) and [issue](https://github.com/pytorch/pytorch/pull/59178#issuecomment-897485295), the [Line 134](https://github.com/deniskokarev/pytorch/blob/47e286d024c183cb26a464447b34fde88b80d17d/torch/nn/modules/normalization.py#L134) will overwrite the "embedding" variate which would cause an error when initiating `nn.LayerNorm` function.

I suggest renaming the "embedding" in [Line 133](https://github.com/deniskokarev/pytorch/blob/47e286d024c183cb26a464447b34fde88b80d17d/torch/nn/modules/normalization.py#L133) to "embedding_dim".

The final example is:
```
batch, sentence_length, embedding_dim = 20, 5, 10
embedding = torch.randn(batch, sentence_length, embedding_dim)
layer_norm = nn.LayerNorm(embedding_dim)
```

Fixes #{59178}

Pull Request resolved: https://github.com/pytorch/pytorch/pull/63144

Reviewed By: bdhirsh

Differential Revision: D30288778

Pulled By: jbschlosser

fbshipit-source-id: e74b11430e302dae5661bf6e830ee5ac6c1838c4
---

diff --git a/torch/nn/modules/normalization.py b/torch/nn/modules/normalization.py
index 822a30f..64ef008 100644
--- a/torch/nn/modules/normalization.py
+++ b/torch/nn/modules/normalization.py
@@ -123,6 +123,14 @@ class LayerNorm(Module):
             has learnable per-element affine parameters initialized to ones (for weights)
             and zeros (for biases). Default: ``True``.
 
+    Attributes:
+        weight: the learnable weights of the module of shape
+            :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``.
+            The values are initialized to 1.
+        bias:   the learnable bias of the module of shape
+                :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``.
+                The values are initialized to 0.
+
     Shape:
         - Input: :math:`(N, *)`
         - Output: :math:`(N, *)` (same shape as input)
@@ -130,11 +138,12 @@ class LayerNorm(Module):
     Examples::
 
         >>> # NLP Example
-        >>> batch, sentence_length, embedding = 20, 5, 10
-        >>> embedding = torch.randn(batch, sentence_length, embedding)
-        >>> layer_norm = nn.LayerNorm(embedding)
+        >>> batch, sentence_length, embedding_dim = 20, 5, 10
+        >>> embedding = torch.randn(batch, sentence_length, embedding_dim)
+        >>> layer_norm = nn.LayerNorm(embedding_dim)
         >>> # Activate module
         >>> layer_norm(embedding)
+        >>>
         >>> # Image Example
         >>> N, C, H, W = 20, 5, 10, 10
         >>> input = torch.randn(N, C, H, W)