re-organise built-in layers section

Author: mcabbott

docs/src/models/layers.md

@@ -1,86 +1,130 @@
-# Basic Layers
+# Built-in Layer Types
 
-These core layers form the foundation of almost all neural networks.
+If you started at the beginning, then you have already met the basic [`Dense`](@ref) layer, and seen [`Chain`](@ref) for combining layers. These core layers form the foundation of almost all neural networks. 
+
+The `Dense` layer 
+
+* Weight matrices are created ... Many layers take an `init` keyword, accepts a function acting like `rand`. That is, `init(2,3,4)` creates an array of this size.  ... always on the CPU. 
+
+* An activation function. This is broadcast over the output: `Flux.Scale(3, tanh)([1,2,3]) ≈ tanh.(1:3)`
+
+* The bias vector is always intialised `Flux.zeros32`. The keyword `bias=false` will turn this off.
+
+
+* All layers are annotated with `@layer`, which means that `params` will see the contents, and `gpu` will move their arrays to the GPU.
+
+
+## Fully Connected
 
 ```@docs
-Chain
 Dense
+Flux.Bilinear
+Flux.Scale
 ```
 
-## Convolution and Pooling Layers
+Perhaps `Scale` isn't quite fully connected, but it may be thought of as `Dense(Diagonal(s.weights), s.bias)`, and LinearAlgebra's `Diagonal` is a matrix which just happens to contain many zeros.
+
+## Convolution Models
 
 These layers are used to build convolutional neural networks (CNNs).
 
+They all expect images in what is called WHCN order: a batch of 32 colour images, each 50 x 50 pixels, will have `size(x) == (50, 50, 3, 32)`. A single grayscale image might instead have `size(x) == (28, 28, 1, 1)`.
+
+Besides images, 2D data, they also work with 1D data, where for instance stereo sound recording with 1000 samples might have `size(x) == (1000, 2, 1)`. They will also work with 3D data, `ndims(x) == 5`, where again the last two dimensions are channel and batch.
+
+To understand how `stride` ?? there's a cute article.
+
 ```@docs
 Conv
 Conv(weight::AbstractArray)
-AdaptiveMaxPool
-MaxPool
-GlobalMaxPool
-AdaptiveMeanPool
-MeanPool
-GlobalMeanPool
-DepthwiseConv
 ConvTranspose
 ConvTranspose(weight::AbstractArray)
 CrossCor
 CrossCor(weight::AbstractArray)
+DepthwiseConv
 SamePad
 Flux.flatten
 ```
 
-## Upsampling Layers
+### Pooling
+
+These layers are commonly used after a convolution layer, and reduce the size of its output. They have no trainable parameters.
+
+```@docs
+AdaptiveMaxPool
+MaxPool
+GlobalMaxPool
+AdaptiveMeanPool
+MeanPool
+GlobalMeanPool
+```
+
+## Upsampling
+
+The opposite of pooling, these layers increase the size of an array. They have no trainable parameters. 
 
 ```@docs
 Upsample
 PixelShuffle
 ```
 
-## Recurrent Layers
+## Embedding Vectors
 
-Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).
+These layers accept an index, and return a vector (or several indices, and several vectors). The possible embedding vectors are learned parameters.
 
 ```@docs
-RNN
-LSTM
-GRU
-GRUv3
-Flux.Recur
-Flux.reset!
+Flux.Embedding
+Flux.EmbeddingBag
 ```
 
-## Other General Purpose Layers
+## Dataflow Layers, or Containers
 
-These are marginally more obscure than the Basic Layers.
-But in contrast to the layers described in the other sections are not readily grouped around a particular purpose (e.g. CNNs or RNNs).
+The basic `Chain(F, G, H)` applies the layers it contains in sequence, equivalent to `H ∘ G ∘ F`. Flux has some other layers which contain layers, but connect them up in a more complicated way: `SkipConnection` allows ResNet's ??residual connection.
+
+These are all defined with [`@layer`](@ref)` :exand TypeName`, which tells the pretty-printing code that they contain other layers.
 
 ```@docs
+Chain
+Flux.activations
 Maxout
 SkipConnection
 Parallel
-Flux.Bilinear
-Flux.Scale
-Flux.Embedding
+PairwiseFusion
+```
+
+## Recurrent Models
+
+Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).
+
+```@docs
+RNN
+LSTM
+GRU
+GRUv3
+Flux.Recur
+Flux.reset!
 ```
 
 ## Normalisation & Regularisation
 
-These layers don't affect the structure of the network but may improve training times or reduce overfitting.
+These layers don't affect the structure of the network but may improve training times or reduce overfitting. Some of them contain trainable parameters, while others do not.
 
 ```@docs
-Flux.normalise
 BatchNorm
 Dropout
-Flux.dropout
 AlphaDropout
 LayerNorm
 InstanceNorm
 GroupNorm
+Flux.normalise
+Flux.dropout
 ```
 
-### Testmode
+### Test vs. Train
+
+Several normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. 
 
-Many normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. Still, depending on your use case, it may be helpful to manually specify when these layers should be treated as being trained or not. For this, Flux provides `Flux.testmode!`. When called on a model (e.g. a layer or chain of layers), this function will place the model into the mode specified.
+The functions `Flux.trainmode!` and `Flux.testmode!` let you manually specify which behaviour you want. When called on a model, they will place all layers within the model into the specified mode.
 
 ```@docs
 Flux.testmode!

src/layers/basic.jl

@@ -182,6 +182,9 @@ function Base.show(io::IO, l::Dense)
   print(io, ")")
 end
 
+Dense(W::LinearAlgebra.Diagonal, bias = true, σ = identity) =
+  Scale(W.diag, bias, σ)
+
 """
     Scale(size::Integer..., σ=identity; bias=true, init=ones32)
     Scale(scale::AbstractArray, [bias, σ])