Gains in the last affine layer seem to drop to +- minus 50. https://gist.github.com/supergrover/733504ee836506603ea4