deep-learning-with-python-notebooks/4.4-overfitting-and-underfitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文代码作者：François Chollet\n",
    "\n",
    "github：https://github.com/fchollet/deep-learning-with-python-notebooks\n",
    "\n",
    "中文注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：keras 2.2.1（原文是2.0.8，运行结果一致），tensorflow 1.8，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](data/gongzhong.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.1.5'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting and underfitting\n",
    "# 过拟合与欠拟合\n",
    "\n",
    "This notebook contains the code samples found in Chapter 3, Section 6 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "\n",
    "In all the examples we saw in the previous chapter -- movie review sentiment prediction, topic classification, and house price regression -- \n",
    "we could notice that the performance of our model on the held-out validation data would always peak after a few epochs and would then start \n",
    "degrading, i.e. our model would quickly start to _overfit_ to the training data. Overfitting happens in every single machine learning \n",
    "problem. Learning how to deal with overfitting is essential to mastering machine learning.\n",
    "\n",
    "The fundamental issue in machine learning is the tension between optimization and generalization. \"Optimization\" refers to the process of \n",
    "adjusting a model to get the best performance possible on the training data (the \"learning\" in \"machine learning\"), while \"generalization\" \n",
    "refers to how well the trained model would perform on data it has never seen before. The goal of the game is to get good generalization, of \n",
    "course, but you do not control generalization; you can only adjust the model based on its training data.\n",
    "\n",
    "At the beginning of training, optimization and generalization are correlated: the lower your loss on training data, the lower your loss on \n",
    "test data. While this is happening, your model is said to be _under-fit_: there is still progress to be made; the network hasn't yet \n",
    "modeled all relevant patterns in the training data. But after a certain number of iterations on the training data, generalization stops \n",
    "improving, validation metrics stall then start degrading: the model is then starting to over-fit, i.e. is it starting to learn patterns \n",
    "that are specific to the training data but that are misleading or irrelevant when it comes to new data.\n",
    "\n",
    "To prevent a model from learning misleading or irrelevant patterns found in the training data, _the best solution is of course to get \n",
    "more training data_. A model trained on more data will naturally generalize better. When that is no longer possible, the next best solution \n",
    "is to modulate the quantity of information that your model is allowed to store, or to add constraints on what information it is allowed to \n",
    "store. If a network can only afford to memorize a small number of patterns, the optimization process will force it to focus on the most \n",
    "prominent patterns, which have a better chance of generalizing well.\n",
    "\n",
    "The processing of fighting overfitting in this way is called _regularization_. Let's review some of the most common regularization \n",
    "techniques, and let's apply them in practice to improve our movie classification model from  the previous chapter.\n",
    "\n",
    "\n",
    "在上一章的三个例子（预测电影评论、主题分类和房价回归）中，模型在留出验证数据上的性能总是在几轮后达到最高点，然后开始下降。也就是说，模型很快就在训练数据上开始过拟合。过拟合存在于所有机器学习问题中。学会如何处理过拟合对掌握机器学习至关重要。\n",
    "\n",
    "机器学习的根本问题是优化和泛化之间的对立。优化（optimization）是指调节模型以在训练数据上得到最佳性能（即机器学习中的学习），而泛化（generalization）是指训练好的模型在前所未见的数据上的性能好坏。机器学习的目的当然是得到良好的泛化，但你无法控制泛化，只能基于训练数据调节模型。\n",
    "\n",
    "训练开始时，优化和泛化是相关的：训练数据上的损失越小，测试数据上的损失也越小。这时的模型是欠拟合（underfit）的，即仍有改进的空间，网络还没有对训练数据中所有相关模式建模。但在训练数据上迭代一定次数之后，泛化不再提高，验证指标先是不变，然后开始变差，即模型开始过拟合。这时模型开始学习仅和训练数据有关的模式，但这种模式对新数据来说是错误的或无关紧要的。\n",
    "\n",
    "为了防止模型从训练数据中学到错误或无关紧要的模式，最优解决方法是获取更多的训练数据。模型的训练数据越多，泛化能力自然也越好。如果无法获取更多数据，次优解决方法是调节模型允许存储的信息量，或对模型允许存储的信息加以约束。如果一个网络只能记住几个模式，那么优化过程会迫使模型集中学习最重要的模式，这样更可能得到良好的泛化。\n",
    "\n",
    "这种降低过拟合的方法叫作正则化（regularization）。我们先介绍几种最常见的正则化方法， 然后将其应用于实践中，以改进上一节的电影分类模型。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: in this notebook we will be using the IMDB test set as our validation set. It doesn't matter in this context.\n",
    "\n",
    "Let's prepare the data using the code from Chapter 3, Section 5:\n",
    "\n",
    "注意：在这个notebook中，我们将使用IMDB测试集作为我们的验证集。 在这种情况下关系不大。\n",
    "\n",
    "让我们使用准备数据并使用第3章第5节中的代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "import numpy as np\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # Create an all-zero matrix of shape (len(sequences), dimension)（创建一个形状为 (len(sequences), dimension) 的零矩阵）\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # set specific indices of results[i] to 1.（将 results[i] 的指定索引设为 1）\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data（将训练数据向量化）\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data（将测试数据向量化）\n",
    "x_test = vectorize_sequences(test_data)\n",
    "# Our vectorized labels（将标签向量化）\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fighting overfitting\n",
    "\n",
    "## Reducing the network's size\n",
    "\n",
    "\n",
    "The simplest way to prevent overfitting is to reduce the size of the model, i.e. the number of learnable parameters in the model (which is \n",
    "determined by the number of layers and the number of units per layer). In deep learning, the number of learnable parameters in a model is \n",
    "often referred to as the model's \"capacity\". Intuitively, a model with more parameters will have more \"memorization capacity\" and therefore \n",
    "will be able to easily learn a perfect dictionary-like mapping between training samples and their targets, a mapping without any \n",
    "generalization power. For instance, a model with 500,000 binary parameters could easily be made to learn the class of every digits in the \n",
    "MNIST training set: we would only need 10 binary parameters for each of the 50,000 digits. Such a model would be useless for classifying \n",
    "new digit samples. Always keep this in mind: deep learning models tend to be good at fitting to the training data, but the real challenge \n",
    "is generalization, not fitting.\n",
    "\n",
    "On the other hand, if the network has limited memorization resources, it will not be able to learn this mapping as easily, and thus, in \n",
    "order to minimize its loss, it will have to resort to learning compressed representations that have predictive power regarding the targets \n",
    "-- precisely the type of representations that we are interested in. At the same time, keep in mind that you should be using models that have \n",
    "enough parameters that they won't be underfitting: your model shouldn't be starved for memorization resources. There is a compromise to be \n",
    "found between \"too much capacity\" and \"not enough capacity\".\n",
    "\n",
    "Unfortunately, there is no magical formula to determine what the right number of layers is, or what the right size for each layer is. You \n",
    "will have to evaluate an array of different architectures (on your validation set, not on your test set, of course) in order to find the \n",
    "right model size for your data. The general workflow to find an appropriate model size is to start with relatively few layers and \n",
    "parameters, and start increasing the size of the layers or adding new layers until you see diminishing returns with regard to the \n",
    "validation loss.\n",
    "\n",
    "Let's try this on our movie review classification network. Our original network was as such:\n",
    "\n",
    "# 克服过拟合\n",
    "## 减小网络大小\n",
    "\n",
    "防止过拟合的最简单的方法就是减小模型大小，即减少模型中可学习参数的个数（这由层 数和每层的单元个数决定）。在深度学习中，模型中可学习参数的个数通常被称为模型的容量（capacity）。直观上来看，参数更多的模型拥有更大的记忆容量（memorization capacity），因此能 够在训练样本和目标之间轻松地学会完美的字典式映射，这种映射没有任何泛化能力。例如，拥 有 500 000 个二进制参数的模型，能够轻松学会 MNIST 训练集中所有数字对应的类别——我们 只需让 50 000 个数字每个都对应 10 个二进制参数。但这种模型对于新数字样本的分类毫无用处。 始终牢记：深度学习模型通常都很擅长拟合训练数据，但真正的挑战在于泛化，而不是拟合。\n",
    "\n",
    "与此相反，如果网络的记忆资源有限，则无法轻松学会这种映射。因此，为了让损失最小化，网络必须学会对目标具有很强预测能力的压缩表示，这也正是我们感兴趣的数据表示。同时请记住，你使用的模型应该具有足够多的参数，以防欠拟合，即模型应避免记忆资源不足。在容 量过大与容量不足之间要找到一个折中。\n",
    "\n",
    "不幸的是，没有一个魔法公式能够确定最佳层数或每层的最佳大小。你必须评估一系列不 同的网络架构（当然是在验证集上评估，而不是在测试集上），以便为数据找到最佳的模型大小。 要找到合适的模型大小，一般的工作流程是开始时选择相对较少的层和参数，然后逐渐增加层的大小或增加新层，直到这种增加对验证损失的影响变得很小。\n",
    "\n",
    "我们在电影评论分类的网络上试一下。原始网络如下所示。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "original_model = models.Sequential()\n",
    "original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "original_model.add(layers.Dense(16, activation='relu'))\n",
    "original_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "original_model.compile(optimizer='rmsprop',\n",
    "                       loss='binary_crossentropy',\n",
    "                       metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to replace it with this smaller network:\n",
    "\n",
    "现在我们尝试用下面这个更小的网络来替换它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "smaller_model = models.Sequential()\n",
    "smaller_model.add(layers.Dense(4, activation='relu', input_shape=(10000,)))\n",
    "smaller_model.add(layers.Dense(4, activation='relu'))\n",
    "smaller_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "smaller_model.compile(optimizer='rmsprop',\n",
    "                      loss='binary_crossentropy',\n",
    "                      metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Here's a comparison of the validation losses of the original network and the smaller network. The dots are the validation loss values of \n",
    "the smaller network, and the crosses are the initial network (remember: a lower validation loss signals a better model).\n",
    "\n",
    "比较了原始网络与更小网络的验证损失。圆点是更小网络的验证损失值，十字是原 始网络的验证损失值（请记住，更小的验证损失对应更好的模型）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 4s 172us/step - loss: 0.4440 - acc: 0.8251 - val_loss: 0.3286 - val_acc: 0.8835\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 4s 156us/step - loss: 0.2573 - acc: 0.9078 - val_loss: 0.2864 - val_acc: 0.8882\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 4s 147us/step - loss: 0.1991 - acc: 0.9292 - val_loss: 0.2821 - val_acc: 0.8891\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 4s 148us/step - loss: 0.1666 - acc: 0.9412 - val_loss: 0.2939 - val_acc: 0.8844\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 4s 148us/step - loss: 0.1435 - acc: 0.9501 - val_loss: 0.3116 - val_acc: 0.8804\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.1257 - acc: 0.9558 - val_loss: 0.3483 - val_acc: 0.8721\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.1104 - acc: 0.9615 - val_loss: 0.3598 - val_acc: 0.8722\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 4s 150us/step - loss: 0.0977 - acc: 0.9669 - val_loss: 0.3975 - val_acc: 0.8662\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.0841 - acc: 0.9721 - val_loss: 0.4339 - val_acc: 0.8611\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.0755 - acc: 0.9757 - val_loss: 0.4997 - val_acc: 0.8524\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.0684 - acc: 0.9778 - val_loss: 0.4872 - val_acc: 0.8587\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.0563 - acc: 0.9831 - val_loss: 0.5220 - val_acc: 0.8557\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 4s 149us/step - loss: 0.0513 - acc: 0.9843 - val_loss: 0.5351 - val_acc: 0.8579\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.0423 - acc: 0.9881 - val_loss: 0.6093 - val_acc: 0.8501\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.0363 - acc: 0.9894 - val_loss: 0.6115 - val_acc: 0.8558\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.0310 - acc: 0.9909 - val_loss: 0.6426 - val_acc: 0.8536\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.0265 - acc: 0.9926 - val_loss: 0.7694 - val_acc: 0.8417\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.0208 - acc: 0.9946 - val_loss: 0.7221 - val_acc: 0.8516\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.0198 - acc: 0.9949 - val_loss: 0.7739 - val_acc: 0.8476\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.0151 - acc: 0.9965 - val_loss: 0.9725 - val_acc: 0.8298\n"
     ]
    }
   ],
   "source": [
    "original_hist = original_model.fit(x_train, y_train,\n",
    "                                   epochs=20,\n",
    "                                   batch_size=512,\n",
    "                                   validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 4s 161us/step - loss: 0.5804 - acc: 0.7027 - val_loss: 0.5309 - val_acc: 0.7558\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.4843 - acc: 0.8455 - val_loss: 0.4876 - val_acc: 0.8280\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 4s 150us/step - loss: 0.4389 - acc: 0.8920 - val_loss: 0.4663 - val_acc: 0.8497\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 4s 150us/step - loss: 0.4077 - acc: 0.9154 - val_loss: 0.4526 - val_acc: 0.8663\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.3832 - acc: 0.9306 - val_loss: 0.4513 - val_acc: 0.8620\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.3620 - acc: 0.9415 - val_loss: 0.4424 - val_acc: 0.8728\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.3425 - acc: 0.9515 - val_loss: 0.4489 - val_acc: 0.8671\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 4s 160us/step - loss: 0.3261 - acc: 0.9560 - val_loss: 0.4515 - val_acc: 0.8675\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 4s 163us/step - loss: 0.3099 - acc: 0.9613 - val_loss: 0.4687 - val_acc: 0.8626\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 4s 166us/step - loss: 0.2961 - acc: 0.9649 - val_loss: 0.4632 - val_acc: 0.8649\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 4s 163us/step - loss: 0.2816 - acc: 0.9693 - val_loss: 0.4883 - val_acc: 0.8597\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 4s 165us/step - loss: 0.2684 - acc: 0.9722 - val_loss: 0.4756 - val_acc: 0.8644\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.2567 - acc: 0.9749 - val_loss: 0.5080 - val_acc: 0.8598\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 4s 156us/step - loss: 0.2455 - acc: 0.9763 - val_loss: 0.5514 - val_acc: 0.8548\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 4s 150us/step - loss: 0.2340 - acc: 0.9779 - val_loss: 0.5259 - val_acc: 0.8596\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.2239 - acc: 0.9798 - val_loss: 0.5398 - val_acc: 0.8577\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 4s 157us/step - loss: 0.2151 - acc: 0.9808 - val_loss: 0.6084 - val_acc: 0.8521\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 4s 160us/step - loss: 0.2052 - acc: 0.9824 - val_loss: 0.6265 - val_acc: 0.8517\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.1966 - acc: 0.9835 - val_loss: 0.5736 - val_acc: 0.8539\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 4s 160us/step - loss: 0.1887 - acc: 0.9839 - val_loss: 0.7008 - val_acc: 0.8476\n"
     ]
    }
   ],
   "source": [
    "smaller_model_hist = smaller_model.fit(x_train, y_train,\n",
    "                                       epochs=20,\n",
    "                                       batch_size=512,\n",
    "                                       validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = range(1, 21)\n",
    "original_val_loss = original_hist.history['val_loss']\n",
    "smaller_model_val_loss = smaller_model_hist.history['val_loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X+cFNWZ7/HPwwASEH9CDArMIIEIOAPCCLImEVZRdBcwmqjIazeQGK4aFb3ZXI1kZdCYaG4SN/6MmBjMZhbjj6tyc8lqVJDoqmFwAQVWQAI6wuqIASSD4YfP/aNqmmbonu6Z7urqnvm+X696TdfpU1XPNE09U+ecOmXujoiICECnuAMQEZHioaQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJkSUFM3vQzN43szfSvG9mdqeZbTCzVWY2MqpYREQkO50j3Pd84G7gV2nePxcYFC5jgPvCny3q1auXV1RU5CdCEZEOYvny5R+4e+9M9SJLCu6+1MwqWqgyBfiVB/NsvGJmR5lZH3ff2tJ+KyoqqKury2OkIiLtn5ltzqZenH0KJwDvJK3Xh2WHMLOZZlZnZnUNDQ0FCU5EpCOKMylYirKUs/O5+zx3r3b36t69M179iIhIG8WZFOqBfknrfYEtMcUiIiJE29GcyULgKjN7mKCDeUem/oR09u7dS319PR9//HFeA5TodOvWjb59+9KlS5e4QxGRJJElBTNbAIwDeplZPTAH6ALg7j8DFgHnARuARmBGW49VX19Pz549qaiowCxVq5QUE3dn27Zt1NfXM2DAgLjDEZEkUY4+mprhfQe+mY9jffzxx0oIJcTMOPbYY9GgAZHWqakJlii1mzualRBKi/69RFpv7tzoj9FukoKIiOROSSFP6uvrmTJlCoMGDWLgwIHMmjWLPXv2pKy7ZcsWvvzlL2fc53nnncf27dvbFE9NTQ0/+tGP2rRttubPn89VV12Vcx0RSa+mBsyCBQ68jqoZqUMnhXx9qO7OBRdcwPnnn8/69etZt24du3btYvbs2YfU3bdvH8cffzyPPfZYxv0uWrSIo446Kj9BikhJqqkB92CBA6+VFCKQr/a5559/nm7dujFjRjCAqqysjDvuuIMHH3yQxsZG5s+fz1e+8hUmTZrE2WefzaZNmzj55JMBaGxs5KKLLqKqqoqLL76YMWPGJKbxqKio4IMPPmDTpk0MGTKEb3zjGwwbNoyzzz6b3bt3A/DAAw9w6qmnMnz4cC688EIaGxtbjHX69OlcccUVjB8/nhNPPJEXXniBr33tawwZMoTp06cn6i1YsIDKykpOPvlkrr/++kT5L3/5SwYPHswZZ5zBSy+9lChvaGjgwgsv5NRTT+XUU0896D0RKR0dOinky+rVqxk1atRBZUcccQT9+/dnw4YNALz88ss89NBDPP/88wfVu/feezn66KNZtWoV//zP/8zy5ctTHmP9+vV885vfZPXq1Rx11FE8/vjjAFxwwQUsW7aMlStXMmTIEH7xi19kjPfPf/4zzz//PHfccQeTJk3iuuuuY/Xq1bz++uusWLGCLVu2cP311/P888+zYsUKli1bxpNPPsnWrVuZM2cOL730Er///e9Zs2ZNYp+zZs3iuuuuY9myZTz++ONcdtllrfoMRSSzOXOiP0acN6/Foqbm4CuEpna6OXPafjnm7ilH0ySXT5gwgWOOOeaQOi+++CKzZs0C4OSTT6aqqirlMQYMGMCIESMAGDVqFJs2bQLgjTfe4Lvf/S7bt29n165dnHPOORnjnTRpEmZGZWUlxx13HJWVlQAMGzaMTZs2sXnzZsaNG0fTlCLTpk1j6dKlAAeVX3zxxaxbtw6AZ5999qAksXPnTj766KOMsYhI9qIejgodNCk0fbBmB9rpcjFs2LDEX+5Ndu7cyTvvvMPAgQNZvnw5PXr0SLmtZxnAYYcdlnhdVlaWaD6aPn06Tz75JMOHD2f+/PksWbIk63116tTpoP126tSJffv20blz+q9FuqGkn3zyCS+//DKf+tSnsvl1RKRIqfkoD84880waGxv51a+CR0fs37+fb33rW0yfPp3u3bu3uO3nP/95HnnkEQDWrFnD66+/3qpjf/TRR/Tp04e9e/dSW1vbtl+gmTFjxvDCCy/wwQcfsH//fhYsWMAZZ5zBmDFjWLJkCdu2bWPv3r08+uijiW3OPvts7r777sT6ihUr8hKLiBRWh04K+WqfMzOeeOIJHn30UQYNGsTgwYPp1q0b3//+9zNue+WVV9LQ0EBVVRW33347VVVVHHnkkVkf+5ZbbmHMmDFMmDCBk046KZdfI6FPnz784Ac/YPz48QwfPpyRI0cyZcoU+vTpQ01NDWPHjuWss85i5MgDD8u78847qauro6qqiqFDh/Kzn/0sL7GISGFZts0XxaK6utqbP2Rn7dq1DBkyJKaIcrN//3727t1Lt27deOuttzjzzDNZt24dXbt2jTu0yJXyv5tIqTGz5e5enaleh+tTKDaNjY2MHz+evXv34u7cd999HSIhiEhxUlKIWc+ePfV4UREpGh26T0FERA6mpCAiIglKCiIikqCkICIiCUoKeXLrrbcybNgwqqqqGDFiBK+++mpe9nv44YcDHDSJXjEYN25cxg7ybOqISHHpkEmhthYqKqBTp+BnrjcCv/zyy/z2t7/ltddeY9WqVTz77LP069cvH6G22f79+2M9voiUpkiTgplNNLM3zWyDmd2Q4v1yM3vOzFaZ2RIz6xtlPBAkgJkzYfPmYN6jzZuD9VwSw9atW+nVq1diHqFevXpx/PHHA8H01zfeeCNjx46lurqa1157jXPOOYeBAwcm7vrdtWsXZ555JiNHjqSyspKnnnqqxePt37+fb3/725x66qlUVVVx//33A7BkyRLGjx/PpZdempjkLtnhhx/O9ddfz6hRozjrrLP44x//yLhx4zjxxBNZuHAhEDzvesaMGVRWVnLKKaewePFiAHbv3s0ll1ySmOK7ae4lgGeeeYaxY8cycuRIvvKVr7Br1662f5giEi93j2QByoC3gBOBrsBKYGizOo8CXw1f/y3wr5n2O2rUKG9uzZo1h5SlU17e9IiKg5fy8qx3cYiPPvrIhw8f7oMGDfIrrrjClyxZknS8cr/33nvd3f3aa6/1yspK37lzp7///vveu3dvd3ffu3ev79ixw93dGxoafODAgf7JJ5+4u3uPHj3c3f1Pf/qTDxs2zN3d77//fr/lllvc3f3jjz/2UaNG+caNG33x4sXevXt337hxY8o4AV+0aJG7u59//vk+YcIE37Nnj69YscKHDx/u7u4/+tGPfPr06e7uvnbtWu/Xr5/v3r3bf/zjH/uMGTPc3X3lypVeVlbmy5Yt84aGBv/CF77gu3btcnf32267zefOnevu7meccYYvW7Ys7efWmn83EckNUOdZnLujvHltNLDB3TcCmNnDwBRgTVKdocB14evFwJMRxgPA22+3rjwbhx9+OMuXL+cPf/gDixcv5uKLL+a2225LPLRm8uTJAFRWVrJr1y569uxJz5496datG9u3b6dHjx7ceOONLF26lE6dOvHuu+/y3nvv8ZnPfCbl8Z555hlWrVqVeHrbjh07WL9+PV27dmX06NEMGDAg5XZdu3Zl4sSJiVgOO+wwunTpQmVlZWIq7hdffJGrr74agJNOOony8nLWrVvH0qVLueaaawCoqqpKTPH9yiuvsGbNGk4//XQA9uzZw9ixY9v+YYpIrKJMCicA7ySt1wNjmtVZCVwI/BT4EtDTzI51921RBdW/f9BklKo8F2VlZYwbN45x48ZRWVnJQw89lEgKmaaqrq2tpaGhgeXLl9OlSxcqKir4+OOP0x7L3bnrrrsOeXbCkiVL0k7RDdClS5fE1NfJsTTF0bTvdNI9M2LChAksWLAg7XYiUjqi7FNINfF+8zPOPwFnmNl/AmcA7wL7DtmR2UwzqzOzuoaGhpyCuvVWaD6bdffuQXlbvfnmm6xfvz6xvmLFCsrLy7PefseOHXz605+mS5cuLF68mM2pslaSc845h/vuu4+9e/cCsG7dOv7yl7+0LfhmvvjFLyam4F63bh1vv/02n/vc5w4qf+ONN1i1ahUAp512Gi+99FLiCXONjY2JB++ISOmJ8kqhHkgegtMX2JJcwd23ABcAmNnhwIXuvqP5jtx9HjAPgllScwlq2rTg5+zZQZNR//5BQmgqb4tdu3Zx9dVXs337djp37sxnP/tZ5s2b14qYpjFp0iSqq6sZMWJEximwL7vsMjZt2sTIkSNxd3r37s2TT+an5e3KK6/k8ssvp7Kyks6dOzN//nwOO+wwrrjiCmbMmJEYcjt69GgAevfuzfz585k6dSp//etfAfje977H4MGD8xKPiBRWZFNnm1lnYB1wJsEVwDLgUndfnVSnF/Chu39iZrcC+939ppb2296mzu7I9O8mUjjZTp0dWfORu+8DrgKeBtYCj7j7ajO72cwmh9XGAW+a2TrgOCCHRhwREclVpFNnu/siYFGzspuSXj8GPBZlDCIikr12c0dzVM1gEg39e4kUp3aRFLp168a2bdt0oikR7s62bdvo1q1b3KGISDPt4slrffv2pb6+nlyHq0rhdOvWjb59I5/VRERaqV0khS5duqS9i1dERLLXLpqPREQkP5QUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBRERCQh0qRgZhPN7E0z22BmN6R4v7+ZLTaz/zSzVWZ2XpTxiIhIyyJLCmZWBtwDnAsMBaaa2dBm1b4LPOLupwCXAPdGFY+IiGQW5ZXCaGCDu2909z3Aw8CUZnUcOCJ8fSSwJcJ4REQkgyif0XwC8E7Sej0wplmdGuAZM7sa6AGcFWE8IiKSQZRXCpaizJutTwXmu3tf4DzgX83skJjMbKaZ1ZlZXUNDQwShiogIRJsU6oF+Set9ObR56OvAIwDu/jLQDejVfEfuPs/dq929unfv3hGFKyIiUSaFZcAgMxtgZl0JOpIXNqvzNnAmgJkNIUgKuhQQEYlJZEnB3fcBVwFPA2sJRhmtNrObzWxyWO1bwDfMbCWwAJju7s2bmEREpECi7GjG3RcBi5qV3ZT0eg1wepQxiIgUi5qaYClmuqNZRDqMuE/Ic+fGe/xsKCmISIdRCifluCkpiIhEqKYGzIIFDryO+6olHSUFEWnX4j4p19SAe7DAgdfFmhSs1Ab7VFdXe11dXdxhiEgJMjtwcu5oxzez5e5enamerhRERApkzpy4I8hMSUFEOoy4T8rF2mSUTElBRDqMUjgpx01JQUREEjImBTPr0TRzqZkNNrPJZtYl+tBERKTQsrlSWAp0M7MTgOeAGcD8KIMSEZF4ZJMUzN0bgQuAu9z9SwSP1xQRkXYmq6RgZmOBacD/C8sinUhPRETikU1SuBb4DvBEOPX1icDiaMMSEZE4ZEwK7v6Cu09299vDDucP3P2aAsQmIlJUOsKQ1mxGH/2bmR1hZj2ANcCbZvbt6EMTESkuHWGW1Wyaj4a6+07gfIIH5vQH/iHSqEREJBbZJIUu4X0J5wNPufteoLRm0RMRaaO4Z1kttGySwv3AJqAHsNTMyoGdUQYlIlIsSm3q61xlHFrq7ncCdyYVbTaz8dGFJCIiccmmo/lIM/uJmdWFy48JrhoyMrOJZvammW0wsxtSvH+Hma0Il3Vmtr0Nv4OISEHENctqbS1UVECnTsHP2trojpXNTWgPAm8AF4Xr/wD8kuAO57TMrAy4B5gA1APLzGyhu69pquPu1yXVvxo4pVXRi4gUUBxNRrW1MHMmNDYG65s3B+sA06bl/3jZ9CkMdPc57r4xXOYCJ2ax3WhgQ7jNHuBhYEoL9acCC7LYr4hIhzF79oGE0KSxMSiPQjZJYbeZfb5pxcxOB3Znsd0JwDtJ6/Vh2SHCzusBwPNp3p/Z1HzV0NCQxaFFRNqHt99uXXmusmk+ugJ4yMyOBAz4EJiexXaWoizdUNZLgMfcfX+qN919HjAPgmc0Z3FsEZF2oX//oMkoVXkUspnmYoW7DweqgEp3P8XdV2ax73qgX9J6X2BLmrqXoKYjEZFD3HordO9+cFn37kF5FNJeKZjZ/0xTDoC7/yTDvpcBg8xsAPAuwYn/0hT7+xxwNPBydiGLiHQcTZ3Js2cHTUb9+wcJIYpOZmi5+ahnLjt2931mdhXwNFAGPBjOsnozUOfuC8OqU4GH3V3NQiIiKUybFl0SaM5K7VxcXV3tdXV1cYchIlJSzGy5u1dnqpfN6CMREekglBRERCRBSUFERBIy3qdgZocBFwIVyfXd/ebowhIRkThkc/PaU8AOYDnw12jDERGROGWTFPq6+8TIIxERkdhl06fwH2ZWGXkkIiISu2yuFD4PTDezPxE0Hxng7l4VaWQiIlJw2SSFcyOPQkQkCzU17fcxmMUimwnxNgNHAZPC5aiwTESkoObOjTuC9i+bx3HOAmqBT4fLr8OnpImISDuTTUfz14Ex7n6Tu98EnAZ8I9qwREQCNTVgFixw4LWakaKRTVIwIPnhN/tJ/QAdEZG8q6kB92CBA68LmRRqa6GiAjp1Cn7W1hbu2IWWTUfzL4FXzeyJcP184BfRhSQiUjxqa2HmzAPPSd68OViHwk1nXUjZdDT/BJhB8BjOPwMz3P1fog5MRKS5OXMKf8zZsw8khCaNjUF5e5T2eQpmdoS77zSzY1K97+4fRhpZGnqegogUUqdOB5qukpnBJ58UPp62yvZ5Ci01H/0b8PcEcx4lfyQWrp+YU4QiIiWgf/+gyShVeXuUtvnI3f8+/DnA3U9MWga4uxKCSAeUa+duXCOGcukovvVW6N794LLu3YPy9iib+xSey6ZMRNq/XG8ei+Pms6aO4s2bg2agpo7ibBPDtGkwbx6UlwdNRuXlwXp77GSGlvsUugHdgcXAOA4MQz0C+J27D8m4c7OJwE+BMuDn7n5bijoXATUETVIr3f3SlvapPgWR+Jilbl8v1PZtUVGRuvmnvBw2bSpsLHHKxzOa/wdBf8JJ4c+m5SngniwCKAvrnQsMBaaa2dBmdQYB3wFOd/dhwLWZ9isihZXrzWNx33z29tutK+/o0l4pJCqYXe3ud7V6x2ZjgRp3Pydc/w6Au/8gqc4PgXXu/vNs96srBZH46EqhdOXjSgEAd7/LzE42s4vM7B+blixiOAF4J2m9PixLNhgYbGYvmdkrYXOTiEjedLSO4lxl84zmOQR9CkOBRQTNQS8Cv8q0aYqy5n8jdAYGhfvvC/zBzE529+3NYpgJzATo317HgYmUgFxvHovj5rOmDuHZs4Mmo/79g4TQXjuKc5VN89HrwHDgP919uJkdR9BpPCnDdtk0H/0MeMXd54frzwE3uPuydPtV85GISOvlrfkI2O3unwD7zOwI4H2yu3FtGTDIzAaYWVfgEmBhszpPAuPDgHsRNCdtzGLfIiISgWwmxKszs6OABwhGH+0C/phpI3ffZ2ZXAU8TDEl90N1Xm9nNQJ27LwzfO9vM1hDMvvptd9/Wxt9FRERylLH56KDKZhXAEe6+KqqAMlHzkYiUmtra+Ps0cp77yMxGtvSeu7/W1uBERDqKUpt6u6U7mheHL7sB1cBKghFFVcCr7v75gkTYjK4URKSUFMt9Ejl3NLv7eHcfD2wGRrp7tbuPAk4BNuQvVBGR9qvU7qjOZvTRSe7+etOKu78BjIguJBGR9iPdrVXFestVNklhrZn93MzGmdkZZvYAsDbqwERE2oNSu6M6m6QwA1gNzCKYsG5NWCYiIhmU2tTbrRqSWgzU0SxSeMUwpFJyk48hqY+4+0XhNBeHZA53r8oxRhEpAaU2pFJy09KQ1D7uvtXMylO97+4pBllFT1cKIoVVLEMqJTf5GJK6Nfy5OdWSz2BFpHgVw5DKXJ6xLK2TNimY2UdmtjPF8pGZ7SxkkCKSH2152lncQypzfcaytE5LVwo93f2IFEtPdz+ikEGKSH7Mndv6beIeUjl79oH+jCaNjUG55F82Q1IBMLNPm1n/piXKoPJNl54ibRf3kMpiaL7qSDImBTObbGbrgT8BLwCbgN9FHFfe6NJTOrqamuBkbuGzEJtet6Ypadq0oFP5k0+Cn61NCLn8YRZ381WH4+4tLgQT4R1L8OQ1CB6KMy/TdlEto0aN8tYoL3cP0sHBS3l5q3YjUhTmzMlte8hLGK3y61+7d+9+8P+/7t2D8kJsLwGC59hkPMdm03y014MH33Qys07uvpgSmvtIl57SnrSlTyBuufYJxN181dFk8+S17WZ2OLAUqDWz94F90YaVP/37px5jrUtP6YjmzCn8MfPxh9m0aUoChZLNlcIUYDdwHfDvwFvApCiDyqe4R06I5CoffQLJ+yo09QmUlpbuU7jbzP7G3f/i7vvdfZ+7P+Tud3oJPUdZl55S6mpqDrSmw4HXrTnBxzkCT3+YlZaWmo/WAz82sz7Ab4AF7r6iMGHlly49pSOLe+6ipmNoQr3S0NLNaz9197HAGcCHwC/NbK2Z3WRmg7PZuZlNNLM3zWyDmd2Q4v3pZtZgZivC5bI2/yYiHUBb+gSK4eavXIe0SuFk7FPwYK6j2939FOBS4Etk8ZAdMysD7gHOBYYCU81saIqqv3H3EeHy89aFL1I4cbTHN9eWGDQCT1ojm5vXupjZJDOrJbhpbR1wYRb7Hg1scPeN7r4HeJig01qkJJXicFBQR6+0TksdzRPM7EGgHpgJLAIGuvvF7v5kFvs+AXgnab0+LGvuQjNbZWaPmVm/VsQuIllQR6+0RktXCjcCLwND3H2Su9e6+19asW9LUdb84Q3/F6jw4IE9zwIPpdyR2UwzqzOzuoaGhlaEkB+aO6njysdw0Kbvj1k83x+NwJPWiOxxnGY2Fqhx93PC9e8AuPsP0tQvAz509yNb2m+hH7LTfOQGBH9l6T9Vx2N2YFhotvT9kWKR80N28mAZMMjMBphZV+ASYGFyhXC4a5PJZNGBXWjFMHJDSpe+P1Jqspnmok3cfZ+ZXQU8DZQBD7r7ajO7mWBipoXANWY2mWDajA+B6VHF01YauSFN2jIcNNUUKy2Vi8QtyisF3H2Ruw9294HufmtYdlOYEHD377j7MHcf7u7j3f2/ooynLTRyQ5q0ZThoeconnKcvT0f9WlIokSaF9kAjN0pfqU/xoGeCSEFlM792MS2tfZ5CPvz618HzF8yCn62dxz3X7aXtimEu/qZ//6bneLT22HomiOQDWT5PIbLRR1Ep9OijXGn0SbwqKlK335eXB9MtZKO2Nt55ezp1Sj3qySyYNkIkG8Uw+kjQ6JO45TpQoBiabtSvJYWkpBAxjV6KV64n1GJI6urXkkJSUohYPv7Ky7WjtNRHruQSf64n1GJI6rojWQoqm46HYlri6GjORdwPLS+mjta2dLTnI/5cjq9OXmkvyLKjOfaTfGuXUksK7vGelOI+qeV6Ui/1+EWKRbZJQaOPilyuI0/yMXIll9E3uY7+KYaRN3GPPhLJB40+aidy7ZPIdftcR9/k2iZfDCNv9NQw6UiUFIpcrh2luW6f6+ibXE/q+Rx5UwxPThMpetm0MRXTUop9CrmK845qs9Rt+mbZHzvOjuJk0LbtRNoD1Kcg+dAe7ghu0pbnIYi0F+pTkLzIR/NNnG3y+XhymkhHoisFyahY/tLPla4UpCPL9kohsofsSPsxbVppJgERaT01H0mH0ZYnp4l0NEoK0mGoH0EkMyUFERFJiDQpmNlEM3vTzDaY2Q0t1PuymbmZZewEERGR6ESWFMysDLgHOBcYCkw1s6Ep6vUErgFejSoWERHJTpRXCqOBDe6+0d33AA8DU1LUuwX4IfBxhLFIO6A+AZHoRZkUTgDeSVqvD8sSzOwUoJ+7/zbCOKSdmDs37ghE2r8ok4KlKEvcOmRmnYA7gG9l3JHZTDOrM7O6hoaGPIYoIiLJokwK9UC/pPW+wJak9Z7AycASM9sEnAYsTNXZ7O7z3L3a3at79+4dYchSbDRNhUhhRTbNhZl1BtYBZwLvAsuAS919dZr6S4B/cvcW57DQNBcdl6apEGm72CfEc/d9wFXA08Ba4BF3X21mN5vZ5KiOKyIibRfp3EfuvghY1KzspjR1x0UZi5Q+TVMhEj3d0SwlQ/0IItFTUhARkQQlBRERSVBSEBGRBCUFKRj1CYgUPyUFKRhNUyFS/JQUREQkQUlBstaW5h9NUyFSWiKb5iIqmuYiPrlOM6FpKkTiE/s0FyIiUnqUFKRF+Wz+0TQVIsVPzUeSNTX/iJQuNR+JiEirdaikoBEvuVHzj0j716Gaj9T8ISIdlZqPRESk1dp9UtDNUyIi2esQScH9QLNR0+uOmBQ64u8sIq3T7pOCHKAJ6UQkkw6VFDR6RkSkZZEmBTObaGZvmtkGM7shxfuXm9nrZrbCzF40s6FRxtMRm0/UpyIirRHZkFQzKwPWAROAemAZMNXd1yTVOcLdd4avJwNXuvvElvarO5rbTkNyRTquYhiSOhrY4O4b3X0P8DAwJblCU0II9QB0yhIRiVGUSeEE4J2k9fqw7CBm9k0zewv4IXBNhPGUvFybfNSnIiKZRJkULEXZIVcC7n6Puw8Erge+m3JHZjPNrM7M6hoaGvIcZunIdfSQ+hFEJJMok0I90C9pvS+wpYX6DwPnp3rD3ee5e7W7V/fu3TuPIbaOTqoi0t5FmRSWAYPMbICZdQUuARYmVzCzQUmrfwesjzCenMXxl7pGD4lIIUU6IZ6ZnQf8C1AGPOjut5rZzUCduy80s58CZwF7gT8DV7n76pb2Gefoo7gfR6nRQyLSVsUw+gh3X+Tug919oLvfGpbd5O4Lw9ez3H2Yu49w9/GZEkIc9Je6iHQkHeqO5rbIde4kPc5SREpJh3qeQq7U/CMipaoomo/aG/2lLiLtnZJCK+jmMRFp75QUCkid0yJS7JQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJKHkbl4zswZgc9xxpNEL+CDuIFqg+HJT7PFB8ceo+HKTS3zl7p5xmumSSwrjy/MeAAAGZklEQVTFzMzqsrljMC6KLzfFHh8Uf4yKLzeFiE/NRyIikqCkICIiCUoK+TUv7gAyUHy5Kfb4oPhjVHy5iTw+9SmIiEiCrhRERCRBSaGVzKyfmS02s7VmttrMZqWoM87MdpjZinC5qcAxbjKz18NjH/LwCQvcaWYbzGyVmY0sYGyfS/pcVpjZTjO7tlmdgn9+Zvagmb1vZm8klR1jZr83s/Xhz6PTbPvVsM56M/tqgWL732b2X+G/3xNmdlSabVv8LkQcY42ZvZv073hemm0nmtmb4ffxhgLG95uk2DaZ2Yo020b6GaY7p8T2/XN3La1YgD7AyPB1T2AdMLRZnXHAb2OMcRPQq4X3zwN+BxhwGvBqTHGWAf9NMH461s8P+CIwEngjqeyHwA3h6xuA21NsdwywMfx5dPj66ALEdjbQOXx9e6rYsvkuRBxjDfBPWXwH3gJOBLoCK5v/f4oqvmbv/xi4KY7PMN05Ja7vn64UWsndt7r7a+Hrj4C1wAnxRtVqU4BfeeAV4Cgz6xNDHGcCb7l77DcjuvtS4MNmxVOAh8LXDwHnp9j0HOD37v6hu/8Z+D0wMerY3P0Zd98Xrr4C9M3nMVsrzeeXjdHABnff6O57gIcJPve8aik+MzPgImBBvo+bjRbOKbF8/5QUcmBmFcApwKsp3h5rZivN7HdmNqyggYEDz5jZcjObmeL9E4B3ktbriSexXUL6/4hxfn5NjnP3rRD8xwU+naJOMXyWXyO48ksl03chaleFTVwPpmn+KIbP7wvAe+6+Ps37BfsMm51TYvn+KSm0kZkdDjwOXOvuO5u9/RpBk8hw4C7gyQKHd7q7jwTOBb5pZl9s9r6l2Kagw9DMrCswGXg0xdtxf36tEetnaWazgX1AbZoqmb4LUboPGAiMALYSNNE0F/t3EZhKy1cJBfkMM5xT0m6Woiynz09JoQ3MrAvBP16tu/+f5u+7+0533xW+XgR0MbNehYrP3beEP98HniC4RE9WD/RLWu8LbClMdAnnAq+5+3vN34j780vyXlOzWvjz/RR1Yvssw07FvwemedjA3FwW34XIuPt77r7f3T8BHkhz7Fi/i2bWGbgA+E26OoX4DNOcU2L5/ikptFLY/vgLYK27/yRNnc+E9TCz0QSf87YCxdfDzHo2vSbokHyjWbWFwD+Go5BOA3Y0XaYWUNq/zuL8/JpZCDSN5vgq8FSKOk8DZ5vZ0WHzyNlhWaTMbCJwPTDZ3RvT1MnmuxBljMn9VF9Kc+xlwCAzGxBePV5C8LkXylnAf7l7fao3C/EZtnBOief7F1WPentdgM8TXJ6tAlaEy3nA5cDlYZ2rgNUEIyleAf6mgPGdGB53ZRjD7LA8OT4D7iEY9fE6UF3gz7A7wUn+yKSyWD8/ggS1FdhL8NfX14FjgeeA9eHPY8K61cDPk7b9GrAhXGYUKLYNBG3JTd/Bn4V1jwcWtfRdKODn96/h92sVwQmuT/MYw/XzCEbcvBVVjKniC8vnN33vkuoW9DNs4ZwSy/dPdzSLiEiCmo9ERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBJGRm++3gGVzzNmOnmVUkz9ApUqw6xx2ASBHZ7e4j4g5CJE66UhDJIJxP/3Yz+2O4fDYsLzez58IJ354zs/5h+XEWPONgZbj8TbirMjN7IJwz/xkz+1RY/xozWxPu5+GYfk0RQElBJNmnmjUfXZz03k53Hw3cDfxLWHY3wRTkVQQT0t0Zlt8JvODBhH4jCe6EBRgE3OPuw4DtwIVh+Q3AKeF+Lo/qlxPJhu5oFgmZ2S53PzxF+Sbgb919Yzhx2X+7+7Fm9gHB1A17w/Kt7t7LzBqAvu7+16R9VBDMez8oXL8e6OLu3zOzfwd2EcwG+6SHkwGKxEFXCiLZ8TSv09VJ5a9Jr/dzoE/v7wjmohoFLA9n7hSJhZKCSHYuTvr5cvj6Pwhm9QSYBrwYvn4OuALAzMrM7Ih0OzWzTkA/d18M/C/gKOCQqxWRQtFfJCIHfMoOfnj7v7t707DUw8zsVYI/pKaGZdcAD5rZt4EGYEZYPguYZ2ZfJ7giuIJghs5UyoBfm9mRBLPX3uHu2/P2G4m0kvoURDII+xSq3f2DuGMRiZqaj0REJEFXCiIikqArBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkYT/D9jgivnOPx5kAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20ad1631e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# b+ is for \"blue cross\"\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "As you can see, the smaller network starts overfitting later than the reference one (after 6 epochs rather than 4) and its performance \n",
    "degrades much more slowly once it starts overfitting.\n",
    "\n",
    "Now, for kicks, let's add to this benchmark a network that has much more capacity, far more than the problem would warrant:\n",
    "\n",
    "如你所见，更小的网络开始过拟合的时间要晚于参考网络（前者 6 轮后开始过拟合，而后者 4 轮后开始），而且开始过拟合之后，它的性能变差的速度也更慢。\n",
    "\n",
    "现在，为了好玩，我们再向这个基准中添加一个容量更大的网络（容量远大于问题所需）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigger_model = models.Sequential()\n",
    "bigger_model.add(layers.Dense(512, activation='relu', input_shape=(10000,)))\n",
    "bigger_model.add(layers.Dense(512, activation='relu'))\n",
    "bigger_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "bigger_model.compile(optimizer='rmsprop',\n",
    "                     loss='binary_crossentropy',\n",
    "                     metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 10s 415us/step - loss: 0.4641 - acc: 0.7966 - val_loss: 0.2839 - val_acc: 0.8859\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 11s 439us/step - loss: 0.2229 - acc: 0.9114 - val_loss: 0.2867 - val_acc: 0.8812\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 11s 453us/step - loss: 0.1329 - acc: 0.9518 - val_loss: 0.4041 - val_acc: 0.8474\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 12s 460us/step - loss: 0.0728 - acc: 0.9786 - val_loss: 0.4165 - val_acc: 0.8822\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 11s 442us/step - loss: 0.0783 - acc: 0.9856 - val_loss: 0.4597 - val_acc: 0.8804\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 11s 446us/step - loss: 0.0024 - acc: 0.9998 - val_loss: 0.6145 - val_acc: 0.8803\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 11s 442us/step - loss: 0.0903 - acc: 0.9883 - val_loss: 0.6565 - val_acc: 0.8739\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 11s 441us/step - loss: 3.0651e-04 - acc: 1.0000 - val_loss: 0.7293 - val_acc: 0.8801\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 11s 442us/step - loss: 0.0892 - acc: 0.9898 - val_loss: 0.6508 - val_acc: 0.8761\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 11s 442us/step - loss: 1.5321e-04 - acc: 1.0000 - val_loss: 0.7375 - val_acc: 0.8790\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 11s 442us/step - loss: 0.0797 - acc: 0.9901 - val_loss: 0.6648 - val_acc: 0.8770\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 11s 444us/step - loss: 1.0792e-04 - acc: 1.0000 - val_loss: 0.7347 - val_acc: 0.8787\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 11s 445us/step - loss: 2.6608e-05 - acc: 1.0000 - val_loss: 0.8078 - val_acc: 0.8797\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 11s 449us/step - loss: 6.0030e-06 - acc: 1.0000 - val_loss: 0.9082 - val_acc: 0.8788\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 11s 456us/step - loss: 0.0801 - acc: 0.9897 - val_loss: 0.8778 - val_acc: 0.8700\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 11s 445us/step - loss: 7.0182e-04 - acc: 1.0000 - val_loss: 0.8752 - val_acc: 0.8780\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 11s 452us/step - loss: 0.0826 - acc: 0.9903 - val_loss: 0.7967 - val_acc: 0.8765\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 11s 449us/step - loss: 7.9302e-04 - acc: 0.9999 - val_loss: 0.8242 - val_acc: 0.8780\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 11s 448us/step - loss: 6.5457e-04 - acc: 1.0000 - val_loss: 0.8800 - val_acc: 0.8793\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 11s 454us/step - loss: 6.4702e-04 - acc: 1.0000 - val_loss: 0.9431 - val_acc: 0.8794\n"
     ]
    }
   ],
   "source": [
    "bigger_model_hist = bigger_model.fit(x_train, y_train,\n",
    "                                     epochs=20,\n",
    "                                     batch_size=512,\n",
    "                                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how the bigger network fares compared to the reference one. The dots are the validation loss values of the bigger network, and the \n",
    "crosses are the initial network.\n",
    "\n",
    "下图显示了更大的网络与参考网络的性能对比。圆点是更大网络的验证损失值，十字是原始网络的验证损失值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X2cFPWV7/HPYURHFHwkuQgygy5EngZEHkLkBl0iPiSASqLibBRNJFFZiTe7VwyJjOayvoybGI2uuSQiZp1INK7Gu9eo8QmjMQljAiKggDqjI24cUIPs6MrA2T+qpukZeqa76a6u7unv+/WqV3dV/7r6TNP06fr9fnXK3B0RERGAXnEHICIixUNJQUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEiJLCma2zMzeMbOXunjczOwWM9tsZi+a2bioYhERkczsF+G+lwO3Aj/r4vHTgaHhMgm4Pbzt1pFHHunV1dX5iVBEpEy88MILW929f7p2kSUFd3/GzKq7aTIL+JkHdTZ+b2aHmtkAd3+7u/1WV1fT0NCQx0hFRHo+M2vKpF2cYwoDgTeT1pvDbXsxs3lm1mBmDS0tLQUJTkSkHMWZFCzFtpTV+dx9qbuPd/fx/funPfoREZF9FGdSaAaOTlofBGyJKRYRESHageZ0HgLmm9kKggHmv6YbT+jKzp07aW5u5qOPPsprgBKdyspKBg0aRO/eveMORUSSRJYUzOwe4CTgSDNrBhYDvQHc/cfAw8AZwGagFbhoX1+rubmZvn37Ul1djVmqXikpJu7Otm3baG5uZsiQIXGHIyJJopx9NCfN4w5cno/X+uijj5QQSoiZccQRR6BJAyLZqasLlij1mDOalRBKi/69RLJ37bXRv0aPSQoiIpI7JYU8aW5uZtasWQwdOpRjjz2WBQsW8PHHH6dsu2XLFr74xS+m3ecZZ5zB+++/v0/x1NXV8c///M/79NxMLV++nPnz5+fcRkS6VlcHZsECe+5H1Y1U1kkhX2+qu3P22Wdz5plnsmnTJjZu3MiOHTtYtGjRXm3b2to46qij+OUvf5l2vw8//DCHHnpofoIUkZJUVwfuwQJ77ispRCBf/XNPPvkklZWVXHRRMIGqoqKCm266iWXLltHa2sry5cv50pe+xIwZM5g+fTqNjY2MGjUKgNbWVs455xxqamo499xzmTRpUqKMR3V1NVu3bqWxsZHhw4dzySWXMHLkSKZPn86HH34IwE9+8hMmTJjAmDFjmD17Nq2trd3GOnfuXC699FJOPvlkjjnmGFauXMnFF1/M8OHDmTt3bqLdPffcw+jRoxk1ahRXXXVVYvudd97JsGHDmDp1Ks8991xie0tLC7Nnz2bChAlMmDChw2MiUjrKOinky7p16zjhhBM6bOvXrx+DBw9m8+bNADz//PPcddddPPnkkx3a/cu//AuHHXYYL774It/5znd44YUXUr7Gpk2buPzyy1m3bh2HHnoo999/PwBnn302q1atYs2aNQwfPpw77rgjbbzvvfceTz75JDfddBMzZszgyiuvZN26daxdu5bVq1ezZcsWrrrqKp588klWr17NqlWrePDBB3n77bdZvHgxzz33HL/5zW9Yv359Yp8LFizgyiuvZNWqVdx///189atfzeo9FJH0Fi+O/jXiPHktFnV1HY8Q2vvpFi/e98Mxd085myZ5+ymnnMLhhx++V5tnn32WBQsWADBq1ChqampSvsaQIUMYO3YsACeccAKNjY0AvPTSS3z729/m/fffZ8eOHZx66qlp450xYwZmxujRo/nkJz/J6NGjARg5ciSNjY00NTVx0kkn0V5SpLa2lmeeeQagw/Zzzz2XjRs3AvD44493SBLbt2/ngw8+SBuLiGQu6umoUKZJof2NNdvTT5eLkSNHJn65t9u+fTtvvvkmxx57LC+88AIHHXRQyud6hgEccMABifsVFRWJ7qO5c+fy4IMPMmbMGJYvX87TTz+d8b569erVYb+9evWira2N/fbr+mPR1VTS3bt38/zzz3PggQdm8ueISJFS91EeTJs2jdbWVn72s+DSEbt27eKb3/wmc+fOpU+fPt0+d8qUKdx7770ArF+/nrVr12b12h988AEDBgxg586d1NfX79sf0MmkSZNYuXIlW7duZdeuXdxzzz1MnTqVSZMm8fTTT7Nt2zZ27tzJfffdl3jO9OnTufXWWxPrq1evzkssIgL19VBdDb16Bbd5+q+eUlknhXz1z5kZDzzwAPfddx9Dhw5l2LBhVFZW8k//9E9pn3vZZZfR0tJCTU0NN9xwAzU1NRxyyCEZv/Z3v/tdJk2axCmnnMJxxx2Xy5+RMGDAAK6//npOPvlkxowZw7hx45g1axYDBgygrq6OyZMn87nPfY5x4/ZcLO+WW26hoaGBmpoaRowYwY9//OO8xCJS7urrYd48aGoKejaamoL1qBKDZdp9USzGjx/vnS+ys2HDBoYPHx5TRLnZtWsXO3fupLKykldffZVp06axceNG9t9//7hDi1wp/7uJFEp1dZAIOquqgnBoMSNm9oK7j0/XruzGFIpNa2srJ598Mjt37sTduf3228siIYhIZt54I7vtuVJSiFnfvn11eVER6dLgwamPFAYPjub1ynpMQUSk2C1ZAp3nq/TpE2yPgpKCiEgRq62FpUuDMQSz4Hbp0mB7FNR9JCJS5Gpro0sCnelIQUREEpQU8qSiooKxY8cm5vX/7ne/AzIvk13MDj744Ly0EZHiV5ZJIYqzAw888EBWr17NmjVruP7667n66qsBMi6TnYu2trZI9y9SyDNqJV6RJgUzO83MXjGzzWa2MMXjVWb2hJm9aGZPm9mgKOOBwpwduH37dg477DCAjMtk33HHHQwbNoyTTjqJSy65JHFhmq5KUtfV1TFv3jymT5/OBRdc0OH1n376aaZOnco555zDsGHDWLhwIfX19UycOJHRo0fz6quvAtDU1MS0adOoqalh2rRpvBFOfH799deZPHkyEyZM4Dvf+U6Hfd94441MmDCBmpoaFheiZKPErtBn1ErM3D2SBagAXgWOAfYH1gAjOrW5D7gwvP+3wL+m2+8JJ5zgna1fv36vbV2pqmq/REXHpaoq412k1KtXLx8zZox/6lOf8n79+nlDQ4O7u7/++us+cuRId3e/8cYbfd68ee7uvnbtWq+oqPBVq1b5W2+95VVVVb5t2zb/+OOPfcqUKX755Ze7u/ucOXP8t7/9rbu7NzU1+XHHHefu7osXL/Zx48Z5a2vrXrE89dRTfsghh/iWLVv8o48+8qOOOsqvueYad3f/4Q9/6AsWLHB39y984Qu+fPlyd3e/4447fNasWe7uPmPGDL/rrrvc3f3WW2/1gw46yN3dH330Ub/kkkt89+7dvmvXLv/85z/vK1eudHdPtMlGNv9uEp+o/s9IYQENnsF3d5RHChOBze7+mrt/DKwAZnVqMwJ4Irz/VIrH8y6qswPbu49efvllHnnkES644IK9KqA+++yznHfeeUDHMtl//OMfmTp1Kocffji9e/fmS1/6UuI5jz/+OPPnz2fs2LHMnDmzQ0nqmTNndlmVdMKECQwYMIADDjiAY489lunTpwMwevToRNnt559/nvPPPx+AL3/5yzz77LMAPPfcc8yZMyexvd1jjz3GY489xvHHH8+4ceN4+eWX2bRpU07vmxS/Qp9RK/GKckrqQODNpPVmYFKnNmuA2cDNwFlAXzM7wt23RRVUIc4OnDx5Mlu3bqWlpaXD9s5JIt126L4kdVfluIG9SmInl8vuagwiuSx2V9eHuPrqq/na177W5etKz1PoM2olXlEeKaQqvN/52+8fgKlm9mdgKvAWsNc3lpnNM7MGM2vo/EWbrUKcHfjyyy+za9cujjjiiA7buyqTPXHiRFauXMl7771HW1tbh2szRFmS+jOf+QwrVqwAoL6+nilTpgBw4okndtje7tRTT2XZsmXs2LEDgLfeeot33nknb/FIcSr0GbUSryiPFJqBo5PWBwFbkhu4+xbgbAAzOxiY7e5/7bwjd18KLIWgSmouQbWfALJoUXD4O3hw8OHO9cSQDz/8MHFlNHfnrrvuoqKiokObyy67jAsvvJCamhqOP/74RJnsgQMH8q1vfYtJkyZx1FFHMWLEiET57FtuuYXLL7+cmpoa2tra+OxnP5u3stS33HILF198MTfeeCP9+/fnzjvvBODmm2/m/PPP5+abb2b27NmJ9tOnT2fDhg1MnjwZCKah3n333XziE5/ISzxSnKL6PyPFKbLS2Wa2H7ARmEZwBLAKON/d1yW1ORJ41913m9kSYJe7X9Pdfku5dHZ3ZbJ37NjBwQcfTFtbG2eddRYXX3wxZ511VtwhR6pU/t1EeoJMS2dH1n3k7m3AfOBRYANwr7uvM7PrzGxm2Owk4BUz2wh8EujRB6Stra1MmTKFMWPGcNZZZ3Uok11XV8fYsWMZNWoUQ4YM4cwzz4w5WpH8KffzHErq789kilIxLblOSZXioX+38nD33e59+nScztqnT7A9m31UVbmbBbfZPDdu+fj784EimJJaUF5iV5Ard/r3Kh+LFkFra8dtra3B9kyU+slzuf79hdYjkkJlZSXbtm3TF02JcHe2bdtGZWVl3KFIAeR6nkOpfal2VmrnefSI0tmDBg2iubl5r/MCpHhVVlYyaFDkVU2kCOR6nkOpfal2VmrnefSIpNC7d2+GDBkSdxgiksKSJUF3T/Kv/WzOcyi1L9XOcv37C61HdB+JSPHK9cphpX7yXKGvnJaryM5TiEqq8xREpGerr9fJc7nK9DyFHtF9JCI9WyEvR1nu1H0kIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiImmU1PUQcqST10REutFeuru9dlF76W7omSfU6UhBRKQbpV66O1tKCiIi3Sj10t3ZUlIQKQPl1Ceeb12V6C6V0t3ZUlIQ6eFK/XKWcSv10t3ZijQpmNlpZvaKmW02s4UpHh9sZk+Z2Z/N7EUzOyPKeETKUbn1iedbqV0PIVeRXU/BzCqAjcApQDOwCpjj7uuT2iwF/uzut5vZCOBhd6/ubr+6noJIdnr1Co4QOjOD3bsLH4/EI9PrKUR5pDAR2Ozur7n7x8AKYFanNg70C+8fAmyJMB6RslRufeKpaEwlc1EmhYHAm0nrzeG2ZHXA35lZM/Aw8PcRxiNSlsqtT7wzjalkJ8qkYCm2dT6InQMsd/dBwBnAv5rZXjGZ2TwzazCzhpaWlghCFem5yq1PvDONqWQnyjGFyUCdu58arl8N4O7XJ7VZB5zm7m+G668Bn3b3d7rar8YURCQbGlMJFMOYwipgqJkNMbP9gfOAhzq1eQOYBmBmw4FKQIcCIpI3GlPJTmRJwd3bgPnAo8AG4F53X2dm15nZzLDZN4FLzGwNcA8w16M6dBGRslTuYyrZirQgnrs/TDCAnLztmqT764ETo4xBRMpb+9jJokVBaYrBg4OEEMeYSl1dsBSzyMYUoqIxBRHZV3F/KZulHt8ozGvHP6YgIlJUrr027giKn5KCiEiE6uqCIwQLJ+m33y/WbiQlBRHp0eL+Uq6rC7qM2ruN2u8Xa1LQmIKIlI04+/Tjfn2NKYiIFJnFi+OOID0lBREpG3F/KRdrl1EyJQURKRul8KUcNyUFERFJSJsUzOyg9sqlZjbMzGaaWe/oQxMRkULL5EjhGaDSzAYCTwAXAcujDEqkmBTDBVqKIQYpD5nUPjJ3bzWzrwA/cvfvmdmfow5MpBi0X6ClvR5/+wVaoHC1c4ohBikfmRwpWHhthFrg/4fbIi2kJ5Iszl/JxXCBlmKIQcpHJl/u3wCuBh4IS18fAzwVbVgigbh/Jb/xRnbbe2oMUj7SHim4+0p3n+nuN4QDzlvd/YoCxCYS+6/kYrhASzHEIIFymNKayeyjn5tZPzM7CFgPvGJm/xh9aCLx/0ouhgu0FEMMEiiHKquZjCmMcPftwJkEF8wZDHw50qhEQnH/Si6Gi94XQwxSPjJJCr3D8xLOBH7l7juB0qqiJyWrGH4l19ZCY2NwkffGxni+jIshhnIVd5XVQsskKfxfoBE4CHjGzKqA7VEGJdJOv5IlbqVW+jpX+1Q628z2c/e2COJJS6WzRSQucZfezkXeSmeb2SFm9gMzawiX7xMcNWQSxGlm9oqZbTazhSkev8nMVofLRjN7P5P9iojEIe4qq4WQSffRMuAD4Jxw2Q7cme5JZlYB3AacDowA5pjZiOQ27n6lu49197HAj4B/yy58EZHC6aldRskyOXntWHefnbR+rZmtzuB5E4HN7v4agJmtAGYRTGtNZQ5QBnlYRKR4ZXKk8KGZTWlfMbMTgQ8zeN5A4M2k9eZw217CweshwJNdPD6vvfuqpaUlg5cWEZF9kcmRwqXAXWZ2CGDAu8DcDJ5nKbZ1NURzHvBLd9+V6kF3XwoshWCgOYPXFhGRfZA2Kbj7amCMmfUL1zOdjtoMHJ20PgjY0kXb84DLM9yviIhEpMukYGb/q4vtALj7D9LsexUw1MyGAG8RfPGfn2J/nwIOA57PLGQREYlKd0cKfXPZsbu3mdl84FGgAlgWVlm9Dmhw94fCpnOAFb4vJ0yIiEhe7dPJa3HSyWsiItnL28lrIiJSPpQURCKm6ytLKdFlNUUiFPeV40SylXZMwcwOAGYD1SQlEXe/LtLIuqAxBSkl1dVBIuisqioogS1SKPkcU/gVQXmKNuA/kxaRjJRz90ncV44TyVYmSWGQu5/r7t9z9++3L5FHJj1Ce/dJU1NQcri9+6SQiSHOpBT3leNEspVJUvidmY2OPBLpkRYt2tOf3q61NdheCHEnpWK4cpxINjIZU1gP/A3wOvBfBDWN3N1rog9vbxpTKC29eqW+KIlZcGnJqBVDn359fZAE33gjOEJYskSDzFJ4mY4pZDL76PQ8xCNlavDg1F/Kheo+KYY+/dpaJYF8qasrj2saxClt95G7NwGHAjPC5dBwm0hacXefqE+/Z7n22rgj6PkyuRznAqAe+ES43G1mfx91YNIz1NbC0qVBd41ZcLt0aeF+OcedlERKTSYDzV8BJrn7Ne5+DfBp4JJow5KepLY26L/fvTu4LWRXStxJSXJXVxf824UFmhP31Y0UjUwGmtcCE9z9o3C9Eljl7rHMSNJAs0j5Mks9cUHSy+dA853AH8zsgXD9TOCOXIITEZHilMmV135gZk8DUwimo17k7n+OOjARkc4WL447gp6vuyuv9XP37WZ2ONAYLu2PHe7u70YfnojIHhpHiF53A80/D29fABqSlvZ1KRPlXLtIpNx0mRTc/Qvh7RB3PyZpGeLuxxQuRIlT3GUipLjk+ktdv/SLXyazj55w92npthWKZh8VVjGUiZDikevsH80eik/OpbPNrDIcTzjSzA4zs8PDpRo4KsMgTjOzV8xss5kt7KLNOWa23szWmdnPU7WR+BRDmQgRKZzuxhS+RjB+cFx42778Crgt3Y7NrCJsdzowAphjZiM6tRkKXA2c6O4jgW/sw98gEVKZCMn15DGdfFZaMuk++nt3/1HWOzabDNS5+6nh+tUA7n59UpvvARvd/aeZ7lfdR4XV+XKSEJSJ0FnB5UndR6UrbyevufuPzGwUwa/9yqTtP0vz1IHAm0nrzcCkTm2GhcE+B1QQJJFH0sUkhdP+xa/SzyLlIW1SMLPFwEkESeFhgu6gZ4F0ScFSbOv8G2E/YGi4/0HAb81slLu/3ymGecA8gMHqtyg4lX6WdrmePKaTz4pfJgXxvghMA/7D3S8CxgAHZPC8ZuDopPVBwJYUbX7l7jvd/XXgFYIk0YG7L3X38e4+vn///hm8tIhEQVNSe75MksKH7r4baDOzfsA7QCbnKawChprZEDPbHzgPeKhTmweBkwHM7EiC7qTXMg1eRETyK5OCeA1mdijwE4LZRzuAP6Z7kru3mdl84FGC8YJl7r7OzK4DGtz9ofCx6eElP3cB/+ju2/bxbxERkRylnX3UoXFwjkI/d38xqoDS0ewjEZHs5Tz7yMzGdfeYu/9pX4MTEZHi1F330ffD20pgPLCGYEZRDfAHglLaIiLSg3RXEO9kdz8ZaALGhbN/TgCOBzYXKkARESmcTGYfHefua9tX3P0lYGx0IYmISFwymX20wcx+CtxNcPLZ3wEbIo1KRERikUlSuAi4FFgQrj8D3B5ZRCIiEptMah99BNwULiIi0oN1NyX1Xnc/x8zWsnfNIty9JtLIRESk4Lo7UmjvLvpCIQIREZH4dZkU3P3t8DbFxRhFRKQn6u5ynB+Y2fYUywdmtr2QQYpIfqhKqaTT3clrfd29X4qlr7v3K2SQIpIf114bdwRS7DKZkgqAmX2Cjlde06XbRUR6mLRnNJvZTDPbBLwOrAQagV9HHJeI5EldXXBtZAuvhdh+X11JkkomZS6+C3wa2OjuQwiuwvZcpFGJSEr78kVeVwfuwQJ77ispSCqZJIWd4YVveplZL3d/CtU+EomFxgQkapmMKbxvZgcTlLeoN7N3gLZowxKRKCxeHHcEUuwyOVKYBXwIXAk8ArwKzIgyKBHZI59jAuoyknS6vBynmd0K/Nzdf1fYkLqny3FKOTPbMzYgko1ML8fZ3ZHCJuD7ZtZoZjeYmcYRRER6uO5OXrvZ3ScDU4F3gTvNbIOZXWNmwzLZuZmdZmavmNlmM1uY4vG5ZtZiZqvD5av7/JeIlAGNCUjU0o4puHuTu9/g7scD5wNnkcFFdsysArgNOB0YAcwxsxEpmv7C3ceGy0+zC1+kcIqhP74YYpCeLZOT13qb2Qwzqyc4aW0jMDuDfU8ENrv7a+7+MbCCYNBapCRpOqiUg+4K4p1iZsuAZmAe8DBwrLuf6+4PZrDvgcCbSevN4bbOZpvZi2b2SzM7OovYRUQkz7o7UvgW8Dww3N1nuHu9u/9nFvu2FNs6z5v4f0B1eMGex4G7Uu7IbJ6ZNZhZQ0tLSxYhiORG00Gl3HQ5JTXnHZtNBurc/dRw/WoAd7++i/YVwLvufkh3+9WUVIlLrtNBNZ1U4pSPKam5WgUMNbMhZrY/cB7wUHIDMxuQtDqTDAawRUQkOpElBXdvA+YDjxJ82d/r7uvM7Dozmxk2u8LM1pnZGuAKYG5U8Yjkal+mg6pCqZSayLqPoqLuIylV6j6SOBVD95GIiJQYJQWRAtHZyFIKlBRECkTjCFIKlBTKQH09VFdDr17BbX193BGJSLHK5CI7UsLq62HePGhtDdabmoJ1gNra+OISkeKkI4UebtGiPQmhXWtrsF1EpDMlhR7ujTey2y4i5U1JoYcbPDi77SJS3pQUerglS6BPn47b+vQJtouIdKak0MPV1sLSpVBVFZxRW1UVrGuQWURSUVIoA7W10NgIu3cHt+WaEHSegEh6SgpSNnTlNJH0lBRERCRBSUF6NJWuFsmOSmdL2VDpailnKp3dg6h2kYgUimofFTnVLsofla4WSU/dR0WuujpIBJ1VVQXTS0VEMqHuox5CtYtEpJAiTQpmdpqZvWJmm81sYTftvmhmbmZps1gpymVMQLWLRKSQIksKZlYB3AacDowA5pjZiBTt+gJXAH+IKpY4tY8JNDUFM1/axwQyTQyqXSQihRTlkcJEYLO7v+buHwMrgFkp2n0X+B7wUYSxxCbX6xmodtEeOrdAJHpRJoWBwJtJ683htgQzOx442t3/PcI4YpWPMQHVLgqoTIVI9KJMCpZiW2Kqk5n1Am4Cvpl2R2bzzKzBzBpaWlryGGL0NCYgIqUkyqTQDBydtD4I2JK03hcYBTxtZo3Ap4GHUg02u/tSdx/v7uP79+8fYcj5pzGB3KhMhUhhRXaegpntB2wEpgFvAauA8919XRftnwb+wd27PQmhFM9TqK8PxhDeeCM4QliypHy7gHKhMhUi+y7T8xQiO6PZ3dvMbD7wKFABLHP3dWZ2HdDg7g9F9drFprZWSUBESkOkZS7c/WHg4U7brumi7UlRxiKlT2UqRKKnM5qlZGgcQSR6SgoiIpKgpCAiIglKCiIikqCkIAWjMQGR4qekIAWjMhUixU9JQUREEpQUJGP70v2jMhUipUWX45SM5VpmQmUqROKjy3GKiEjWlBSkW/ns/lGZCpHip+4jyZi6f0RKl7qPREQka2WVFDTjJTfq/hHp+cqq+0jdHyJSrtR9JCIiWevxSUEnT4mIZK4skoL7nm6j9vvlmBTK8W8Wkez0+KQge6ggnYikU1ZJQbNnRES6F2lSMLPTzOwVM9tsZgtTPP51M1trZqvN7FkzGxFlPOXYfaIxFRHJRmRTUs2sAtgInAI0A6uAOe6+PqlNP3ffHt6fCVzm7qd1t1+d0bzvNCVXpHwVw5TUicBmd3/N3T8GVgCzkhu0J4TQQYC+skREYhRlUhgIvJm03hxu68DMLjezV4HvAVdEGE/Jy7XLR2MqIpJOlEnBUmzb60jA3W9z92OBq4Bvp9yR2TwzazCzhpaWljyHWTpynT2kcQQRSSfKpNAMHJ20PgjY0k37FcCZqR5w96XuPt7dx/fv3z/rQOrroboaevUKbuvr9+35Zvv2fBGRUhFlUlgFDDWzIWa2P3Ae8FByAzMbmrT6eWBTvoOor4d586CpKRhkbWoK1jP9Yk9+PmT//GS6nKWIFLtIC+KZ2RnAD4EKYJm7LzGz64AGd3/IzG4GPgfsBN4D5rv7uu72me3so+rqPV/oyaqqoLEx+ucn0+UsRSQumc4+6vFVUnv1Sv1Faga7d6d/vqUaGQll+9YpKYhIXIphSmpRGDw4u+2dVVVlt70zXc5SREpJj08KS5ZAnz4dt/XpE2wvxPPzWZBP4wgiErUenxRqa2Hp0uCXvVlwu3RpsD3b50P2zxcRKSU9fkyhmNTV6de+iMRDYwpFSAlBRIqdkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgklNyUVDNrAVJUIyoKRwJb4w6iG4ovN8UeHxR/jIovN7nEV+XuactMl1xSKGZm1pDJPOC4KL7cFHt8UPwxKr7cFCI+dR+JiEiCkoKIiCQoKeTX0rgDSEPx5abY44Pij1Hx5Sby+DSmICIiCTpSEBGTuEbEAAAGKElEQVSRBCWFLJnZ0Wb2lJltMLN1ZrYgRZuTzOyvZrY6XK4pcIyNZrY2fO29Sspa4BYz22xmL5rZuALG9qmk92W1mW03s290alPw98/MlpnZO2b2UtK2w83sN2a2Kbw9rIvnXhi22WRmFxYothvN7OXw3+8BMzu0i+d2+1mIOMY6M3sr6d/xjC6ee5qZvRJ+HhcWML5fJMXWaGaru3hupO9hV98psX3+3F1LFgswABgX3u8LbARGdGpzEvDvMcbYCBzZzeNnAL8GDPg08IeY4qwA/oNg/nSs7x/wWWAc8FLStu8BC8P7C4EbUjzvcOC18Paw8P5hBYhtOrBfeP+GVLFl8lmIOMY64B8y+Ay8ChwD7A+s6fz/Kar4Oj3+feCaON7Drr5T4vr86UghS+7+trv/Kbz/AbABGBhvVFmbBfzMA78HDjWzATHEMQ141d1jPxnR3Z8B3u20eRZwV3j/LuDMFE89FfiNu7/r7u8BvwFOizo2d3/M3dvC1d8Dg/L5mtnq4v3LxERgs7u/5u4fAysI3ve86i4+MzPgHOCefL9uJrr5Tonl86ekkAMzqwaOB/6Q4uHJZrbGzH5tZiMLGhg48JiZvWBm81I8PhB4M2m9mXgS23l0/R8xzvev3Sfd/W0I/uMCn0jRphjey4sJjvxSSfdZiNr8sItrWRfdH8Xw/v1P4C/uvqmLxwv2Hnb6Tonl86eksI/M7GDgfuAb7r6908N/IugSGQP8CHiwwOGd6O7jgNOBy83ss50etxTPKeg0NDPbH5gJ3Jfi4bjfv2zE+l6a2SKgDajvokm6z0KUbgeOBcYCbxN00XQW+2cRmEP3RwkFeQ/TfKd0+bQU23J6/5QU9oGZ9Sb4x6t393/r/Li7b3f3HeH9h4HeZnZkoeJz9y3h7TvAAwSH6MmagaOT1gcBWwoTXcLpwJ/c/S+dH4j7/Uvyl/ZutfD2nRRtYnsvw0HFLwC1HnYwd5bBZyEy7v4Xd9/l7ruBn3Tx2rF+Fs1sP+Bs4BddtSnEe9jFd0osnz8lhSyF/Y93ABvc/QddtPkfYTvMbCLB+7ytQPEdZGZ92+8TDEi+1KnZQ8AF4SykTwN/bT9MLaAuf53F+f518hDQPpvjQuBXKdo8Ckw3s8PC7pHp4bZImdlpwFXATHdv7aJNJp+FKGNMHqc6q4vXXgUMNbMh4dHjeQTve6F8DnjZ3ZtTPViI97Cb75R4Pn9Rjaj31AWYQnB49iKwOlzOAL4OfD1sMx9YRzCT4vfAZwoY3zHh664JY1gUbk+Oz4DbCGZ9rAXGF/g97EPwJX9I0rZY3z+CBPU2sJPg19dXgCOAJ4BN4e3hYdvxwE+TnnsxsDlcLipQbJsJ+pLbP4M/DtseBTzc3WehgO/fv4afrxcJvuAGdI4xXD+DYMbNq1HFmCq+cPvy9s9dUtuCvofdfKfE8vnTGc0iIpKg7iMREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQCZnZLutYwTVvFTvNrDq5QqdIsdov7gBEisiH7j427iBE4qQjBZE0wnr6N5jZH8Plb8LtVWb2RFjw7QkzGxxu/6QF1zhYEy6fCXdVYWY/CWvmP2ZmB4btrzCz9eF+VsT0Z4oASgoiyQ7s1H10btJj2919InAr8MNw260EJchrCArS3RJuvwVY6UFBv3EEZ8ICDAVuc/eRwPvA7HD7QuD4cD9fj+qPE8mEzmgWCZnZDnc/OMX2RuBv3f21sHDZf7j7EWa2laB0w85w+9vufqSZtQCD3P2/kvZRTVD3fmi4fhXQ293/j5k9AuwgqAb7oIfFAEXioCMFkcx4F/e7apPKfyXd38WeMb3PE9SiOgF4IazcKRILJQWRzJybdPt8eP93BFU9AWqBZ8P7TwCXAphZhZn162qnZtYLONrdnwL+N3AosNfRikih6BeJyB4HWseLtz/i7u3TUg8wsz8Q/JCaE267AlhmZv8ItAAXhdsXAEvN7CsERwSXElToTKUCuNvMDiGoXnuTu7+ft79IJEsaUxBJIxxTGO/uW+OORSRq6j4SEZEEHSmIiEiCjhRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQS/hujjmP8AqXPuAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20ad24ae208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bigger_model_val_loss = bigger_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The bigger network starts overfitting almost right away, after just one epoch, and overfits much more severely. Its validation loss is also \n",
    "more noisy.\n",
    "\n",
    "Meanwhile, here are the training losses for our two networks:\n",
    "\n",
    "更大的网络只过了一轮就开始过拟合，过拟合也更严重。其验证损失的波动也更大。\n",
    "\n",
    "下图同时给出了这两个网络的训练损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt4VNW9//H3l3jBKN6Q9qBIghyogATkWlqO6EHjpUUUr8ivilhpFZ5yPP2do5YqqT3UY/XUSrUXWhE9pFqtpx5/fbyL0mJtJVgUUSugBCO2BqpFnkgl8P39sWfGSZhJdjKzZ88kn9fzzDOz1+zLNzuT+Wattfda5u6IiIgA9Ig7ABERKR5KCiIikqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIilKCiIikqKkICIiKfvEHUBHHXHEEV5ZWRl3GCIiJWX16tVb3b1Pe+uVXFKorKykrq4u7jBEREqKmdWHWU/NRyIikqKkICIiKUoKIiKSUnJ9CiJSOLt27aKhoYGdO3fGHYqE1LNnT/r168e+++7bqe2VFEQkq4aGBnr16kVlZSVmFnc40g53Z9u2bTQ0NDBgwIBO7aNbNB/V1kJlJfToETzX1sYdkUhp2LlzJ71791ZCKBFmRu/evXOq2XX5mkJtLcyeDU1NwXJ9fbAMMGNGfHGJlAolhNKS6++ry9cU5s//JCEkNTUF5SIi0lKXTwqbN3esXESKS0NDA1OnTmXQoEEMHDiQefPm8fHHH2dcd8uWLZx77rnt7vOMM87ggw8+6FQ8NTU13HLLLZ3aNqylS5cyd+7cnNfpjC6fFPr371i5iOSupiY/+3F3pk2bxllnncX69et544032LFjB/MzVPWbm5s58sgj+eUvf9nufh955BEOPfTQ/ATZxXT5pLBwIZSXtywrLw/KRSQa3/pWfvazfPlyevbsyaWXXgpAWVkZt956K0uWLKGpqYmlS5dy3nnnMWXKFKqrq9m0aRPHHXccAE1NTZx//vlUVVVxwQUXMH78+NQQOZWVlWzdupVNmzYxZMgQLr/8coYNG0Z1dTUfffQRAD/96U8ZO3YsI0aM4JxzzqGpdTt0KzNnzuSKK67gpJNO4phjjmHFihXMmjWLIUOGMHPmzNR69957L8OHD+e4447j6quvTpXfddddDB48mEmTJvHcc8+lyhsbGznnnHMYO3YsY8eObfFeFLp8UpgxAxYvhooKMAueFy9WJ7NIKVi3bh2jR49uUXbwwQfTv39/NmzYAMDzzz/P3XffzfLly1us98Mf/pDDDjuMl19+meuuu47Vq1dnPMb69euZM2cO69at49BDD+XBBx8EYNq0aaxatYqXXnqJIUOGcOedd7Yb7/vvv8/y5cu59dZbmTJlCldddRXr1q1j7dq1rFmzhi1btnD11VezfPly1qxZw6pVq3jooYd49913WbBgAc899xxPPvkkr776amqf8+bN46qrrmLVqlU8+OCDfPnLX+7QOeyoLn/1EQQJQElAJFo1NS1rCMmLYBYs6HxzkrtnvJomvfyUU07h8MMP32udlStXMm/ePACOO+44qqqqMh5jwIABjBw5EoDRo0ezadMmAF555RW++c1v8sEHH7Bjxw5OPfXUduOdMmUKZsbw4cP59Kc/zfDhwwEYNmwYmzZtor6+nhNPPJE+fYLBSmfMmMFvfvMbgBblF1xwAW+88QYATz31VIsksX37dj788MN2Y+msbpEURCR6NTWffPmbgXvu+xw2bFjqP/ek7du38/bbbzNw4EBWr17NgQcemHFbDxnA/vvvn3pdVlaWaj6aOXMmDz30ECNGjGDp0qU8++yzoffVo0ePFvvt0aMHzc3N7LNP9q/cbJeS7tmzh+eff54DDjggzI+Tsy7ffJQuX51fIlIYkydPpqmpiXvuuQeA3bt38/Wvf52ZM2dS3rqzsJWJEydy//33A/Dqq6+ydu3aDh37ww8/pG/fvuzatYvaPN3xOn78eFasWMHWrVvZvXs39957L5MmTWL8+PE8++yzbNu2jV27dvHAAw+ktqmurub2229PLa9ZsyYvsWTTrZJCvjq/RKRtCxbkZz9mxq9+9SseeOABBg0axODBg+nZsyff+c532t32yiuvpLGxkaqqKm666Saqqqo45JBDQh/729/+NuPHj+eUU07h2GOPzeXHSOnbty833ngjJ510EiNGjGDUqFFMnTqVvn37UlNTw4QJEzj55JMZNWpUaptFixZRV1dHVVUVQ4cO5cc//nFeYsnGwlaxisWYMWO8s5Ps5KtKK9JdvPbaawwZMiTuMDpl9+7d7Nq1i549e7Jx40YmT57MG2+8wX777Rd3aJHL9Hszs9XuPqa9bbt8TaGmJkgGyea65Gs1JYl0bU1NTUycOJERI0Zw9tln86Mf/ahbJIRcdfmO5ig6v0Sk+PXq1UtT93ZCl68piIhIeN0qKeSr80tEpKvqVklB/QgiIm3rVklBRETapqQgIkWtrKyMkSNHpq7r/93vfgeEHya7mB100EF5WSeflBREJG+imPr2gAMOYM2aNbz00kvceOONXHvttQChh8nORXNzc6T7L0ZKCiKSF8mpb+vrg0u/k1Pf5nNO9O3bt3PYYYcBhB4m+84772Tw4MGceOKJXH755amJabINSV1TU8Ps2bOprq7m4osvbnH8Z599lkmTJnH++eczePBgrrnmGmpraxk3bhzDhw9n48aNANTX1zN58mSqqqqYPHkymxOzer311ltMmDCBsWPHct1117XY980338zYsWOpqqpiQZxXxbh7ST1Gjx7tIlIYr776auh1Kyrcg3TQ8lFRkVsMPXr08BEjRvhnPvMZP/jgg72urs7d3d966y0fNmyYu7vffPPNPnv2bHd3X7t2rZeVlfmqVav8nXfe8YqKCt+2bZt//PHHPnHiRJ8zZ467u0+fPt1/+9vfurt7fX29H3vsse7uvmDBAh81apQ3NTXtFcszzzzjhxxyiG/ZssV37tzpRx55pF9//fXu7v7973/f582b5+7uX/ziF33p0qXu7n7nnXf61KlT3d19ypQpfvfdd7u7++233+4HHnigu7s//vjjfvnll/uePXt89+7d/oUvfMFXrFjh7p5apyMy/d6AOg/xHauagojkRVRT3yabj15//XUee+wxLr744r1GQF25ciUXXngh0HKY7BdeeIFJkyZx+OGHs++++3LeeeeltnnqqaeYO3cuI0eO5Mwzz2wxJPWZZ56ZdVTSsWPH0rdvX/bff38GDhxIdXU1AMOHD08Nu/38889z0UUXAfClL32JlStXAvDcc88xffr0VHnSE088wRNPPMHxxx/PqFGjeP3111m/fn1O562zuvwdzSJSGP37B01GmcrzZcKECWzdupXGxsYW5a2TRHvl0PaQ1NmG4wb2GhI7fbjsbH0Q6cNiZ5sf4tprr+UrX/lK1uMWimoKIpIXhZj69vXXX2f37t307t27RXm2YbLHjRvHihUreP/992lubm4xN0OUQ1J/7nOf47777gOgtraWiRMnAvD5z3++RXnSqaeeypIlS9ixYwcA77zzDu+9917e4ukI1RREJC+SsxvOnx80GfXvHySEXGc9/Oijj1Izo7k7d999N2VlZS3WufLKK7nkkkuoqqri+OOPTw2TfdRRR/GNb3yD8ePHc+SRRzJ06NDU8NmLFi1izpw5VFVV0dzczAknnJC3YakXLVrErFmzuPnmm+nTpw933XUXALfddhsXXXQRt912G+ecc05q/erqal577TUmTJgABJehLlu2jE996lN5iacjutXQ2SLSMaUydHZbw2Tv2LGDgw46iObmZs4++2xmzZrF2WefHXfIkcpl6OxIawpmdhpwG1AG/Mzd/zPLeucCDwBj3V3f+CLSIU1NTZx00kns2rULd28xTHZNTQ1PPfUUO3fupLq6mrPOOivmaItbZEnBzMqAO4BTgAZglZk97O6vtlqvF/A14A9RxSIiXVtbw2TfcsstBY6mtEXZ0TwO2ODub7r7x8B9wNQM630b+C6wM8JYRKSTSq2JubvL9fcVZVI4Cng7bbkhUZZiZscDR7v7ryOMQ0Q6qWfPnmzbtk2JoUS4O9u2baNnz56d3keUfQp7X4wLqU+WmfUAbgVmtrsjs9nAbID++bzoWUTa1K9fPxoaGva6L0CKV8+ePenXr1+nt48yKTQAR6ct9wO2pC33Ao4Dnk3czPEPwMNmdmbrzmZ3XwwshuDqowhjFpE0++67LwMGDIg7DCmgKJuPVgGDzGyAme0HXAg8nHzT3f/m7ke4e6W7VwK/B/ZKCCIiUjiRJQV3bwbmAo8DrwH3u/s6M7vBzM6M6rgiItJ5kd6n4O6PAI+0Krs+y7onRhmLiIi0T2MfiYhIipKCiIikKCmIiEiKkoKIiKQoKYiISIqSgoiIpCgpiIhIipKCiIikKCmIiEiKkoKIiKQoKYiISIqSgoiIpCgpiIhIipKCiIikKCmIiEiKkoKIiKQoKYiISIqSgoiIpCgpiIhIipKCiIikKCmIiEiKkoKIiKQoKYiISIqSgoiIpCgpiIhIipKCiIikKCmIiEiKkoKIiKQoKYiISIqSgoiIpCgpiIhIipKCiIikKCmIiEhKpEnBzE4zsz+Z2QYzuybD+181s7VmtsbMVprZ0CjjERGRtkWWFMysDLgDOB0YCkzP8KX/c3cf7u4jge8C34sqHhERaV+HkoIFDgy5+jhgg7u/6e4fA/cBU9NXcPftaYsHAt6ReEREJL/aTQpmdo+ZHWxm5cA64C0z+9cQ+z4KeDttuSFR1nr/c8xsI0FN4WvhwhYRkSiEqSkMT/xHfxbwBNAPmBliO8tQtldNwN3vcPeBwNXANzPuyGy2mdWZWV1jY2OIQ4uISGeESQr7mdk+BE0/DyWagvaE2K4BODptuR+wpY317yNIPHtx98XuPsbdx/Tp0yfEoUVEpDPCJIWfAZuBw4AVZtYf2BFiu1XAIDMbYGb7ARcCD6evYGaD0ha/AKwPFbWIiERin/ZWcPdbgVuTy2b2NvDPIbZrNrO5wONAGbDE3deZ2Q1Anbs/DMw1s5OBXcD7wCWd+zFERCQf2k0KiS/2e9x9u5n9BDgeuBZ4ur1t3f0R4JFWZdenvZ7X4YhFRCQyYZqPZicSQjXB1UNXEFwpJCIiXUyYpJC8Yuh04C53Xx1yOxERKTFhvtxfMrNHgCnAo2Z2ELrJTESkS2q3TwG4FBhNcHdyk5kdAVwWbVgiIhKHMFcf7U4kgmlmBrDC3R+NPDIRESm4MMNcLAT+HXgz8fg3M/uPqAMTEZHCC9N8NAUY5e7NAGa2BHiRLENSiIhI6Qp7FVGvLK9FRKQLCVNT+C7wopk9TTDI3YnA9W1uISIiJSlMR/MyM3sGGE+QFK5393cij0xERAoua/ORmVUlH0BvYAPBgHW9E2XdTk1N3BGIiETL3DPfh2Zmv21jO3f3E6IJqW1jxozxurq6OA6NGWQ5XSIiRc3MVrv7mPbWy9p85O7/lN+QRESk2GkMo3bU1AQ1BEvMI5d8raYkEemKsjYfFSs1H4mIdFzY5iPVFEREJCXMJDuZrjT6G/C2u4eZq7nLWLAg7ghERKIV5ua1O4GRwDqC+xSGAK8Ah5jZbHdvdwa2rkL9CCLS1YVpPloPjHb3ke4+gmAY7TXAqcB/RRmciIgUVpikMMTdX04uuPtaggHyNkQXloiIxCFM89FGM/sBcF9i+QJgg5ntDzRHFpmIiBRcmJrCxUADcA1wLbAFuIQgIUyOLjQRESm0MAPiNQE3JR6t/S3vEYmISGzCXJL6WWABUJG+vrsPjjAuERGJQZg+hbsIpuNcDeyONhwREYlTmD6F7e7+/9x9i7v/JfmIPLIiUlsLlZXQo0fwXFsbd0QiItEIU1NYbmY3Av8D/D1ZmH6ZaldWWwuzZ0NTU7BcXx8sA8yYEV9cIiJRaHdAvCzzKnSb+RQqK4NE0FpFBWzaVLAwRERykvN8CkndfV6FzZs7Vi4iUsqyJgUzm+7u95rZ1zK97+6LogurePTvn7mm0L9/4WMREYlaWx3NhyWe+2R5dAsLF0J5ecuy8vKgXESkq2lrOs4fJp6vK1w4xSfZmTx/ftBk1L9/kBDUySwiXVGYm9eOAGYBlbS8eW12dGEVlxkzlAREpHsIc0nq/wK/B1aim9dERLq0MEnhQHf/emd2bmanAbcBZcDP3P0/W73/r8CXCQbXawRmuXuGbl0RESmEMHc0P2pm1R3dsZmVAXcApwNDgelmNrTVan8Exrh7FfBL4LsdPU4p0cxtIlLswiSFrwKPmdkOM/urmb1vZn8Nsd04YIO7v+nuHxPMxzA1fQV3fyYxCisETVT9OhJ8qfnWt+KOQESkbWGaj47o5L6PAt5OW24Axrex/mXAo508loiI5EHWmoKZDUq8HJbl0R7LUJZxTA0z+z/AGODmLO/PNrM6M6trbGwMcejiUVMDZsEDPnmtpiQRKUZZxz4yszvd/bLOjn1kZhOAGnc/NbF8bWLDG1utdzLwA2CSu7/XXsCFHvson8ygnaGmREQikfPYR+5+WeK5s2MfrQIGmdkA4B3gQuCiVkEeD/wEOC1MQhARkWiF6VPAzI4luIKoZ7LM3X/e1jbu3mxmc4HHCS5JXeLu68zsBqDO3R8maC46CHjAgvaVze5+Zqd+khKwYEHcEYiItC3M0NnfBKqBYwm+4E8FVrr7tOjD21spNx+JiMQlbPNRmEtSLwBOAt519y8BIwhZwxARkdISJil85O67gWYz6wX8GTgm2rBERCQOYf7j/6OZHQosAeqA7cCLkUYlIiKxaDMpWND7W+PuHwB3mNnjwMHurqQgItIFtdl85EEv9K/TljcoIYiIdF1h+hReMLNRkUciIiKxa2uYi2TT0kSCxPAnM3vRzP5oZqotxEBDY4hI1Noa5uJFdx9lZgMzve/uGyONLIvufJ+ChskQkc7KeZgLEgPaxfXlLyIihddWUuiTmBktI3f/XgTxSCs1NS3nYUiOtrpggZqTRCT/2koKZQTjEmUaAlsKpKbmky9/NR+JSNTaSgrvuvsNBYtERERi19YlqaohFBmNsioiUWsrKUwuWBQSivoQRCRqWZOCu/+1kIGIiEj8wtzRLCIi3YSSQjei5icRaY+SQjeSfr+DiEgmSgoiIpKipNDF1dQEN70l74ROvlZTkohkoqRQALW1UFkJPXoEz7W1hTt2TU1wF3TyTujkayUFEckkzHSckoPaWpg9G5qaguX6+mAZYMaM+OISEclENYWIzZ//SUJIamoKygtNd0SLSHuUFCK2eXPHyqOUa5ORmpxEuj4lhYj179+x8mKmS1pFuj4lhYgtXAjl5S3LysuDchGRYqOkELEZM2DxYqioCC4FragIlkulk1mXtIp0L1nnaC5W3XmO5rhpkh+R0hV2jmbVFEREJEVJQULTJa0iXZ+SgoSmS1pFuj4lBSkYXdIqUvyUFEREJEVJQSKlS1pFSkukScHMTjOzP5nZBjO7JsP7J5jZi2bWbGbnRhmLxEOjtIqUlsiSgpmVAXcApwNDgelmNrTVapuBmcDPo4pDug4lEpHoRVlTGAdscPc33f1j4D5gavoK7r7J3V8G9kQYhxSJXC9pVUe1SPSiTApHAW+nLTckyjrMzGabWZ2Z1TU2NuYlOCk8/acvUvyiTAqWoaxTgyS4+2J3H+PuY/r06ZNjWFJoucw8p45qkcKKMik0AEenLfcDtkR4PClCyZnn6uuDDubkzHNhE0M+OqrjnA5VpNREmRRWAYPMbICZ7QdcCDwc4fGkCMU981yuSUmku4ksKbh7MzAXeBx4Dbjf3deZ2Q1mdiaAmY01swbgPOAnZrYuqngkHvmcea4zHdVxJyWRUhPpfQru/oi7D3b3ge6+MFF2vbs/nHi9yt37ufuB7t7b3YdFGY8UXj5nnutMP0IxTYcq8VETYni6o1kiFffMc8UwHaq+kOI9B2pC7CB3L6nH6NGjXTpm2TL3igp3s+B52bLuc/xp05Jd0y0f06YV5vjLlrmXl7c8dnl54X8HcYr7HFRUZP4MVFQU5vjFAqjzEN+xsX/Jd/ShpNAxcf9BFoNkUkp+EXT2Z1+woOPb6Asp/nNglvn4ZoU5frEImxQ0HWcJqK0NOkY3bw6aPRYuDD/Hc2VlUF1uraICNm3KZ5TFL9fpRDuzfY8embcxgz3d5D7+uM+B/gYCmo6zi8i1PVQdrZ+IY+a4YujTiFvc5yDufq1So6RQ5HK9pDLuP8hi0pmrl3K9o1pfSPGfgxkzYPHioGZgFjwvXhy+tt3thGljKqZHd+tTyLU9VH0K+QOd2y5ffRqlLO6LHUR9Cl1GPtpDc+mTkE/E0Schki/qU+gi8lH1njEjSCB79gTPSgid05k+CQ3oJ6VGNYUSoP/0uwbVFCROqil0IfpPX5JUw5CoKSmIFEg+LonV7HMSNSUFkQIphv/yiyEGKW5KCiJFLp+d1appSHvU0SxSQnRZrHSWOppFBNBlsdIxSgoiJaSz90ok72eHzs1znb4v6drUfCTSjaj5qftS85GI7CWOkWLTqaZR/JQURLqROEaKTaern4qfmo9EJDQ1P5UuNR+JSFHIZ01DzU/RU01BREKrqcnti1k1jfiopiAieVfq/6mXevyFoKQgIgUT95wU6uhun5KCiBRMZ/sR8nXzXa5yPWYp1FSUFESkSyummkYp1FT2iTsAEZGwOtv8lEwA6qhun2oKIlIy4moyyqWmUWqX5OqSVBHpNkr9ktpcttclqSIirZRCR2/clBRERELKdUDBuC/JDUPNRyIiJaLkm4/M7DQz+5OZbTCzazK8v7+Z/SLx/h/MrDLKeKRzamuhshJ69Aiea2vjjkhEohJZUjCzMuAO4HRgKDDdzIa2Wu0y4H13/0fgVuCmqOKRzqmthdmzob4++A+lvj5YLmRiiDspxX38fMQQ9/a5KvXj52t7KMDP7+6RPIAJwONpy9cC17Za53FgQuL1PsBWEk1a2R6jR492KZyKiuT9oy0fFRWFOf6yZe7l5S2PXV4elHeH4+cjhri3z1WpHz/u7ZOAOg/z3R1mpc48gHOBn6Utfwm4vdU6rwD90pY3Ake0tV8lhcIyy5wUzApz/LiTUtzHz0cMcW+fq1I/ftzbJ4VNClH2KViGstZdJGHWwcxmm1mdmdU1NjbmJTgJp3//jpXn2+bNHSvvasfPRwxxb5+rUj9+3Nt3VJRJoQE4Om25H7Al2zpmtg9wCPDX1jty98XuPsbdx/Tp0yeicCWThQuhvLxlWXl5UF4IcSeluI+fjxji3j5XpX78uLfvsDDVic48CPoI3gQGAPsBLwHDWq0zB/hx4vWFwP3t7VfNR4W3bFlQVTULnkupPb3Uj5+PGOLePlelfvy4t08i7j6FIAbOAN4g6CuYnyi7ATgz8bon8ACwAXgBOKa9fSopdD9xJqViOH4+Yoh7+1yV+vHj3t49fFLQzWsiIt1AUdy8JiIipUVJQUREUpQUREQkRUlBRERSlBRERCSl5K4+MrNGoD7uOLI4gmD8pmKl+HJT7PFB8ceo+HKTS3wV7t7u3b8llxSKmZnVhbnkKy6KLzfFHh8Uf4yKLzeFiE/NRyIikqKkICIiKUoK+bU47gDaofhyU+zxQfHHqPhyE3l86lMQEZEU1RRERCRFSaGDzOxoM3vGzF4zs3VmNi/DOiea2d/MbE3icX2BY9xkZmsTx95r9EALLDKzDWb2spmNKmBsn0k7L2vMbLuZ/UurdQp+/sxsiZm9Z2avpJUdbmZPmtn6xPNhWba9JLHOejO7pECx3Wxmryd+f78ys0OzbNvmZyHiGGvM7J203+MZWbY9zcz+lPg8XlPA+H6RFtsmM1uTZdtIz2G275TYPn9hhlLVo8Vw4H2BUYnXvQiGBh/aap0TgV/HGOMm2pjWlGBI80cJZr77LPCHmOIsA/5McP10rOcPOAEYBbySVvZd4JrE62uAmzJsdzjBvCGHA4clXh9WgNiqgX0Sr2/KFFuYz0LEMdYA/zfEZ2AjcAyfzLsytBDxtXr/v4Dr4ziH2b5T4vr8qabQQe7+rru/mHj9IfAacFS8UXXYVOAeD/weONTM+sYQx2Rgo7vHfjOiu/+GvWf9mwrcnXh9N3BWhk1PBZ5097+6+/vAk8BpUcfm7k+4e3Ni8fcEMxvGJsv5C2McsMHd33T3j4H7CM57XrUVn5kZcD5wb76PG0Yb3ymxfP6UFHJgZpXA8cAfMrw9wcxeMrNHzWxYQQML5rl+wsxWm9nsDO8fBbydttxAPIntQrL/IcZ5/pI+7e7vQvCHC3wqwzrFcC5nEdT8MmnvsxC1uYkmriVZmj+K4fz9E/AXd1+f5f2CncNW3ymxfP6UFDrJzA4CHgT+xd23t3r7RYImkRHAD4CHChze5919FHA6MMfMTmj1vmXYpqCXoZnZfsCZBDPvtRb3+euIWM+lmc0HmoHaLKu091mI0o+AgcBI4F2CJprWYv8sAtNpu5ZQkHPYzndK1s0ylOV0/pQUOsHM9iX45dW6+/+0ft/dt7v7jsTrR4B9zeyIQsXn7lsSz+8BvyKooqdrAI5OW+4HbClMdCmnAy+6+19avxH3+Uvzl2SzWuL5vQzrxHYuE52KXwRmeKKBubUQn4XIuPtf3H23u+8Bfprl2LF+Fs1sH2Aa8Its6xTiHGb5Tonl86ek0EGJ9sc7gdfc/XtZ1vmHxHqY2TiC87ytQPEdaGa9kq8JOiRfabXaw8DFiauQPgv8LVlNLaCs/53Fef5aeRhIXs1xCfC/GdZ5HKg2s8MSzSPVibJImdlpwNUE8503ZVknzGchyhjT+6nOznLsVcAgMxuQqD1eSHDeC+Vk4HV3b8j0ZiHOYRvfKfF8/qLqUe+qD2AiQfXsZWBN4nEG8FXgq4l15gLrCK6k+D3wuQLGd0ziuC8lYpifKE+Pz4A7CK76WAuMKfA5LCf4kj8krSzW80eQoN4FdhH893UZ0Bt4GlifeD48se4Y4Gdp284CNiQelxYotg0EbcnJz+CPE+seCTzS1mehgOfvvxOfr5cJvuD6to4xsXwGwRU3G6OKMVN8ifKlyc9d2roFPYdtfKfE8vkhhV5vAAAB+klEQVTTHc0iIpKi5iMREUlRUhARkRQlBRERSVFSEBGRFCUFERFJUVIQSTCz3dZyBNe8jdhpZpXpI3SKFKt94g5ApIh85O4j4w5CJE6qKYi0IzGe/k1m9kLi8Y+J8gozezox4NvTZtY/Uf5pC+Y4eCnx+FxiV2Vm9tPEmPlPmNkBifW/ZmavJvZzX0w/pgigpCCS7oBWzUcXpL233d3HAbcD30+U3U4wBHkVwYB0ixLli4AVHgzoN4rgTliAQcAd7j4M+AA4J1F+DXB8Yj9fjeqHEwlDdzSLJJjZDnc/KEP5JuCf3f3NxMBlf3b33ma2lWDohl2J8nfd/QgzawT6ufvf0/ZRSTDu/aDE8tXAvu7+H2b2GLCDYDTYhzwxGKBIHFRTEAnHs7zOtk4mf097vZtP+vS+QDAW1WhgdWLkTpFYKCmIhHNB2vPzide/IxjVE2AGsDLx+mngCgAzKzOzg7Pt1Mx6AEe7+zPAvwOHAnvVVkQKRf+RiHziAGs5eftj7p68LHV/M/sDwT9S0xNlXwOWmNm/AY3ApYnyecBiM7uMoEZwBcEInZmUAcvM7BCC0WtvdfcP8vYTiXSQ+hRE2pHoUxjj7lvjjkUkamo+EhGRFNUUREQkRTUFERFJUVIQEZEUJQUREUlRUhARkRQlBRERSVFSEBGRlP8P/q8jSDIZYEkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20ad1f6fe10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "original_train_loss = original_hist.history['loss']\n",
    "bigger_model_train_loss = bigger_model_hist.history['loss']\n",
    "\n",
    "plt.plot(epochs, original_train_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Training loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the bigger network gets its training loss near zero very quickly. The more capacity the network has, the quicker it will be \n",
    "able to model the training data (resulting in a low training loss), but the more susceptible it is to overfitting (resulting in a large \n",
    "difference between the training and validation loss).\n",
    "\n",
    "如你所见，更大网络的训练损失很快就接近于零。 网络的容量越大，它拟合训练数据（即得到很小的训练损失）的速度就越快，但也更容易过拟合（导致训练损失和验证损失有很大差异）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding weight regularization\n",
    "\n",
    "\n",
    "You may be familiar with _Occam's Razor_ principle: given two explanations for something, the explanation most likely to be correct is the \n",
    "\"simplest\" one, the one that makes the least amount of assumptions. This also applies to the models learned by neural networks: given some \n",
    "training data and a network architecture, there are multiple sets of weights values (multiple _models_) that could explain the data, and \n",
    "simpler models are less likely to overfit than complex ones.\n",
    "\n",
    "A \"simple model\" in this context is a model where the distribution of parameter values has less entropy (or a model with fewer \n",
    "parameters altogether, as we saw in the section above). Thus a common way to mitigate overfitting is to put constraints on the complexity \n",
    "of a network by forcing its weights to only take small values, which makes the distribution of weight values more \"regular\". This is called \n",
    "\"weight regularization\", and it is done by adding to the loss function of the network a _cost_ associated with having large weights. This \n",
    "cost comes in two flavors:\n",
    "\n",
    "* L1 regularization, where the cost added is proportional to the _absolute value of the weights coefficients_ (i.e. to what is called the \n",
    "\"L1 norm\" of the weights).\n",
    "* L2 regularization, where the cost added is proportional to the _square of the value of the weights coefficients_ (i.e. to what is called \n",
    "the \"L2 norm\" of the weights). L2 regularization is also called _weight decay_ in the context of neural networks. Don't let the different \n",
    "name confuse you: weight decay is mathematically the exact same as L2 regularization.\n",
    "\n",
    "In Keras, weight regularization is added by passing _weight regularizer instances_ to layers as keyword arguments. Let's add L2 weight \n",
    "regularization to our movie review classification network:\n",
    "\n",
    "## 添加权重正则化\n",
    "\n",
    "你可能知道奥卡姆剃刀（Occam’s razor）原理：如果一件事情有两种解释，那么最可能正 确的解释就是最简单的那个，即假设更少的那个。这个原理也适用于神经网络学到的模型：给 定一些训练数据和一种网络架构，很多组权重值（即很多模型）都可以解释这些数据。简单模 型比复杂模型更不容易过拟合。\n",
    "\n",
    "这里的简单模型（simple model）是指参数值分布的熵更小的模型（或参数更少的模型，比如上一节的例子）。因此，一种常见的降低过拟合的方法就是强制让模型权重只能取较小的值， 从而限制模型的复杂度，这使得权重值的分布更加规则（regular）。这种方法叫作权重正则化（weight regularization），其实现方法是向网络损失函数中添加与较大权重值相关的成本（cost）。 这个成本有两种形式。\n",
    "* L1 正则化（L1 regularization）：添加的成本与权重系数的绝对值［权重的 L1 范数（norm）］ 成正比。\n",
    "* L2 正则化（L2 regularization）：添加的成本与权重系数的平方（权重的 L2 范数）成正比。 神经网络的 L2 正则化也叫权重衰减（weight decay）。不要被不同的名称搞混，权重衰减 与 L2 正则化在数学上是完全相同的。\n",
    "\n",
    "在 Keras 中，添加权重正则化的方法是向层传递 权重正则化项实例（weight regularizer instance）作为关键字参数。下列代码将向电影评论分类网络中添加 L2 权重正则化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "l2_model = models.Sequential()\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu', input_shape=(10000,)))\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu'))\n",
    "l2_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "l2_model.compile(optimizer='rmsprop',\n",
    "                 loss='binary_crossentropy',\n",
    "                 metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`l2(0.001)` means that every coefficient in the weight matrix of the layer will add `0.001 * weight_coefficient_value` to the total loss of \n",
    "the network. Note that because this penalty is _only added at training time_, the loss for this network will be much higher at training \n",
    "than at test time.\n",
    "\n",
    "Here's the impact of our L2 regularization penalty:\n",
    "\n",
    "`l2(0.001)` 的意思是该层权重矩阵的每个系数都会使网络总损失增加 `0.001 * weight_coefficient_value`。注意，由于这个惩罚项只在训练时添加，所以这个网络的训练损失会 比测试损失大很多。\n",
    "\n",
    "下图显示了 L2 正则化惩罚的影响。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 4s 162us/step - loss: 0.4879 - acc: 0.8152 - val_loss: 0.3895 - val_acc: 0.8656\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.3100 - acc: 0.9061 - val_loss: 0.3306 - val_acc: 0.8891\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.2657 - acc: 0.9202 - val_loss: 0.3301 - val_acc: 0.8872\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.2459 - acc: 0.9286 - val_loss: 0.3413 - val_acc: 0.8821\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.2323 - acc: 0.9352 - val_loss: 0.3818 - val_acc: 0.8685\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.2237 - acc: 0.9390 - val_loss: 0.3676 - val_acc: 0.8752\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.2173 - acc: 0.9409 - val_loss: 0.3748 - val_acc: 0.8735\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.2106 - acc: 0.9445 - val_loss: 0.3726 - val_acc: 0.8752\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.2045 - acc: 0.9466 - val_loss: 0.3762 - val_acc: 0.8753\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 4s 156us/step - loss: 0.2009 - acc: 0.9500 - val_loss: 0.3822 - val_acc: 0.8748\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 4s 158us/step - loss: 0.1950 - acc: 0.9495 - val_loss: 0.3838 - val_acc: 0.8755\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 4s 156us/step - loss: 0.1922 - acc: 0.9518 - val_loss: 0.3917 - val_acc: 0.8745\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 4s 162us/step - loss: 0.1876 - acc: 0.9547 - val_loss: 0.3973 - val_acc: 0.8734\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.1819 - acc: 0.9570 - val_loss: 0.4332 - val_acc: 0.8640\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 4s 170us/step - loss: 0.1815 - acc: 0.9577 - val_loss: 0.4573 - val_acc: 0.8565\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 4s 157us/step - loss: 0.1756 - acc: 0.9600 - val_loss: 0.4465 - val_acc: 0.8649\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.1717 - acc: 0.9612 - val_loss: 0.4231 - val_acc: 0.8696\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 4s 157us/step - loss: 0.1686 - acc: 0.9624 - val_loss: 0.4345 - val_acc: 0.8680\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.1654 - acc: 0.9630 - val_loss: 0.4447 - val_acc: 0.8650\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.1612 - acc: 0.9656 - val_loss: 0.4292 - val_acc: 0.8688\n"
     ]
    }
   ],
   "source": [
    "l2_model_hist = l2_model.fit(x_train, y_train,\n",
    "                             epochs=20,\n",
    "                             batch_size=512,\n",
    "                             validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XucFPWZ7/HPw4gZUbxFYlAugxw03AaQATQXEG+oUfCuLGdPwETiBUXNMRoxMprjGj2J7nqJiReCrkRUjEoSNjGKwtElWQblqstFF3QWVgcMIiIr4HP+qJqmGXqma6a7urpnvu/Xq17dVf3rqmeKop6uX1U9Ze6OiIgIQLukAxARkeKhpCAiIilKCiIikqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIimxJQUzm2ZmH5rZ8kY+NzO718zWmNlSMzs2rlhERCSafWKc93TgfuDxRj4/HegVDsOAB8PXJh122GFeUVGRnwhFRNqIRYsWbXT3TtnaxZYU3H2+mVU00WQM8LgHdTb+YmYHm1lnd9/Q1HwrKiqoqanJY6QiIq2fma2L0i7JcwpHAu+njdeG0/ZiZhPNrMbMaurq6goSnIhIW5RkUrAM0zJW53P3h9y9yt2rOnXKevQjIiItlGRSqAW6po13AdYnFIuIiBDvieZsZgOTzGwmwQnmj7OdT2jMjh07qK2tZfv27XkNUFqv8vJyunTpQvv27ZMORaSoxJYUzOxJ4ATgMDOrBaYC7QHc/ZfAHOAMYA2wDZjQ0mXV1tbSsWNHKioqMMvUKyWym7uzadMmamtr6dGjR9LhiBSVOK8+GpvlcweuzMeytm/froQgkZkZX/7yl9FFC1JqqquDIU6t5o5mJQRpDm0vUopuvTX+ZbSapCAiIrlTUsiT2tpaxowZQ69evejZsyeTJ0/m888/z9h2/fr1nH/++VnnecYZZ7B58+YWxVNdXc3PfvazFn03qunTpzNp0qSc24hI46qrwSwYYPf7uLqR2nRSyNdKdXfOPfdczj77bFavXs2qVavYunUrU6ZM2avtzp07OeKII5g1a1bW+c6ZM4eDDz44P0GKSEmqrgb3YIDd75UUYpCv/rm5c+dSXl7OhAnBBVRlZWXcc889TJs2jW3btjF9+nQuuOACzjrrLE499VTWrl1Lv379ANi2bRsXXnghlZWVXHTRRQwbNixVxqOiooKNGzeydu1aevfuzaWXXkrfvn059dRT+eyzzwB4+OGHGTJkCAMGDOC8885j27ZtTcY6fvx4Lr/8ckaOHMlRRx3FvHnzuOSSS+jduzfjx49PtXvyySfp378//fr144YbbkhN//Wvf83RRx/NiBEjeP3111PT6+rqOO+88xgyZAhDhgzZ4zMRKR1tOinky4oVKxg8ePAe0w488EC6devGmjVrAFiwYAGPPfYYc+fO3aPdL37xCw455BCWLl3Kj3/8YxYtWpRxGatXr+bKK69kxYoVHHzwwTz77LMAnHvuuSxcuJAlS5bQu3dvHn300azx/u1vf2Pu3Lncc889nHXWWVx77bWsWLGCZcuWsXjxYtavX88NN9zA3LlzWbx4MQsXLuT5559nw4YNTJ06lddff50///nPvPXWW6l5Tp48mWuvvZaFCxfy7LPP8r3vfa9Z61BEsps6Nf5lJHnzWiKqq/c8Qqjvp5s6teWHY+6e8WqW9OmnnHIKhx566F5tXnvtNSZPngxAv379qKyszLiMHj16MHDgQAAGDx7M2rVrAVi+fDk333wzmzdvZuvWrYwaNSprvGeddRZmRv/+/Tn88MPp378/AH379mXt2rWsW7eOE044gfqSIuPGjWP+/PkAe0y/6KKLWLVqFQAvvfTSHkliy5YtfPLJJ1ljEZHo4r4cFdpoUqhfsWa7++ly0bdv39Qv93pbtmzh/fffp2fPnixatIj9998/43c9YgBf+tKXUu/LyspS3Ufjx4/n+eefZ8CAAUyfPp1XX3018rzatWu3x3zbtWvHzp072WefxjeLxi7l/OKLL1iwYAH77bdflD9HRIqUuo/y4KSTTmLbtm08/njw6Ihdu3bxgx/8gPHjx9OhQ4cmv/vNb36Tp59+GoC33nqLZcuWNWvZn3zyCZ07d2bHjh3MmDGjZX9AA8OGDWPevHls3LiRXbt28eSTTzJixAiGDRvGq6++yqZNm9ixYwfPPPNM6junnnoq999/f2p88eLFeYlFRAqrTSeFfPXPmRnPPfcczzzzDL169eLoo4+mvLycf/iHf8j63SuuuIK6ujoqKyu58847qays5KCDDoq87J/85CcMGzaMU045ha997Wu5/BkpnTt35o477mDkyJEMGDCAY489ljFjxtC5c2eqq6s5/vjjOfnkkzn22N0Py7v33nupqamhsrKSPn368Mtf/jIvsYhIYVnU7otiUVVV5Q0fsvP222/Tu3fvhCLKza5du9ixYwfl5eW88847nHTSSaxatYp999036dBavVLebkSay8wWuXtVtnZt7pxCsdm2bRsjR45kx44duDsPPvigEoKIJEZJIWEdO3bU40VFpGi06XMKIiKyJyUFERFJUVIQEZEUJQUREUlRUsiTAw44YK9pd999N3369KGyspKTTjqJdevWFTyulpTQnj17Nj/96U9zXvYJJ5wQ+0n08ePHZ604G6WNiATaZFKYMQMqKqBdu+A1TzcC72XQoEHU1NSwdOlSzj//fH74wx9m/c7OnTvjCSainTt3Mnr0aG688cZE4xCRZMSaFMzsNDNbaWZrzGyvvYyZdTezl81sqZm9amZd4owHggQwcSKsWxfUPVq3LhiPIzGMHDkyVebiuOOOo7a2NmO78ePHc9111zFy5EhuuOEGPv30Uy655BKGDBnCoEGDeOGFF4Cmy2ynH6nMmjVrjzLY9Rors91w+ekPxhk4cGBq2G+//Zg3b16j8X322WdcfPHFqfjq6zM1VFFRwU033cTxxx9PVVUVb7zxBqNGjaJnz56pO6Hdneuvv55+/frRv39/nnrqqdT0SZMm0adPH7797W/z4Ycfpua7aNEiRowYweDBgxk1ahQbNmyI9g8lIimx3adgZmXAA8ApQC2w0Mxmu/tbac1+Bjzu7o+Z2YnAHcDfxxUTwJQp0PCRA9u2BdPHjYtvuY8++iinn356o5+vWrWKl156ibKyMm666SZOPPFEpk2bxubNmxk6dCgnn3wyDz74YKrM9vLly1NVU6M699xzufTSSwG4+eabefTRR7nqqqv2Wv706dNT36mvYfS73/2Ou+66i69//etMnTo1Y3y/+tWv6NChA0uXLmXp0qV7lMFoqGvXrixYsIBrr72W8ePH8/rrr7N9+3b69u3LZZddxm9/+1sWL17MkiVL2LhxI0OGDGH48OEsWLCAlStXsmzZMj744AP69OnDJZdcwo4dO7jqqqt44YUX6NSpE0899RRTpkxh2rRpzVpHIm1dnDevDQXWuPu7AGY2ExgDpCeFPsC14ftXgOdjjAeA995r3vR8eOKJJ6ipqWHevHmNtrngggsoKysD4MUXX2T27NmpcwHbt2/nvffei1xmuzFNldlOX35Dq1ev5vrrr2fu3Lm0b9++0fjmz5/P1VdfDUBlZWWT8Y0ePRqA/v37s3XrVjp27EjHjh0pLy9n8+bNvPbaa4wdO5aysjIOP/xwRowYwcKFC5k/f35q+hFHHMGJJ54IwMqVK1m+fDmnnHIKEJQP6dy5c7PWj4jEmxSOBN5PG68FhjVoswQ4D/gn4Bygo5l92d03xRVUt25Bl1Gm6XF46aWXuP3225k3b16qTPWUKVP4wx/+AOz+JZ5eWtvdefbZZznmmGP2mFdTdarSS1pv3749Y5umymw3Vtr7008/5cILL+Thhx/miCOOaDK+hnE0JVv57qh/az13p2/fvixYsCDS8kUkszjPKWTaOzT8n/6/gRFm9iYwAvhPYK8zrWY20cxqzKymrq4up6Buvx0aVrPu0CGYnm9vvvkm3//+95k9ezZf+cpX0mK4ncWLFzdaXnrUqFHcd999qR3jm2++CTRdZvvwww/n7bff5osvvuC5557LON+WlNmeMGECEyZM4Fvf+lbW+IYPH56a7/Lly1m6dGmkZWQyfPhwnnrqKXbt2kVdXR3z589n6NChDB8+nJkzZ7Jr1y42bNjAK6+8AsAxxxxDXV1dKins2LGDFStWtHj5Im1VnEcKtUDXtPEuwPr0Bu6+HjgXwMwOAM5z948bzsjdHwIegqBKai5B1Z83mDIl6DLq1i1ICLmeT9i2bRtduuw+T37dddcxZ84ctm7dygUXXABAt27dmD17dtZ5/fjHP+aaa66hsrISd6eiooLf//73XHHFFXznO9+hsrKSQYMG7VFm+6c//SlnnnkmXbt2pV+/fmzdunWv+daX2e7evTv9+/fP+mS0devWMWvWLFatWpXqm3/kkUcaje/yyy9nwoQJVFZWMnDgQIYOHRp5/TV0zjnnsGDBAgYMGICZcdddd/HVr36Vc845h7lz59K/f//Us6IB9t13X2bNmsXVV1/Nxx9/zM6dO7nmmmvo27dvi2MQaYtiK51tZvsAq4CTCI4AFgJ/5+4r0tocBnzk7l+Y2e3ALne/pan5trbS2c2hMtv51Va2GxEogtLZ7r7TzCYBfwLKgGnuvsLMbgNq3H02cAJwh5k5MB+4Mq54WgOV2RaRuMVaOtvd5wBzGky7Je39LEC3mkakMtsiErdWc0dzqT1BTpKl7UUks1aRFMrLy9m0aZP+o0sk7s6mTZsoLy9POhSRotMqnrzWpUsXamtryfVyVWk7ysvL97haTEQCrSIptG/fnh49eiQdhohIyWsV3UciIpIfSgoiIpKipCAiIilKCiIikqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIilKCiIikqKkICIiKUoKIiKSoqQgIiIpSgoiIpKipCAiIilKCiIikhJrUjCz08xspZmtMbMbM3zezcxeMbM3zWypmZ0RZzwiItK02JKCmZUBDwCnA32AsWbWp0Gzm4Gn3X0QcDHwi7jiERGR7OI8UhgKrHH3d939c2AmMKZBGwcODN8fBKyPMR4REckizmc0Hwm8nzZeCwxr0KYaeNHMrgL2B06OMR4REckiziMFyzDNG4yPBaa7exfgDOCfzWyvmMxsopnVmFlNXV1dDKGKiAjEmxRqga5p413Yu3vou8DTAO6+ACgHDms4I3d/yN2r3L2qU6dOMYUrIiJxJoWFQC8z62Fm+xKcSJ7doM17wEkAZtabICnoUEBEJCGxJQV33wlMAv4EvE1wldEKM7vNzEaHzX4AXGpmS4AngfHu3rCLSURECiTOE824+xxgToNpt6S9fwv4RpwxiIgUi+rqYChmuqNZRNqMpHfIt96a7PKjUFIQkTajFHbKSVNSEBGJUXU1mAUD7H6f9FFLY5QURKRVS3qnXF0N7sEAu98Xa1KwUrvYp6qqymtqapIOQ0RKkNnunXNbW76ZLXL3qmztdKQgIlIgU6cmHUF2Sgoi0mYkvVMu1i6jdEoKItJmlMJOOWlKCiIikpI1KZjZ/vWVS83saDMbbWbt4w9NREQKLcqRwnyg3MyOBF4GJgDT4wxKRESSESUpmLtvA84F7nP3cwgerykiIq1MpKRgZscD44A/hNNiLaQnIiLJiJIUrgF+BDwXlr4+Cngl3rBERCQJWZOCu89z99Hufmd4wnmju19dgNhERIpKW7ikNcrVR78xswPNbH/gLWClmV0ff2giIsWlLVRZjdJ91MfdtwBnEzwwpxvw97FGJSIiiYiSFNqH9yWcDbzg7juA0qqiJyLSQklXWS20KEnhV8BaYH9gvpl1B7bEGZSISLEotdLXucp6aam73wvcmzZpnZmNjC8kERFJSpQTzQeZ2d1mVhMOPyc4asjKzE4zs5VmtsbMbszw+T1mtjgcVpnZ5hb8DSIiBZF0ldVCiNJ9NA34BLgwHLYAv872JTMrAx4ATie4A3qsme1xJ7S7X+vuA919IHAf8NvmhS8iUjittcsoXZQ7k3u6+3lp47ea2eII3xsKrHH3dwHMbCYwhuCy1kzGAm0gD4uIFK8oRwqfmdk360fM7BvAZxG+dyTwftp4bThtL+HJ6x7A3EY+n1jffVVXVxdh0SIi0hJRjhQuBx4zs4MAAz4Cxkf4nmWY1tilrBcDs9x9V6YP3f0h4CEIntEcYdkiItICUa4+WgwMMLMDw/Gol6PWAl3TxrsA6xtpezFwZcT5iohITBpNCmZ2XSPTAXD3u7PMeyHQy8x6AP9JsOP/uwzzOwY4BFgQLWQREYlLU0cKHXOZsbvvNLNJwJ+AMmBaWGX1NqDG3WeHTccCM91d3UIiIgmzUtsXV1VVeU1NTdJhiIiUFDNb5O5V2dpFufpIRETaCCUFERFJUVIQEZGUrJekmtmXgPOAivT27n5bfGGJiEgSoty89gLwMbAI+O94wxERkSRFSQpd3P202CMREZHERTmn8K9m1j/2SEREJHFRjhS+CYw3s/8g6D4ywN29MtbIRESk4KIkhdNjj0JEJILq6rbxTIMkZe0+cvd1wMHAWeFwcDhNRKSgbr016QhavyiP45wMzAC+Eg5PmNlVcQcmIiKFF+VE83eBYe5+i7vfAhwHXBpvWCIigepqMAsG2P1e3UjxiHJOwYD0h9/sIvMDdERE8i79PIIZlFgNz5ITJSn8GvirmT0Xjp8NPBpfSCIikpQoT16728xeJbg01YAJ7v5m3IGJiDQ0dWrSEbR+TT157UB332JmhwJrw6H+s0Pd/aP4wxMR2U3nEeLX1JHCb4AzCWoepffiWTh+VIxxiYhIAhq9+sjdzwxfe7j7UWlDD3dXQhBpg3L9pa5f+sUvyn0KL0eZJiKtX643j+nms+LXaFIws/LwfMJhZnaImR0aDhXAEVFmbmanmdlKM1tjZjc20uZCM3vLzFaY2W9a8keIiEh+NHWk8H2C8wlfC1/rhxeAB7LN2MzKwnanA32AsWbWp0GbXsCPgG+4e1/gmhb8DSISo1xvHtPNZ6XFPMudIGZ2lbvf1+wZmx0PVLv7qHD8RwDufkdam7uAVe7+SNT5VlVVeU1NTXPDEZE8yPXmMd18lhwzW+TuVdnaRblP4T4z60fwa788bfrjWb56JPB+2ngtMKxBm6PDYF8HygiSyB+zxSQiIvGI8ozmqcAJBElhDkF30GtAtqSQqRRGw98I+wC9wvl3Af6fmfVz980NYpgITATo1q1btpBFJCa53jymm8+KX5SCeOcDJwH/5e4TgAHAlyJ8rxbomjbeBVifoc0L7r7D3f8DWEmQJPbg7g+5e5W7V3Xq1CnCokUkDroktfWLkhQ+c/cvgJ1mdiDwIdFuXFsI9DKzHma2L3AxMLtBm+eBkQBmdhhBd9K7UYMXEZH8ilIQr8bMDgYeJrj6aCvwb9m+5O47zWwS8CeC8wXT3H2Fmd0G1Lj77PCzU83sLYLqq9e7+6YW/i0iIpKjrFcf7dE4uEfhQHdfGldA2ejqIxGR5sv56iMzO7apz9z9jZYGJyIixamp7qOfh6/lQBWwhOCKokrgrwSltEVEpBVpqiDeSHcfCawDjg2v/hkMDALWFCpAEREpnChXH33N3ZfVj7j7cmBgfCGJiEhSolx99LaZPQI8QXDz2f8E3o41KhERSUSUpDABuByYHI7PBx6MLSIREUlMlNpH24F7wkFERFqxpi5JfdrdLzSzZexdswh3r4w1MhERKbimjhTqu4vOLEQgIiKSvEaTgrtvCF/XFS4cERFJUlOP4/zEzLZkGD4xsy2FDFJE8kNVSiWbpm5e6+juB2YYOrr7gYUMUkTy49Zbk45Ail2US1IBMLOvsOeT196LJSIREUlM1juazWy0ma0G/gOYB6wF/iXmuEQkT6qrg2cjW/gsxPr36kqSTKKUufgJcBywyt17EDyF7fVYoxKRjFqyI6+uBvdggN3vlRQkkyhJYUf44Jt2ZtbO3V9BtY9EEqFzAhK3KOcUNpvZAQTlLWaY2YfAznjDEpE4TJ2adARS7KIcKYwBPgOuBf4IvAOcFWdQIrJbPs8JqMtIsmn0cZxmdj/wG3f/18KG1DQ9jlPaMrPd5wZEmiPq4zibOlJYDfzczNaa2Z1mpvMIIiKtXFM3r/2Tux8PjAA+An5tZm+b2S1mdnSUmZvZaWa20szWmNmNGT4fb2Z1ZrY4HL7X4r9EpA3QOQGJW9ZzCu6+zt3vdPdBwN8B5xDhITtmVgY8AJwO9AHGmlmfDE2fcveB4fBI88IXKZxi6I8vhhikdYty81p7MzvLzGYQ3LS2CjgvwryHAmvc/V13/xyYSXDSWqQk6XJQaQuaKoh3iplNA2qBicAcoKe7X+Tuz0eY95HA+2njteG0hs4zs6VmNsvMujYjdhERybOmjhRuAhYAvd39LHef4e6fNmPelmFaw+smfgdUhA/seQl4LOOMzCaaWY2Z1dTV1TUjBJHc6HJQaWsavSQ15xmbHQ9Uu/uocPxHAO5+RyPty4CP3P2gpuarS1IlKbleDqrLSSVJ+bgkNVcLgV5m1sPM9gUuBmanNzCzzmmjo4lwAltEROITW1Jw953AJOBPBDv7p919hZndZmajw2ZXm9kKM1sCXA2MjysekVy15HJQVSiVUhNb91Fc1H0kpUrdR5KkYug+EhGREqOkIFIguhtZSoGSgkiB6DyClAIlBRERSVFSEJHYzZgBFRXQrl3wOmNG0hFJY6I8eU1EpMVmzICJE2HbtmB83bpgHGDcuOTiksx0pCAisZoyZXdCqLdtWzBdio+SgojE6r33mjc9k1Lvfiql+NV9JCKx6tYt6DLKND2KUu9+KrX4daQgIrG6/Xbo0GHPaR06BNOjKPXup1KLX0lB2gzdJ5CMcePgoYege/eg1Ef37sF41F/J+eh+ylUu3T/FEH9zKClIm6Enp7Vcrn3i48bB2rXwxRfBa3O6TRrrZora/ZSr+u6fdeuC2lX13T9R10HS8TeXkoKINCnXnWKucu1+ylWu3T/5iL+gJ6rdvaSGwYMHu0hUU6e6B7uyPYepU5OOrHR07555HXbvXrgYnngiWJ5Z8PrEE4Vbtlnmv98s+jxyif+JJ9w7dNhz2R06NH8dADUeYR+r0tnSZqh0dcu0a5d5vZkF3UGlYMaM4Jf9e+8F3Ta33x69C6uiIvPVU927B11hccvX8lU6W0TyotT6xBvKtfsr6e6rQp+oVlKQNkOlq1sm6Z1irnI9J5Dr1VO5KnRSVveRiGSVS/dL0kq9+6vhzW8QJOXmJiZ1H4lI3uRySWnSSr37q9BHKrEmBTM7zcxWmtkaM7uxiXbnm5mbWdYsJiLSHKXe/QWFTcqxJQUzKwMeAE4H+gBjzaxPhnYdgauBv8YVi4i0XUmfEyg1cR4pDAXWuPu77v45MBMYk6HdT4C7gO0xxiKtgMpUtFwpVemMQyl3fxVanEnhSOD9tPHacFqKmQ0Curr772OMQ1oJlalomaTvSJbSEmdSsAzTUtcAmFk74B7gB1lnZDbRzGrMrKaurq7ZgbT1X0lS+nLZhkutSqckK86kUAt0TRvvAqxPG+8I9ANeNbO1wHHA7Ewnm939IXevcveqTp06NSsI/UoqbdXVQT+whT8x6t+3pa6kXLfhUqvSKQmLUgujJQPBA3zeBXoA+wJLgL5NtH8VqMo23+bWPiqGui1JS7JuTD5BMsvNdf3l+v1ct2H9HxD36LWPYi1eB5wBrALeAaaE024DRmdoG0tSyEcxq1KWr2JaxSCJpJDr+svH+s91G25N24C0XFEkhTgGHSk0TzH8/fk6UmlpZdNcll8Mv9LzMY/WcrQoLaekEGrrv5KSPlLKx/pPsuxwrusvX2WX2/I2LPmhpJCmLf9KSvpXZq7Lz3WHmPQv/XwdqbXlbVjyQ0lB3D35PvFcfynnulNNuj9ev/KlWCgpSEop96knnVTck7/6SCQfoiYFlc6WJuVadjjXsr+5PnUqX2WHRUqdSmdLXuRadji9GBk0vxhZrhUuVQxNpHl0pCBNyucv7ZY+I7mUH/AiUix0pCB5kesv/XyUqVCFS5HC0ZGCRNbSX/r5+r6ItJyOFFoRVXkVkUJRUihySVd5zWeV0qlT8xmZiMRB3UdFLtdLMvNJ3T8ipUvdR3mUa/dNLt9XLXwRKaQ2lRRa0uWRa/dNrt/P9T6BfFL3j0jr16a6j1rS/ZFr943uyBWRYqDuozzJtfsm1+/rjlwRKaRWnxRyvXom1+6bfHT/6OYtESmUNpEU6mtjwu73UZNCrrV3cv1+PrWlh92LSMu0+qSQq1y7b4qp++fWWwu/TBEpLW3qRHN1ddv+taz7DETarqI40Wxmp5nZSjNbY2Y3Zvj8MjNbZmaLzew1M+sTZzxtMSHk845kEWn9YjtSMLMyYBVwClALLATGuvtbaW0OdPct4fvRwBXuflpT821rdzTnk44URNquYjhSGAqscfd33f1zYCYwJr1BfUII7Q9olyUikqA4k8KRwPtp47XhtD2Y2ZVm9g5wF3B1jPGUvFy7fHRHsohkE2dSsAzT9joScPcH3L0ncANwc8YZmU00sxozq6mrq8tzmKUj16uHdB5BRLKJMynUAl3TxrsA65toPxM4O9MH7v6Qu1e5e1WnTp3yGGLzaKcqIq1dnElhIdDLzHqY2b7AxcDs9AZm1itt9NvA6hjjyVkSv9R19ZCIFFKs9ymY2RnAPwJlwDR3v93MbgNq3H22mf0TcDKwA/gbMMndVzQ1zySvPkr6cZS6ekhEWqoYrj7C3ee4+9Hu3tPdbw+n3eLus8P3k929r7sPdPeR2RJCEvRLXUTaEpW5yCLX2kl6nKWIlJI2VeYiV+r+EZFSVRTdR62NfqmLSGunpNAMunlMRFo7JYUC0slpESl2SgoiIpKipCAiIilKCiIikqKkICIiKUoKIiKSUnI3r5lZHbAu6TgacRiwMekgmqD4clPs8UHxx6j4cpNLfN3dPWuZ6ZJLCsXMzGqi3DGYFMWXm2KPD4o/RsWXm0LEp+4jERFJUVIQEZEUJYX8eijpALJQfLkp9vig+GNUfLmJPT6dUxARkRQdKYiISIqSQjOZWVcze8XM3jb5J+zJAAAGEklEQVSzFWY2OUObE8zsYzNbHA63FDjGtWa2LFz2Xg+fsMC9ZrbGzJaa2bEFjO2YtPWy2My2mNk1DdoUfP2Z2TQz+9DMlqdNO9TM/mxmq8PXQxr57nfCNqvN7DsFiu3/mtm/h/9+z5nZwY18t8ltIeYYq83sP9P+Hc9o5LunmdnKcHu8sYDxPZUW21ozW9zId2Ndh43tUxLb/txdQzMGoDNwbPi+I7AK6NOgzQnA7xOMcS1wWBOfnwH8C2DAccBfE4qzDPgvguunE11/wHDgWGB52rS7gBvD9zcCd2b43qHAu+HrIeH7QwoQ26nAPuH7OzPFFmVbiDnGauB/R9gG3gGOAvYFljT8/xRXfA0+/zlwSxLrsLF9SlLbn44UmsndN7j7G+H7T4C3gSOTjarZxgCPe+AvwMFm1jmBOE4C3nH3xG9GdPf5wEcNJo8BHgvfPwacneGro4A/u/tH7v434M/AaXHH5u4vuvvOcPQvQJd8LrO5Gll/UQwF1rj7u+7+OTCTYL3nVVPxmZkBFwJP5nu5UTSxT0lk+1NSyIGZVQCDgL9m+Ph4M1tiZv9iZn0LGhg48KKZLTKziRk+PxJ4P228lmQS28U0/h8xyfVX73B33wDBf1zgKxnaFMO6vITgyC+TbNtC3CaFXVzTGun+KIb19y3gA3df3cjnBVuHDfYpiWx/SgotZGYHAM8C17j7lgYfv0HQJTIAuA94vsDhfcPdjwVOB640s+ENPrcM3ynoZWhmti8wGngmw8dJr7/mSHRdmtkUYCcwo5Em2baFOD0I9AQGAhsIumgaSnxbBMbS9FFCQdZhln1Ko1/LMC2n9aek0AJm1p7gH2+Gu/+24efuvsXdt4bv5wDtzeywQsXn7uvD1w+B5wgO0dPVAl3TxrsA6wsTXcrpwBvu/kHDD5Jef2k+qO9WC18/zNAmsXUZnlQ8ExjnYQdzQxG2hdi4+wfuvsvdvwAebmTZiW6LZrYPcC7wVGNtCrEOG9mnJLL9KSk0U9j/+Cjwtrvf3Uibr4btMLOhBOt5U4Hi29/MOta/JzghubxBs9nA/wqvQjoO+Lj+MLWAGv11luT6a2A2UH81x3eAFzK0+RNwqpkdEnaPnBpOi5WZnQbcAIx2922NtImyLcQZY/p5qnMaWfZCoJeZ9QiPHi8mWO+FcjLw7+5em+nDQqzDJvYpyWx/cZ1Rb60D8E2Cw7OlwOJwOAO4DLgsbDMJWEFwJcVfgK8XML6jwuUuCWOYEk5Pj8+ABwiu+lgGVBV4HXYg2MkflDYt0fVHkKA2ADsIfn19F/gy8DKwOnw9NGxbBTyS9t1LgDXhMKFAsa0h6Euu3wZ/GbY9ApjT1LZQwPX3z+H2tZRgB9e5YYzh+BkEV9y8E1eMmeILp0+v3+7S2hZ0HTaxT0lk+9MdzSIikqLuIxERSVFSEBGRFCUFERFJUVIQEZEUJQUREUlRUhAJmdku27OCa94qdppZRXqFTpFitU/SAYgUkc/cfWDSQYgkSUcKIlmE9fTvNLN/C4f/EU7vbmYvhwXfXjazbuH0wy14xsGScPh6OKsyM3s4rJn/opntF7a/2szeCuczM6E/UwRQUhBJt1+D7qOL0j7b4u5DgfuBfwyn3U9QgrySoCDdveH0e4F5HhT0O5bgTliAXsAD7t4X2AycF06/ERgUzueyuP44kSh0R7NIyMy2uvsBGaavBU5093fDwmX/5e5fNrONBKUbdoTTN7j7YWZWB3Rx9/9Om0cFQd37XuH4DUB7d/8/ZvZHYCtBNdjnPSwGKJIEHSmIROONvG+sTSb/nfZ+F7vP6X2boBbVYGBRWLlTJBFKCiLRXJT2uiB8/68EVT0BxgGvhe9fBi4HMLMyMzuwsZmaWTugq7u/AvwQOBjY62hFpFD0i0Rkt/1sz4e3/9Hd6y9L/ZKZ/ZXgh9TYcNrVwDQzux6oAyaE0ycDD5nZdwmOCC4nqNCZSRnwhJkdRFC99h5335y3v0ikmXROQSSL8JxClbtvTDoWkbip+0hERFJ0pCAiIik6UhARkRQlBRERSVFSEBGRFCUFERFJUVIQEZEUJQUREUn5/0phTCoMDb/UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20ad5e21e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "l2_model_val_loss = l2_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "As you can see, the model with L2 regularization (dots) has become much more resistant to overfitting than the reference model (crosses), \n",
    "even though both models have the same number of parameters.\n",
    "\n",
    "As alternatives to L2 regularization, you could use one of the following Keras weight regularizers:\n",
    "\n",
    "如你所见，即使两个模型的参数个数相同，具有 L2正则化的模型（圆点）比参考模型（十字）更不容易过拟合。\n",
    "\n",
    "你还可以用 Keras 中以下这些权重正则化项来代替 L2 正则化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.regularizers.L1L2 at 0x20ad24a7198>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "# L1 regularization（L1 正则化）\n",
    "regularizers.l1(0.001)\n",
    "\n",
    "# L1 and L2 regularization at the same time（同时做 L1 和 L2 正则化）\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding dropout\n",
    "\n",
    "\n",
    "Dropout is one of the most effective and most commonly used regularization techniques for neural networks, developed by Hinton and his \n",
    "students at the University of Toronto. Dropout, applied to a layer, consists of randomly \"dropping out\" (i.e. setting to zero) a number of \n",
    "output features of the layer during training. Let's say a given layer would normally have returned a vector `[0.2, 0.5, 1.3, 0.8, 1.1]` for a \n",
    "given input sample during training; after applying dropout, this vector will have a few zero entries distributed at random, e.g. `[0, 0.5,1.3, 0, 1.1]`. The \"dropout rate\" is the fraction of the features that are being zeroed-out; it is usually set between 0.2 and 0.5. At test \n",
    "time, no units are dropped out, and instead the layer's output values are scaled down by a factor equal to the dropout rate, so as to \n",
    "balance for the fact that more units are active than at training time.\n",
    "\n",
    "Consider a Numpy matrix containing the output of a layer, `layer_output`, of shape `(batch_size, features)`. At training time, we would be \n",
    "zero-ing out at random a fraction of the values in the matrix:\n",
    "\n",
    "## 添加 dropout 正则化\n",
    "\n",
    "dropout 是神经网络最有效也最常用的正则化方法之一，它是由多伦多大学的 Geoffrey Hinton和他的学生开发的。对某一层使用 dropout，就是在训练过程中随机将该层的一些输出特征舍弃（设置为 0）。假设在训练过程中，某一层对给定输入样本的返回值应该是向量 `[0.2, 0.5, 1.3, 0.8, 1.1]`。使用 dropout 后，这个向量会有几个随机的元素变成 0，比如 `[0, 0.5,1.3, 0, 1.1]`。dropout 比率（dropout rate）是被设为 0 的特征所占的比例，通常在 0.2~0.5 范围内。测试时没有单元被舍弃，而该层的输出值需要按 dropout 比率缩小，因为这时比训练时有更多的单元被激活，需要加以平衡。\n",
    "\n",
    "假设有一个包含某层输出的 Numpy 矩阵 `layer_output`，其形状为 `(batch_size, features)`。训练时，我们随机将矩阵中一部分值设为 0。\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "#At training time: we drop out 50% of the units in the output(训练时候我们舍弃50%的输出单元)\n",
    "\n",
    "layer_output *= np.randint(0, high=2, size=layer_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "At test time, we would be scaling the output down by the dropout rate. Here we scale by 0.5 (because we were previous dropping half the \n",
    "units):\n",
    "\n",
    "测试时，我们将输出按 dropout 比率缩小。这里我们乘以 0.5（因为前面舍弃了一半的单元）。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "#At test time:（测试时）\n",
    "\n",
    "layer_output *= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Note that this process can be implemented by doing both operations at training time and leaving the output unchanged at test time, which is \n",
    "often the way it is implemented in practice:\n",
    "\n",
    "注意，为了实现这一过程，还可以让两个运算都在训练时进行，而测试时输出保持不变。这通常也是实践中的实现方式：\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# At training time:\n",
    "layer_output *= np.randint(0, high=2, size=layer_output.shape)\n",
    "# Note that we are scaling *up* rather scaling *down* in this case\n",
    "layer_output /= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This technique may seem strange and arbitrary. Why would this help reduce overfitting? Geoff Hinton has said that he was inspired, among \n",
    "other things, by a fraud prevention mechanism used by banks -- in his own words: _\"I went to my bank. The tellers kept changing and I asked \n",
    "one of them why. He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation \n",
    "between employees to successfully defraud the bank. This made me realize that randomly removing a different subset of neurons on each \n",
    "example would prevent conspiracies and thus reduce overfitting\"_.\n",
    "\n",
    "The core idea is that introducing noise in the output values of a layer can break up happenstance patterns that are not significant (what \n",
    "Hinton refers to as \"conspiracies\"), which the network would start memorizing if no noise was present. \n",
    "\n",
    "In Keras you can introduce dropout in a network via the `Dropout` layer, which gets applied to the output of layer right before it, e.g.:\n",
    "\n",
    "这一方法可能看起来有些奇怪和随意。它为什么能够降低过拟合？ Hinton 说他的灵感之一来自于银行的防欺诈机制。用他自己的话来说：“我去银行办理业务。柜员不停地换人，于是我问其中一人这是为什么。他说他不知道，但他们经常换来换去。我猜想，银行工作人员要想成功欺诈银行，他们之间要互相合作才行。这让我意识到，在每个样本中随机删除不同的部分神经元，可以阻止它们的阴谋，因此可以降低过拟合。”\n",
    "\n",
    "其核心思想是在层的输出值中引入噪声， 打破不显著的偶然模式（Hinton 称之为阴谋）。如果没有噪声的话，网络将会记住这些偶然模式。\n",
    "\n",
    "在 Keras 中，你可以通过 Dropout 层向网络中引入 dropout，dropout 将被应用于前面一层的输出。\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "model.add(layers.Dropout(0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add two `Dropout` layers in our IMDB network to see how well they do at reducing overfitting:\n",
    "\n",
    "我们向 IMDB 网络中添加两个 Dropout 层，来看一下它们降低过拟合的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpt_model = models.Sequential()\n",
    "dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(16, activation='relu'))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "dpt_model.compile(optimizer='rmsprop',\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 5s 181us/step - loss: 0.5906 - acc: 0.6839 - val_loss: 0.4324 - val_acc: 0.8620\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 4s 150us/step - loss: 0.4359 - acc: 0.8186 - val_loss: 0.3475 - val_acc: 0.8706\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.3469 - acc: 0.8714 - val_loss: 0.2912 - val_acc: 0.8872\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.2880 - acc: 0.8983 - val_loss: 0.2768 - val_acc: 0.8881\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.2531 - acc: 0.9135 - val_loss: 0.2805 - val_acc: 0.8883\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.2220 - acc: 0.9265 - val_loss: 0.2892 - val_acc: 0.8877\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 4s 156us/step - loss: 0.1965 - acc: 0.9355 - val_loss: 0.3208 - val_acc: 0.8852\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 4s 160us/step - loss: 0.1765 - acc: 0.9425 - val_loss: 0.3333 - val_acc: 0.8842\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.1707 - acc: 0.9454 - val_loss: 0.3629 - val_acc: 0.8840\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.1513 - acc: 0.9512 - val_loss: 0.3759 - val_acc: 0.8827\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 4s 159us/step - loss: 0.1453 - acc: 0.9534 - val_loss: 0.3952 - val_acc: 0.8820\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.1363 - acc: 0.9582 - val_loss: 0.4362 - val_acc: 0.8760\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.1245 - acc: 0.9605 - val_loss: 0.4527 - val_acc: 0.8762\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 4s 155us/step - loss: 0.1205 - acc: 0.9608 - val_loss: 0.4528 - val_acc: 0.8762\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.1147 - acc: 0.9630 - val_loss: 0.4915 - val_acc: 0.8745\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.1145 - acc: 0.9641 - val_loss: 0.5120 - val_acc: 0.8754\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 4s 151us/step - loss: 0.1082 - acc: 0.9657 - val_loss: 0.5390 - val_acc: 0.8738\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 4s 154us/step - loss: 0.1101 - acc: 0.9657 - val_loss: 0.5410 - val_acc: 0.8735\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 4s 152us/step - loss: 0.0979 - acc: 0.9684 - val_loss: 0.5788 - val_acc: 0.8734\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 4s 153us/step - loss: 0.1015 - acc: 0.9684 - val_loss: 0.5901 - val_acc: 0.8720\n"
     ]
    }
   ],
   "source": [
    "dpt_model_hist = dpt_model.fit(x_train, y_train,\n",
    "                               epochs=20,\n",
    "                               batch_size=512,\n",
    "                               validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the results:\n",
    "\n",
    "我们将结果绘制出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XucFPWZ7/HPw4AiiqABjeE2wIFVriMMtw0RiTe8ICoxwhJXdJVFZUWTeDRLIqPmpnH1RDEaNIJG4gWNhJPDRo2KrgYTBjOKgALCEEa8jKggQeIAz/mjappm6J7pme7qnp7+vl+venVX9a+qnmmafrp+VfX8zN0REREBaJXrAEREpPlQUhARkRglBRERiVFSEBGRGCUFERGJUVIQEZEYJQUREYlRUhARkZjIkoKZPWBmH5rZm0leNzO708zWm9kbZjYkqlhERCQ1rSPc9nxgDvBQktdPB/qE0wjgnvCxXp06dfLi4uLMRCgiUiBWrFjxkbt3bqhdZEnB3V8ys+J6mkwAHvKgzsarZtbRzI5x9/fq225xcTHl5eUZjFREpOUzs02ptMvlOYUuwOa4+apw2QHMbJqZlZtZeXV1dVaCExEpRLlMCpZgWcLqfO4+191L3b20c+cGj35ERKSJcpkUqoBucfNdgS05ikVERIj2RHNDFgMzzOxRghPM2xo6n5BMTU0NVVVV7Nq1K6MBimRS27Zt6dq1K23atMl1KCJJRZYUzOwR4ESgk5lVAbOBNgDufi+wBDgDWA/sBC5u6r6qqqpo3749xcXFmCXqlRLJLXdn69atVFVV0bNnz1yHI5JUlFcfTW7gdQeuzMS+du3apYQgzZqZ8aUvfQldKCHpKCsLpii1mDualRCkudNnVNJ1443R76PFJAUREUmfkkKGVFVVMWHCBPr06UPv3r2ZOXMmX3zxRcK2W7Zs4Rvf+EaD2zzjjDP49NNPmxRPWVkZt912W5PWTdX8+fOZMWNG2m1EJLmyMjALJtj3PKpupIJOCpl6U92d8847j3POOYd169axdu1aduzYwaxZsw5ou3v3br7yla/wxBNPNLjdJUuW0LFjx8wEKSJ5qawM3IMJ9j1XUohApvrnnn/+edq2bcvFFwcXUBUVFXHHHXfwwAMPsHPnTubPn8/555/P+PHjOfXUU6msrGTAgAEA7Ny5k29+85sMGjSICy64gBEjRsTKeBQXF/PRRx9RWVnJcccdx2WXXUb//v059dRT+fzzzwG47777GDZsGIMHD2bixIns3Lmz3linTp3K5ZdfztixY+nVqxcvvvgil1xyCccddxxTp06NtXvkkUcYOHAgAwYM4LrrrostnzdvHn379mXMmDG88sorseXV1dVMnDiRYcOGMWzYsP1eE5H8UdBJIVNWrVrF0KFD91t2+OGH0717d9avXw/AsmXLePDBB3n++ef3a/eLX/yCI444gjfeeIMf/OAHrFixIuE+1q1bx5VXXsmqVavo2LEjTz75JADnnXcey5cv5/XXX+e4447jV7/6VYPxfvLJJzz//PPccccdjB8/nmuuuYZVq1axcuVKKioq2LJlC9dddx3PP/88FRUVLF++nEWLFvHee+8xe/ZsXnnlFZ599llWr14d2+bMmTO55pprWL58OU8++SSXXnppo95DEWnY7NnR7yOXN6/lRFnZ/kcItf10s2c3/XDM3RNeWRK//JRTTuHII488oM3LL7/MzJkzARgwYACDBg1KuI+ePXtSUlICwNChQ6msrATgzTff5Pvf/z6ffvopO3bs4LTTTmsw3vHjx2NmDBw4kKOPPpqBAwcC0L9/fyorK9m0aRMnnngitSVFpkyZwksvvQSw3/ILLriAtWvXAvDHP/5xvySxfft2PvvsswZjEZHURX05KhRoUqh9Y8329dOlo3///rFf7rW2b9/O5s2b6d27NytWrODQQw9NuK6nGMDBBx8ce15UVBTrPpo6dSqLFi1i8ODBzJ8/n6VLl6a8rVatWu233VatWrF7925at07+sUh2WeXevXtZtmwZhxxySCp/jog0U+o+yoCTTjqJnTt38tBDwdARe/bs4Tvf+Q5Tp06lXbt29a47evRoHn/8cQBWr17NypUrG7Xvzz77jGOOOYaamhoWLFjQtD+gjhEjRvDiiy/y0UcfsWfPHh555BHGjBnDiBEjWLp0KVu3bqWmpoaFCxfG1jn11FOZM2dObL6ioiIjsYhIdhV0UshU/5yZ8dRTT7Fw4UL69OlD3759adu2LT/+8Y8bXPeKK66gurqaQYMGccsttzBo0CA6dOiQ8r5vvvlmRowYwSmnnMKxxx6bzp8Rc8wxx/CTn/yEsWPHMnjwYIYMGcKECRM45phjKCsrY9SoUZx88skMGbJvsLw777yT8vJyBg0aRL9+/bj33nszEouIZJel2n3RXJSWlnrdQXbWrFnDcccdl6OI0rNnzx5qampo27Yt77zzDieddBJr167loIMOynVoEoF8/qxKfjOzFe5e2lC7gjun0Nzs3LmTsWPHUlNTg7tzzz33KCGISM4oKeRY+/btNbyoiDQbBX1OQURE9qekICIiMUoKIiISo6QgIiIxSgoZUlRURElJCf3792fw4MHcfvvt7N27N2fxLFq0aL+yE9l02GGHNXqddMqE11q6dClnnXVWWttoSHwxw3TaiDRXBZkUFiyA4mJo1Sp4zMSNwIcccggVFRWsWrWKZ599liVLlnBjgjKsu3fvTn9nKUg1Kbh7TpNX7f5VJlykeYg0KZjZODN728zWm9n1CV7vYWbPmdkbZrbUzLpGGQ8ECWDaNNi0Kah7tGlTMJ+hChEAHHXUUcydO5c5c+bg7geUznZ3rr32WgYMGMDAgQN57LHHgOCX7gknnMC5555Lv379mD59euwLO1kp6/hf5U888QRTp07lT3/6E4sXL+baa6+lpKSEd955Z7/4aktxX3HFFQwZMoTNmzfzzDPPMGrUKIYMGcL555/Pjh07gGBMh2OPPZbRo0dz1VVXxX6J1x3EZ8CAAbEifbV27NjBSSedxJAhQxg4cCC/+93vku6/tkz4vffeS0lJCSUlJfTs2ZOxY8cCJI3vD3/4Qyy+3/72twn/PebPn88555zD+PHj6dmzJ3PmzOH222/n+OOPZ+TIkXz88cdAUJpj5MiRDBo0iHPPPZdPPvkEgBUrVjB48GBGjRrF3XffHdvunj17uPbaaxk2bBiDBg3il7/8ZYOfDZFmz90jmYAi4B2gF3AQ8DrQr06bhcBF4fOvA79uaLtDhw71ulavXn3AsmR69KgdomL/qUePlDeR0KGHHnrAso4dO/r777/v8+bN8y5duvjWrVvd3f2JJ57wk08+2Xfv3u3vv/++d+vWzbds2eIvvPCCH3zwwf7OO+/47t27/eSTT/aFCxf6u+++6926dfMPP/zQa2pqfOzYsf7UU08dsN+FCxf6RRdd5O7uF110kS9cuDBhrBs3bnQz82XLlrm7e3V1tX/ta1/zHTt2uLv7T3/6U7/xxhv9888/965du/qGDRvc3X3SpEl+5plnurv77Nmz/Wc/+1lsm/379/eNGzfuF1NNTY1v27Ytto/evXv73r17D9i/u3uPHj28uro6Nv/FF1/46NGjffHixQ3Gt3btWt+7d6+ff/75sfjizZs3z3v37u3bt2/3Dz/80A8//HC/55573N396quv9jvuuMPd3QcOHOhLly51d/cf/OAHPnPmzAOWf/e73/X+/fu7u/svf/lLv/nmm93dfdeuXT506FDfsGGDb9y4MdamrsZ8VkUyCSj3FL67ozxSGA6sd/cN7v4F8CgwoU6bfsBz4fMXEryecX/7W+OWp8PjSojEl85++eWXmTx5MkVFRRx99NGMGTOG5cuXAzB8+HB69epFUVERkydP5uWXX2b58uWxktWtW7fer5R1U/Xo0YORI0cC8Oqrr7J69Wq++tWvUlJSwoMPPsimTZt466236NWrFz179gRg8uTJjf77//M//5NBgwZx8skn8+677/LBBx8csP9EZs6cyde//nXGjx9fb3w9e/akT58+mBnf+ta3km5v7NixtG/fns6dO9OhQwfGjx8PwMCBA6msrGTbtm18+umnjBkzBoCLLrqIl1566YDlF154YWybzzzzDA899BAlJSWMGDGCrVu3sm7duka9RyLNTZR3NHcBNsfNVwEj6rR5HZgI/Bw4F2hvZl9y961RBdW9e9BllGh5Jm3YsIGioiKOOuoogP1KZ8cni7rqlqY2s5Tb79q1K2GbzZs3x74Ep0+fzrhx4w6I55RTTuGRRx7Zb72//vWvSffbunXr/c5FJNr3ggULqK6uZsWKFbRp04bi4uJYu2SlxCHo7tm0aVOs6mqy+CoqKpKW8q6rbonw+PLh9Z3n8SRjZdS+dtdddx0whkXdbjSRfBLlkUKi/0l1v92+C4wxs78CY4B3gQP+h5rZNDMrN7Py6urqtIL60Y+gbjXrdu2C5ZlSXV3N9OnTmTFjRsIvlBNOOIHHHnuMPXv2UF1dzUsvvcTw4cMB+Mtf/sLGjRvZu3cvjz32GKNHj05ayhrg6KOPZs2aNezdu5ennnoqto/27dvHBrnp1q0bFRUVVFRUMH369APiGTlyJK+88kpslLidO3eydu1ajj32WDZs2BD7kqs99wHBUKGvvfYaAK+99hobN248YLvbtm3jqKOOok2bNrzwwgtsSpSN61ixYgW33XYbDz/8MK1atWowvo0bN8bOmdRNGo3RoUMHjjjiCP7nf/4HgF//+teMGTOGjh070qFDB15++WWA/cqTn3baadxzzz3U1NQAsHbtWv7+9783OQaR5iDKI4UqoFvcfFdgS3wDd98CnAdgZocBE919W90NuftcYC4EVVLTCWrKlOBx1qygy6h79yAh1C5vqs8//5ySkhJqampo3bo1F154Id/+9rcTtj333HNZtmwZgwcPxsy49dZb+fKXv8xbb73FqFGjuP7661m5cmXspHOrVq1ipazdnTPOOIMJE4Ketp/+9KecddZZdOvWjQEDBsROwE6aNInLLruMO++8kyeeeILevXsnjb1z587Mnz+fyZMn849//AOAH/7wh/Tt25df/OIXjBs3jk6dOsUSF8DEiRNjXSfDhg2jb9++B2x3ypQpjB8/ntLSUkpKSlIq7T1nzhw+/vjj2Anm0tJS7r///qTxzZ07lzPPPJNOnToxevRo3nzzzQb3kcyDDz7I9OnT2blzJ7169WLevHlAMC71JZdcQrt27fY7Krj00kuprKxkyJAhuDudO3dm0aJFTd6/SHMQWelsM2sNrAVOIjgCWA78i7uvimvTCfjY3fea2Y+APe5+Q33bbWmls+MtXbqU2267jd///ve5DiVmx44dHHbYYbg7V155JX369OGaa67JdVh5q6V8ViX/pFo6O7LuI3ffDcwAngbWAI+7+yozu8nMzg6bnQi8bWZrgaOBDHbiSCbcd999sZvytm3bxr//+7/nOiQRiZAG2RHJIn1WJVdyfqSQbfmW3KTw6DMq+aBFJIW2bduydetW/aeTZsvd2bp1K23bts11KCL1ahEjr3Xt2pWqqirSvVxVJEpt27ala9fIK7mIpKVFJIU2bdrE7roVEZGmaxHdRyIikhlKCiIiEqOkICIiMUoKIiISo6QgIiIxSgoiIhKjpCAiIjFKCiIiEqOkICIiMUoKIiISo6QgIiIxSgoiIhKjpCAiIjFKCiIiEqOkICIiMUoKIiISE2lSMLNxZva2ma03s+sTvN7dzF4ws7+a2RtmdkaU8YiISP0iSwpmVgTcDZwO9AMmm1m/Os2+Dzzu7scDk4BfRBWPiIg0LMojheHAenff4O5fAI8CE+q0ceDw8HkHYEuE8YiISAOiTApdgM1x81XhsnhlwLfMrApYAvxHog2Z2TQzKzez8urq6ihiFRERok0KlmCZ15mfDMx3967AGcCvzeyAmNx9rruXuntp586dIwhVREQg2qRQBXSLm+/Kgd1D/wY8DuDuy4C2QKcIYxIRkXpEmRSWA33MrKeZHURwInlxnTZ/A04CMLPjCJKC+odERHIksqTg7ruBGcDTwBqCq4xWmdlNZnZ22Ow7wGVm9jrwCDDV3et2MYmISJa0jnLj7r6E4ARy/LIb4p6vBr4aZQwiIs1FWVkwNWe6o1lECkauv5BvvDG3+0+FkoKIFIx8+FLONSUFEZEIlZWBWTDBvue5PmpJRklBRFq0XH8pl5WBezDBvufNNSlYvl3sU1pa6uXl5bkOQ0TykNm+L+dC27+ZrXD30oba6UhBRCRLZs/OdQQNU1IQkYKR6y/l5tplFE9JQUQKRj58KeeakoKIiMQ0mBTM7NDayqVm1tfMzjazNtGHJiIi2ZbKkcJLQFsz6wI8B1wMzI8yKBERyY1UkoK5+07gPOAudz+XYHhNERFpYVJKCmY2CpgC/L9wWaSF9EREJDdSSQpXA98DngpLX/cCXog2LBGR5qcQrl5q1B3N4Qnnw9x9e3Qh1U93NItIruT6juh0ZOyOZjP7jZkdbmaHAquBt83s2kwEKSIizUsq3Uf9wiODcwgGzOkOXBhpVCIizUSuC+plWypJoU14X8I5wO/cvQbI0wMoEZHGybcqp+lKJSn8EqgEDgVeMrMeQM7OKYiISHQaTArufqe7d3H3MzywCRibysbNbJyZvW1m683s+gSv32FmFeG01sw+bcLfICKSFbkuqJcNDd5vYGYdgNnACeGiF4GbgG0NrFcE3A2cAlQBy81ssbuvrm3j7tfEtf8P4PjG/gEiItnSUruM4qXSffQA8BnwzXDaDsxLYb3hwHp33+DuXwCPAhPqaT8ZeCSF7YqISERSuTO5t7tPjJu/0cwqUlivC7A5br4KGJGoYXieoifwfArbFRGRiKRypPC5mY2unTGzrwKfp7CeJViW7KqlScAT7r4n4YbMpplZuZmVV1dXp7BrERFpilSOFC4HHgzPLRjwMTA1hfWqgG5x812BLUnaTgKuTLYhd58LzIXgjuYU9i0iIk3QYFJw9wpgsJkdHs6nejnqcqCPmfUE3iX44v+Xuo3M7J+AI4BlqQYtIiLRSJoUzOzbSZYD4O6317dhd99tZjOAp4Ei4IGwoN5NQLm7Lw6bTgYe9cYUYRIRkUjUd6TQPt2Nu/sSgtIY8ctuqDNflu5+REQkM5ImBXe/MZuBiIhI7qVy9ZGIiBQIJQUREYlRUhARkZhUah8dDEwEiuPbu/tN0YUlIiK5kMrNa78jKH63AvhHtOGIiEgupZIUurr7uMgjERGRnEvlnMKfzGxg5JGIiEjOpZIURgMrwsFy3jCzlWb2RtSBiYjUVQjjGeSaNVRdIixrfYBwBLasKy0t9fLy8lzsWkRyzGzfWMnSOGa2wt1LG2qXynCcm4COwPhw6pirhCAiItFqMCmY2UxgAXBUOD0cDp0pIhK5srLgCCGsxRl7rq6kaKTSffQGMMrd/x7OHwosc/dBWYjvAOo+Eilc6j5quox1HxEMrBM/ItoeEo+qJiIieS6V+xTmAX82s6fC+XOAX0UXkohIYrNn5zqCli+VkdduN7OlBJemGnCxu/816sBEROrSeYTo1Tfy2uHuvt3MjgQqw6n2tSPd/ePowxMRkWyq75zCb8LHFUB53FQ7LyIFJt1f6vql3/w1ePVRc6Orj0RyJ92rf3T1UO5k7OojM3sulWVJ1h0XlsdYb2bXJ2nzTTNbbWarzOw3idqIiEh2JE0KZtY2PJ/QycyOMLMjw6kY+EpDGzazIuBu4HSgHzDZzPrVadMH+B7wVXfvD1zd5L9ERCKR7s1juvksvyTtPgrvZL6aIAG8y757E7YD97n7nHo3bDYKKHP308L57wG4+0/i2twKrHX3+1MNWN1HIrmj7qP8lWr3UdKrj9z958DPzew/3P2uJsTQBdgcN18FjKjTpm8Y7CtAEUES+UMT9iUiIhmQSkG8u8xsQNj3/6+1UwrbTnTXc93fCK2BPsCJwGTgfjPreMCGzKaZWbmZlVdXV6ewaxGJQro3j+nms6ZZsACKi6FVq+BxwYLo9pXKiebZwF3hNBa4FTg7hW1XAd3i5rsCWxK0+Z2717j7RuBtgiSxH3ef6+6l7l7auXPnFHYtIlHQJanZt2ABTJsGmzYFXW+bNgXzUSWGVGoffQM4CXjf3S8GBgMHp7DecqCPmfU0s4OAScDiOm0WESQazKwTQXfShhRjFxFp8WbNgp0791+2c2ewPAqpJIXP3X0vsNvMDgc+BHo1tJK77wZmAE8Da4DH3X2Vmd1kZrVHGk8DW81sNfACcK27b23KHyIi0lyl0/3zt781bnm6UimIVx72899HcDfzDuAvqWzc3ZcAS+osuyHuuQPfDicRkRantvun9td+bfcPwJQpDa/fvXuwTqLlUUjlRPMV7v6pu98LnAJcFHYjiYhIA9Lt/vnRj6Bdu/2XtWsXLI9CfQXxhtT3mru/Fk1IIiItR7rdP7VHE7NmBet07x4khFSOMpqivu6j/wof2wKlwOsEl5kOAv5MUEpbRETqkYnunylToksCdSXtPnL3se4+FtgEDAkvCR0KHA+sz054IiL5LdvdP+lK5eqjY919Ze2Mu78JlEQXkohIyzFlCsydCz16BGU+evQI5rP1y7+xUrn6aI2Z3Q88THBH8rcILjEVEZEUZLP7J12pJIWLgcuBmeH8S8A9kUUkIiI5k8oYzbuAO8JJRERasPouSX3c3b9pZis5sJAd7j4o0shERCTr6jvRXNtddBYwPsEkIpIX0q0yms0qpblW33gK74WPCa6wFRHJD+mWmUh3/XxT33Ccn5nZ9gTTZ2a2PZtBikhmFGLp6nTLTGS7SmmuJR2Os7nScJwiTVeIw2G2apX4bzaDvXujX7+5SHU4zlRuXqvd4FFm1r12Si88EZHsSFZOItUyE+mun29SGXntbDNbB2wEXgQqgf+OOC4RSaAp3T9lZcGvWgsHyK19XihdSemWmci3MhXparD7yMxeB74O/NHdjzezscBkd5+WjQDrUveRFLJ0u38KsfsIgpPF6VQZTXf95iDV7qNUkkK5u5eGyeF4d99rZn9x9+GZCrYxlBSkkCkpSFNl8pzCp2Z2GEF5iwVm9nNgd7oBikhqMtn9M3t2JiPLnkK6TyDXUjlSOBTYRTCWwhSgA7AgV2Mp60hBClkh/tKve58ABH36zbnSaHOUdveRmc0BfuPuf8p0cOlQUpBCVohJobg48SA1PXpAZWW2o8lfmeg+Wgf8l5lVmtktZtboMRTMbJyZvW1m683s+gSvTzWzajOrCKdLG7sPkUKSr90/6Uh3OEtpnPpGXvu5u48CxgAfA/PMbI2Z3WBmfRvasJkVAXcDpwP9gMlm1i9B08fcvSSc7m/anyESveZwCWdziCHbCu0+gVxr8ESzu29y91vc/XjgX4BzSW2QneHAenff4O5fAI8CE9KKViSHbrwx1xEUpkK7TyDXUrl5rY2ZjTezBQQ3ra0FJqaw7S7A5rj5qnBZXRPN7A0ze8LMuiWJYZqZlZtZeXV1dQq7FpHmJJ2rh/JtOMt8V19BvFPM7AGCL/NpwBKgt7tf4O6LUti2JVhW9xTZ/wWKw7EZ/gg8mGhD7j7X3UvdvbRz584p7FokMzJ5OWghdv3AvquHNm0KTpLXVhltbGKorAxqDVVWKiFEqb6rj14AfgM86e4fN3rDZqOAMnc/LZz/HoC7/yRJ+yLgY3fvUN92dfWR5IpuHGsaXT3UPKR69VF94ymMTTOG5UAfM+sJvAtMIjgnER/kMbXjNgBnk9q5ChHJI7p6KL+kXCW1sdx9NzADeJrgy/5xd19lZjeZ2dlhs6vMbFVYQuMqYGpU8YikqymXgxZ6MTrQ1UP5RuMpiGRJoXYf6Y7k5iHj4ymIiDSFrh7KL0nPKYhIZhXi3ci1pkxREsgXOlIQyZJCOo8g+UtJQUREYpQUREQkRklBpACkO0iNBrkpHDrRLNLC1b0ktLbMBKR28jfd9SW/6D4FkRYu3TITKlPRMug+BREB0i8zoTIVhUVJQaSFS7fMhMpUFBYlBSkY+XyfQDonetMdpEaD3BSWgkgKunJCIH9HTkt3PIJ0y0yoTEVhafEnmlWMS2rla0E6neiVTNCJ5tCsWfsnBAjmZ83KTTySXS2hdLVO9Eo2tfgjhVatEv86NAuG9pPCoSMFKWQ6UgjpygnJdzrRK9nU4pOC/kNJrXwtXa0TvZJNLb77CIKTzbNmBX2w3bsHCUH/oUSkkKTafVQQtY80wIeISGoi7T4ys3Fm9raZrTez6+tp9w0zczNrMIuJiEh0IksKZlYE3A2cDvQDJptZvwTt2gNXAX+OKhYREUlNlEcKw4H17r7B3b8AHgUmJGh3M3ArsCvCWKQFyKd7C+LpjnrJJ1EmhS7A5rj5qnBZjJkdD3Rz999HGIe0EPlYpiLdEhUi2RZlUrAEy2KXOplZK+AO4DsNbshsmpmVm1l5dXV1BkMUiZbuqJd8E2VSqAK6xc13BbbEzbcHBgBLzawSGAksTnSy2d3nunupu5d27tw5wpClucn3MhUqUSH5JsqksBzoY2Y9zewgYBKwuPZFd9/m7p3cvdjdi4FXgbPdXcOqSUxZWdDtUns7Te3zfEkKuqNe8k1kScHddwMzgKeBNcDj7r7KzG4ys7Oj2q9Ic6I76iXfRHqfgrsvcfe+7t7b3X8ULrvB3RcnaHuijhKkPrkqU5HO1UMqUSH5piDKXIg0lcbjkJZCVVJFMkBXD0mhUVKQFi+d7h9dPSSFRklBsiYXVwyle/OYrh6SQqOkIFmTizuS0+3+0dVDUmiUFKRFS7f7R1cPSaFRUpBI5fqO5Ex0/0yZEoyFvHdv8KiEIC2ZkkIKVOUy0JQv8lzfkazuH5HGUVJogKpc7pOPVUrV/SPSOLp5rQHFxUEiqKtHj6AroZCY7fvF3xRlZflTs0ikpdHNaxlS6NepZ/KcgBKCSPOnpNCAQr9OPdfnBEDndESyqaCSQlO+yHSiMrd0TkckuwoqKTTlRKlOVO6Tiyqlqj0kkl0FlRSaStepB5raZaTaQyL5o8UnhVzfPFXoVHtIJL8U1CWp6V5SKY2X7iW9Gs9AJDN0SaoXShx6AAAI+UlEQVQcIBdHR6o9JJJfCiop5Go4x+YiF3ckq/aQSH4pqKSg8wjZp0t6RfJLpEnBzMaZ2dtmtt7Mrk/w+nQzW2lmFWb2spn1izKeQpTrE+3q/hHJL5GdaDazImAtcApQBSwHJrv76rg2h7v79vD52cAV7j6uvu1mu/ZRS6IT7SKFqzmcaB4OrHf3De7+BfAoMCG+QW1CCB0K6CurHur+EpGoRZkUugCb4+arwmX7MbMrzewd4FbgqkQbMrNpZlZuZuXV1dWRBBulTNXuSfdEcVNPtKv2kEjhiDIpWIJlBxwJuPvd7t4buA74fqINuftcdy9199LOnTtnOMzUNeWXenOq3ZPv8YtI9KJMClVAt7j5rsCWeto/CpwTYTxpa8ov9XRr9+T6RLFqD4kUliiTwnKgj5n1NLODgEnA4vgGZtYnbvZMYF2E8eREujdvZaJ0tWoPiUiqIksK7r4bmAE8DawBHnf3VWZ2U3ilEcAMM1tlZhXAt4GLooqnqdL9pZ6Jm7dqv9Sh8V/qqj0kIo3i7nk1DR061HMFGr/OeefV/rbffzrvvNTWf/hh93bt9l+3XbtgeSp69Ei8/x49srN/EWkegHJP4Tu2oO5ozoUnn4SHHw5u2oLg8eGHg+WpSLdPX7WHRKQxCqpKarrSHXi+KTePtWqVeB2zoBZQQ9KtUioiLUNzuHmtxUn3ip+m3CeQbp++ag+JSGMoKWRRLsaIVvePiDRG61wHIPWr/fKeNSs4D9C9e5AQGvOlPmWKkoCIpEZJIQ/oS11EskXdRyIiEqOkICIiMUoKIiISo6QgIiIxSgoiIhKTd3c0m1k1kOAe3WahE/BRroOoh+JLT3OPD5p/jIovPenE18PdGxyQJu+SQnNmZuWp3EaeK4ovPc09Pmj+MSq+9GQjPnUfiYhIjJKCiIjEKClk1txcB9AAxZee5h4fNP8YFV96Io9P5xRERCRGRwoiIhKjpNBIZtbNzF4wszXh+NIzE7Q50cy2mVlFON2Q5RgrzWxluO8DRiSywJ1mtt7M3jCzIVmM7Z/i3pcKM9tuZlfXaZP198/MHjCzD83szbhlR5rZs2a2Lnw8Ism6F4Vt1plZxscZTxLbz8zsrfDf7ykz65hk3Xo/CxHHWGZm78b9O56RZN1xZvZ2+Hm8PovxPRYXW2U4VnyidSN9D5N9p+Ts85fKmJ2a9k3AMcCQ8Hl7YC3Qr06bE4Hf5zDGSqBTPa+fAfw3YMBI4M85irMIeJ/g+umcvn/ACcAQ4M24ZbcC14fPrwduSbDekcCG8PGI8PkRWYjtVKB1+PyWRLGl8lmIOMYy4LspfAbeAXoBBwGv1/3/FFV8dV7/L+CGXLyHyb5TcvX505FCI7n7e+7+Wvj8M2AN0CW3UTXaBOAhD7wKdDSzY3IQx0nAO+6e85sR3f0l4OM6iycAD4bPHwTOSbDqacCz7v6xu38CPAuMizo2d3/G3XeHs68CXTO5z8ZK8v6lYjiw3t03uPsXwKME73tG1RefmRnwTeCRTO83FfV8p+Tk86ekkAYzKwaOB/6c4OVRZva6mf23mfXPamDgwDNmtsLMpiV4vQuwOW6+itwktkkk/4+Yy/ev1tHu/h4E/3GBoxK0aQ7v5SUER36JNPRZiNqMsIvrgSTdH83h/fsa8IG7r0vyetbewzrfKTn5/CkpNJGZHQY8CVzt7tvrvPwaQZfIYOAuYFGWw/uquw8BTgeuNLMT6rxuCdbJ6mVoZnYQcDawMMHLuX7/GiOn76WZzQJ2AwuSNGnosxCle4DeQAnwHkEXTV05/ywCk6n/KCEr72ED3ylJV0uwLK33T0mhCcysDcE/3gJ3/23d1919u7vvCJ8vAdqYWadsxefuW8LHD4GnCA7R41UB3eLmuwJbshNdzOnAa+7+Qd0Xcv3+xfmgtlstfPwwQZucvZfhScWzgCkedjDXlcJnITLu/oG773H3vcB9Sfad08+imbUGzgMeS9YmG+9hku+UnHz+lBQaKex//BWwxt1vT9Lmy2E7zGw4wfu8NUvxHWpm7WufE5yQfLNOs8XAv4ZXIY0EttUepmZR0l9nuXz/6lgM1F7NcRHwuwRtngZONbMjwu6RU8NlkTKzccB1wNnuvjNJm1Q+C1HGGH+e6twk+14O9DGznuHR4ySC9z1bTgbecveqRC9m4z2s5zslN5+/qM6ot9QJGE1wePYGUBFOZwDTgelhmxnAKoIrKV4F/jmL8fUK9/t6GMOscHl8fAbcTXDVx0qgNMvvYTuCL/kOccty+v4RJKj3gBqCX1//BnwJeA5YFz4eGbYtBe6PW/cSYH04XZyl2NYT9CXXfgbvDdt+BVhS32chi+/fr8PP1xsEX3DH1I0xnD+D4Iqbd6KKMVF84fL5tZ+7uLZZfQ/r+U7JyedPdzSLiEiMuo9ERCRGSUFERGKUFEREJEZJQUREYpQUREQkRklBJGRme2z/Cq4Zq9hpZsXxFTpFmqvWuQ5ApBn53N1Lch2ESC7pSEGkAWE9/VvM7C/h9L/C5T3M7Lmw4NtzZtY9XH60BWMcvB5O/xxuqsjM7gtr5j9jZoeE7a8ys9Xhdh7N0Z8pAigpiMQ7pE730QVxr2139+HAHOD/hMvmEJQgH0RQkO7OcPmdwIseFPQbQnAnLEAf4G537w98CkwMl18PHB9uZ3pUf5xIKnRHs0jIzHa4+2EJllcCX3f3DWHhsvfd/Utm9hFB6YaacPl77t7JzKqBru7+j7htFBPUve8Tzl8HtHH3H5rZH4AdBNVgF3lYDFAkF3SkIJIaT/I8WZtE/hH3fA/7zumdSVCLaiiwIqzcKZITSgoiqbkg7nFZ+PxPBFU9AaYAL4fPnwMuBzCzIjM7PNlGzawV0M3dXwD+N9AROOBoRSRb9ItEZJ9DbP/B2//g7rWXpR5sZn8m+CE1OVx2FfCAmV0LVAMXh8tnAnPN7N8IjgguJ6jQmUgR8LCZdSCoXnuHu3+asb9IpJF0TkGkAeE5hVJ3/yjXsYhETd1HIiISoyMFERGJ0ZGCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIzP8H4gpm0TYcnbsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20ad1a5ae10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dpt_model_val_loss = dpt_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Again, a clear improvement over the reference network.\n",
    "\n",
    "To recap: here the most common ways to prevent overfitting in neural networks:\n",
    "\n",
    "* Getting more training data.\n",
    "* Reducing the capacity of the network.\n",
    "* Adding weight regularization.\n",
    "* Adding dropout.\n",
    "\n",
    "我们再次看到，这种方法的性能相比参考网络有明显提高。\n",
    "\n",
    "总结一下，防止神经网络过拟合的常用方法包括：\n",
    "* 获取更多的训练数据\n",
    "* 减小网络容量\n",
    "* 添加权重正则化\n",
    "* 添加 dropout\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}