deep-learning-with-python-notebooks/6.1-using-word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文代码作者：François Chollet\n",
    "\n",
    "github：https://github.com/fchollet/deep-learning-with-python-notebooks\n",
    "\n",
    "中文注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：keras 2.2.1（原文是2.0.8，运行结果一致），tensorflow 1.8，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](data/gongzhong.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using word embeddings\n",
    "# 使用词嵌入\n",
    "\n",
    "This notebook contains the second code sample found in Chapter 6, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Another popular and powerful way to associate a vector with a word is the use of dense \"word vectors\", also called \"word embeddings\". \n",
    "While the vectors obtained through one-hot encoding are binary, sparse (mostly made of zeros) and very high-dimensional (same dimensionality as the \n",
    "number of words in the vocabulary), \"word embeddings\" are low-dimensional floating point vectors \n",
    "(i.e. \"dense\" vectors, as opposed to sparse vectors). \n",
    "Unlike word vectors obtained via one-hot encoding, word embeddings are learned from data. \n",
    "It is common to see word embeddings that are 256-dimensional, 512-dimensional, or 1024-dimensional when dealing with very large vocabularies. \n",
    "On the other hand, one-hot encoding words generally leads to vectors that are 20,000-dimensional or higher (capturing a vocabulary of 20,000 \n",
    "token in this case). So, word embeddings pack more information into far fewer dimensions. \n",
    " \n",
    "\n",
    "将单词与向量相关联还有另一种常用的强大方法，就是使用密集的词向量（word vector），也叫词嵌入（word embedding）。one-hot 编码得到的向量是二进制的、稀疏的（绝大部分元素都是 0）、维度很高的（维度大小等于词表中的单词个数），而词嵌入是低维的浮点数向量（即密集向量，与稀疏向量相对），参见下图。与 one-hot 编码得到的词向量不同，词嵌入是从数据中学习得到的。常见的词向量维度是 256、512 或 1024（处理非常大的词表时）。与此相对，one-hot 编码的词向量维度通常为 20 000 或更高（对应包含 20 000 个标记的词表）。因此，词向量可以将更多的信息塞入更低的维度中。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![word embeddings vs. one hot encoding](data/word_embeddings.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to obtain word embeddings:\n",
    "\n",
    "* Learn word embeddings jointly with the main task you care about (e.g. document classification or sentiment prediction). \n",
    "In this setup, you would start with random word vectors, then learn your word vectors in the same way that you learn the weights of a neural network.\n",
    "* Load into your model word embeddings that were pre-computed using a different machine learning task than the one you are trying to solve. \n",
    "These are called \"pre-trained word embeddings\". \n",
    "\n",
    "Let's take a look at both.\n",
    "\n",
    "获取词嵌入有两种方法。\n",
    "* 在完成主任务（比如文档分类或情感预测）的同时学习词嵌入。在这种情况下，一开始是随机的词向量，然后对这些词向量进行学习，其学习方式与学习神经网络的权重相同。\n",
    " \n",
    "* 在不同于待解决问题的机器学习任务上预计算好词嵌入，然后将其加载到模型中。这些词嵌入叫作预训练词嵌入（pretrained word embedding）。\n",
    "\n",
    "我们来分别看一下这两种方法。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning word embeddings with the `Embedding` layer\n",
    "\n",
    "\n",
    "The simplest way to associate a dense vector to a word would be to pick the vector at random. The problem with this approach is that the \n",
    "resulting embedding space would have no structure: for instance, the words \"accurate\" and \"exact\" may end up with completely different \n",
    "embeddings, even though they are interchangeable in most sentences. It would be very difficult for a deep neural network to make sense of \n",
    "such a noisy, unstructured embedding space. \n",
    "\n",
    "To get a bit more abstract: the geometric relationships between word vectors should reflect the semantic relationships between these words. \n",
    "Word embeddings are meant to map human language into a geometric space. For instance, in a reasonable embedding space, we would expect \n",
    "synonyms to be embedded into similar word vectors, and in general we would expect the geometric distance (e.g. L2 distance) between any two \n",
    "word vectors to relate to the semantic distance of the associated words (words meaning very different things would be embedded to points \n",
    "far away from each other, while related words would be closer). Even beyond mere distance, we may want specific __directions__ in the \n",
    "embedding space to be meaningful. \n",
    "\n",
    "\n",
    "[...]\n",
    "\n",
    "\n",
    "In real-world word embedding spaces, common examples of meaningful geometric transformations are \"gender vectors\" and \"plural vector\". For \n",
    "instance, by adding a \"female vector\" to the vector \"king\", one obtain the vector \"queen\". By adding a \"plural vector\", one obtain \"kings\". \n",
    "Word embedding spaces typically feature thousands of such interpretable and potentially useful vectors.\n",
    "\n",
    "Is there some \"ideal\" word embedding space that would perfectly map human language and could be used for any natural language processing \n",
    "task? Possibly, but in any case, we have yet to compute anything of the sort. Also, there isn't such a thing as \"human language\", there are \n",
    "many different languages and they are not isomorphic, as a language is the reflection of a specific culture and a specific context. But more \n",
    "pragmatically, what makes a good word embedding space depends heavily on your task: the perfect word embedding space for an \n",
    "English-language movie review sentiment analysis model may look very different from the perfect embedding space for an English-language \n",
    "legal document classification model, because the importance of certain semantic relationships varies from task to task.\n",
    "\n",
    "It is thus reasonable to __learn__ a new embedding space with every new task. Thankfully, backpropagation makes this really easy, and Keras makes it \n",
    "even easier. It's just about learning the weights of a layer: the `Embedding` layer.\n",
    "\n",
    "## 利用 Embedding 层学习词嵌入\n",
    "要将一个词与一个密集向量相关联，最简单的方法就是随机选择向量。这种方法的问题在于，得到的嵌入空间没有任何结构。例如，accurate 和 exact 两个词的嵌入可能完全不同，尽管它们在大多数句子里都是可以互换的 a。深度神经网络很难对这种杂乱的、非结构化的嵌入空间进行学习。\n",
    "\n",
    "说得更抽象一点，词向量之间的几何关系应该表示这些词之间的语义关系。词嵌入的作用应该是将人类的语言映射到几何空间中。例如，在一个合理的嵌入空间中，同义词应该被嵌入到相似的词向量中，一般来说，任意两个词向量之间的几何距离（比如 L2 距离）应该和这两个词的语义距离有关（表示不同事物的词被嵌入到相隔很远的点，而相关的词则更加靠近）。除了距离，你可能还希望嵌入空间中的特定方向也是有意义的。\n",
    "在真实的词嵌入空间中，常见的有意义的几何变换的例子包括“性别”向量和“复数”向量。 例如，将 king（国王）向量加上 female（女性）向量，得到的是 queen（女王）向量。将 king（国王） 向量加上 plural（复数）向量，得到的是 kings 向量。词嵌入空间通常具有几千个这种可解释的、 并且可能很有用的向量。\n",
    "\n",
    "[...]\n",
    "\n",
    "有没有一个理想的词嵌入空间，可以完美地映射人类语言，并可用于所有自然语言处理任 务？可能有，但我们尚未发现。此外，也不存在人类语言（human language）这种东西。世界上有许多种不同的语言，而且它们不是同构的，因为语言是特定文化和特定环境的反射。但从更实际的角度来说，一个好的词嵌入空间在很大程度上取决于你的任务。英语电影评论情感分析 模型的完美词嵌入空间，可能不同于英语法律文档分类模型的完美词嵌入空间，因为某些语义关系的重要性因任务而异。\n",
    "\n",
    "因此，合理的做法是对每个新任务都学习一个新的嵌入空间。幸运的是，反向传播让这种学习变得很简单，而 Keras 使其变得更简单。我们要做的就是学习一个层的权重，这个层就是 Embedding 层。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "\n",
    "# The Embedding layer takes at least two arguments:\n",
    "# the number of possible tokens, here 1000 (1 + maximum word index),\n",
    "# and the dimensionality of the embeddings, here 64.\n",
    "# Embedding 层至少需要两个参数： \n",
    "# 标记的个数（这里是 1000，即最大单词索引 +1）和嵌入的维度（这里是 64）\n",
    "embedding_layer = Embedding(1000, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The `Embedding` layer is best understood as a dictionary mapping integer indices (which stand for specific words) to dense vectors. It takes \n",
    "as input integers, it looks up these integers into an internal dictionary, and it returns the associated vectors. It's effectively a dictionary lookup.\n",
    "\n",
    "最好将 Embedding 层理解为一个字典，将整数索引（表示特定单词）映射为密集向量。它接收整数作为输入，并在内部字典中查找这些整数，然后返回相关联的向量。Embedding 层实际上是一种字典查找。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The `Embedding` layer takes as input a 2D tensor of integers, of shape `(samples, sequence_length)`, where each entry is a sequence of \n",
    "integers. It can embed sequences of variable lengths, so for instance we could feed into our embedding layer above batches that could have \n",
    "shapes `(32, 10)` (batch of 32 sequences of length 10) or `(64, 15)` (batch of 64 sequences of length 15). All sequences in a batch must \n",
    "have the same length, though (since we need to pack them into a single tensor), so sequences that are shorter than others should be padded \n",
    "with zeros, and sequences that are longer should be truncated.\n",
    "\n",
    "This layer returns a 3D floating point tensor, of shape `(samples, sequence_length, embedding_dimensionality)`. Such a 3D tensor can then \n",
    "be processed by a RNN layer or a 1D convolution layer (both will be introduced in the next sections).\n",
    "\n",
    "When you instantiate an `Embedding` layer, its weights (its internal dictionary of token vectors) are initially random, just like with any \n",
    "other layer. During training, these word vectors will be gradually adjusted via backpropagation, structuring the space into something that the \n",
    "downstream model can exploit. Once fully trained, your embedding space will show a lot of structure -- a kind of structure specialized for \n",
    "the specific problem you were training your model for.\n",
    "\n",
    "Let's apply this idea to the IMDB movie review sentiment prediction task that you are already familiar with. Let's quickly prepare \n",
    "the data. We will restrict the movie reviews to the top 10,000 most common words (like we did the first time we worked with this dataset), \n",
    "and cut the reviews after only 20 words. Our network will simply learn 8-dimensional embeddings for each of the 10,000 words, turn the \n",
    "input integer sequences (2D integer tensor) into embedded sequences (3D float tensor), flatten the tensor to 2D, and train a single `Dense` \n",
    "layer on top for classification.\n",
    "\n",
    "Embedding 层的输入是一个二维整数张量，其形状为 (samples, sequence_length)，每个元素是一个整数序列。它能够嵌入长度可变的序列，例如，对于前一个例子中的Embedding 层，你可以输入形状为 (32, 10)（32 个长度为 10 的序列组成的批量）或 (64,15)（64 个长度为 15 的序列组成的批量）的批量。不过一批数据中的所有序列必须具有相同的 长度（因为需要将它们打包成一个张量），所以较短的序列应该用 0 填充，较长的序列应该被截断。\n",
    "\n",
    "这个Embedding 层返回一个形状为 (samples, sequence_length, embedding_dimensionality) 的三维浮点数张量。然后可以用 RNN 层或一维卷积层来处理这个三维张量（二者都会在后面介绍）。\n",
    "\n",
    "将一个Embedding 层实例化时，它的权重（即标记向量的内部字典）最开始是随机的，与其他层一样。在训练过程中，利用反向传播来逐渐调节这些词向量，改变空间结构以便下游模型可以利用。一旦训练完成，嵌入空间将会展示大量结构，这种结构专门针对训练模型所要解决的问题。\n",
    "\n",
    "我们将这个想法应用于你熟悉的 IMDB 电影评论情感预测任务。首先，我们需要快速准备数据。将电影评论限制为前 10 000 个最常见的单词（第一次处理这个数据集时就是这么做的）， 然后将评论长度限制为只有 20 个单词。对于这 10 000 个单词，网络将对每个词都学习一个 8 维嵌入，将输入的整数序列（二维整数张量）转换为嵌入序列（三维浮点数张量），然后将这个张量展平为二维，最后在上面训练一个Dense 层用于分类。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras import preprocessing\n",
    "\n",
    "# Number of words to consider as features\n",
    "# 作为特征的单词个数\n",
    "max_features = 10000\n",
    "# Cut texts after this number of words \n",
    "# (among top max_features most common words)\n",
    "# 在这么多单词后截断文本（这些单词都属于前 max_features 个最常见的单词）\n",
    "maxlen = 20\n",
    "\n",
    "# Load the data as lists of integers.（将数据加载为整数列表）\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# This turns our lists of integers\n",
    "# into a 2D integer tensor of shape `(samples, maxlen)`\n",
    "# 将整数列表转换成形状为 (samples, maxlen) 的二维整数张量\n",
    "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 20, 8)             80000     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 161       \n",
      "=================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 5s 237us/step - loss: 0.6759 - acc: 0.6050 - val_loss: 0.6398 - val_acc: 0.6814\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 3s 126us/step - loss: 0.5657 - acc: 0.7427 - val_loss: 0.5467 - val_acc: 0.7206\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 3s 126us/step - loss: 0.4752 - acc: 0.7808 - val_loss: 0.5113 - val_acc: 0.7384\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 3s 128us/step - loss: 0.4263 - acc: 0.8077 - val_loss: 0.5008 - val_acc: 0.7452\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 2s 122us/step - loss: 0.3930 - acc: 0.8258 - val_loss: 0.4981 - val_acc: 0.7538\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 2s 124us/step - loss: 0.3668 - acc: 0.8395 - val_loss: 0.5014 - val_acc: 0.7530\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 3s 127us/step - loss: 0.3435 - acc: 0.8533 - val_loss: 0.5052 - val_acc: 0.7520\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 2s 115us/step - loss: 0.3223 - acc: 0.8657 - val_loss: 0.5132 - val_acc: 0.7486\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 3s 126us/step - loss: 0.3022 - acc: 0.8766 - val_loss: 0.5213 - val_acc: 0.7490\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 2s 119us/step - loss: 0.2839 - acc: 0.8860 - val_loss: 0.5303 - val_acc: 0.7466\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "# We specify the maximum input length to our Embedding layer\n",
    "# so we can later flatten the embedded inputs\n",
    "# 指定 Embedding 层的最大输入长度，以便后面将嵌入输入展平。\n",
    "model.add(Embedding(10000, 8, input_length=maxlen))\n",
    "# After the Embedding layer, \n",
    "# our activations have shape `(samples, maxlen, 8)`.\n",
    "# Embedding 层 激活的形状为 (samples, maxlen, 8)\n",
    "# We flatten the 3D tensor of embeddings \n",
    "# into a 2D tensor of shape `(samples, maxlen * 8)`\n",
    "# 将三维的嵌入张量展平成形状为 (samples, maxlen * 8) 的二维张量\n",
    "model.add(Flatten())\n",
    "\n",
    "# We add the classifier on top\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get to a validation accuracy of ~76%, which is pretty good considering that we only look at the first 20 words in every review. But \n",
    "note that merely flattening the embedded sequences and training a single `Dense` layer on top leads to a model that treats each word in the \n",
    "input sequence separately, without considering inter-word relationships and structure sentence (e.g. it would likely treat both _\"this movie \n",
    "is shit\"_ and _\"this movie is the shit\"_ as being negative \"reviews\"). It would be much better to add recurrent layers or 1D convolutional \n",
    "layers on top of the embedded sequences to learn features that take into account each sequence as a whole. That's what we will focus on in \n",
    "the next few sections.\n",
    "\n",
    "得到的验证精度约为 76%，考虑到仅查看每条评论的前 20 个单词，这个结果还是相当不错的。但请注意，仅仅将嵌入序列展开并在上面训练一个 Dense 层，会导致模型对输入序列中的每个单词单独处理，而没有考虑单词之间的关系和句子结构（举个例子，这个模型可能会将 this movie is a bomb 和 this movie is the bomb 两条都归为负面评论）。更好的做法是在嵌入序列上添加循环层或一维卷积层，将每个序列作为整体来学习特征。这也是接下来几节的重点。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pre-trained word embeddings\n",
    "\n",
    "\n",
    "Sometimes, you have so little training data available that could never use your data alone to learn an appropriate task-specific embedding \n",
    "of your vocabulary. What to do then?\n",
    "\n",
    "Instead of learning word embeddings jointly with the problem you want to solve, you could be loading embedding vectors from a pre-computed \n",
    "embedding space known to be highly structured and to exhibit useful properties -- that captures generic aspects of language structure. The \n",
    "rationale behind using pre-trained word embeddings in natural language processing is very much the same as for using pre-trained convnets \n",
    "in image classification: we don't have enough data available to learn truly powerful features on our own, but we expect the features that \n",
    "we need to be fairly generic, i.e. common visual features or semantic features. In this case it makes sense to reuse features learned on a \n",
    "different problem.\n",
    "\n",
    "Such word embeddings are generally computed using word occurrence statistics (observations about what words co-occur in sentences or \n",
    "documents), using a variety of techniques, some involving neural networks, others not. The idea of a dense, low-dimensional embedding space \n",
    "for words, computed in an unsupervised way, was initially explored by Bengio et al. in the early 2000s, but it only started really taking \n",
    "off in research and industry applications after the release of one of the most famous and successful word embedding scheme: the Word2Vec \n",
    "algorithm, developed by Mikolov at Google in 2013. Word2Vec dimensions capture specific semantic properties, e.g. gender.\n",
    "\n",
    "There are various pre-computed databases of word embeddings that can download and start using in a Keras `Embedding` layer. Word2Vec is one \n",
    "of them. Another popular one is called \"GloVe\", developed by Stanford researchers in 2014. It stands for \"Global Vectors for Word \n",
    "Representation\", and it is an embedding technique based on factorizing a matrix of word co-occurrence statistics. Its developers have made \n",
    "available pre-computed embeddings for millions of English tokens, obtained from Wikipedia data or from Common Crawl data.\n",
    "\n",
    "Let's take a look at how you can get started using GloVe embeddings in a Keras model. The same method will of course be valid for Word2Vec \n",
    "embeddings or any other word embedding database that you can download. We will also use this example to refresh the text tokenization \n",
    "techniques we introduced a few paragraphs ago: we will start from raw text, and work our way up.\n",
    "\n",
    "## 使用预训练的词嵌入\n",
    "有时可用的训练数据很少，以至于只用手头数据无法学习适合特定任务的词嵌入。那么应该怎么办？\n",
    "\n",
    "你可以从预计算的嵌入空间中加载嵌入向量（你知道这个嵌入空间是高度结构化的，并且具有有用的属性，即抓住了语言结构的一般特点），而不是在解决问题的同时学习词嵌入。在自然语言处理中使用预训练的词嵌入，其背后的原理与在图像分类中使用预训练的卷积神经网络是一样的：没有足够的数据来自己学习真正强大的特征，但你需要的特征应该是非常通用的，比如常见的视觉特征或语义特征。在这种情况下，重复使用在其他问题上学到的特征，这种做法是有道理的。\n",
    "\n",
    "这种词嵌入通常是利用词频统计计算得出的（观察哪些词共同出现在句子或文档中），用到的技术很多，有些涉及神经网络，有些则不涉及。Bengio 等人在 21 世纪初首先研究了一种思路，就是用无监督的方法计算一个密集的低维词嵌入空间，但直到最有名且最成功的词嵌入方案之一 word2vec 算法发布之后，这一思路才开始在研究领域和工业应用中取得成功。word2vec 算法由 Google 的 Tomas Mikolov 于 2013 年开发，其维度抓住了特定的语义属性，比如性别。\n",
    "\n",
    "有许多预计算的词嵌入数据库，你都可以下载并在 Keras 的 Embedding 层中使用。word2vec 就是其中之一。另一个常用的是 GloVe（global vectors for word representation，词表示全局向量），由斯坦福大学的研究人员于 2014 年开发。这种嵌入方法基于对词共现统计矩阵进行因式分解。其开发者已经公开了数百万个英文标记的预计算嵌入，它们都是从维基百科数据和 Common Crawl 数据得到的。\n",
    "\n",
    "我们来看一下如何在 Keras 模型中使用 GloVe 嵌入。同样的方法也适用于 word2vec 嵌入或其他词嵌入数据库。这个例子还可以改进前面刚刚介绍过的文本分词技术，即从原始文本开始，一步步进行处理。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together: from raw text to word embeddings\n",
    "\n",
    "\n",
    "We will be using a model similar to the one we just went over -- embedding sentences in sequences of vectors, flattening them and training a \n",
    "`Dense` layer on top. But we will do it using pre-trained word embeddings, and instead of using the pre-tokenized IMDB data packaged in \n",
    "Keras, we will start from scratch, by downloading the original text data.\n",
    "\n",
    "## 整合在一起：从原始文本到词嵌入\n",
    "\n",
    "本节的模型与之前刚刚见过的那个类似：将句子嵌入到向量序列中，然后将其展平，最后在上面训练一个 Dense 层。但此处将使用预训练的词嵌入。此外，我们将从头开始，先下载IMDB 原始文本数据，而不是使用 Keras 内置的已经预先分词的 IMDB 数据。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the IMDB data as raw text\n",
    "\n",
    "\n",
    "First, head to `http://ai.stanford.edu/~amaas/data/sentiment/` and download the raw IMDB dataset (if the URL isn't working anymore, just \n",
    "Google \"IMDB dataset\"). Uncompress it.\n",
    "\n",
    "Now let's collect the individual training reviews into a list of strings, one string per review, and let's also collect the review labels \n",
    "(positive / negative) into a `labels` list:\n",
    "\n",
    "### 下载 IMDB 数据的原始文本\n",
    "首先，打开 `http://ai.stanford.edu/~amaas/data/sentiment/` ，下载原始 IMDB 数据集并解压。 接下来，我们将训练评论转换成字符串列表，每个字符串对应一条评论。你也可以将评论\n",
    "标签（正面 / 负面）转换成 labels 列表。\n",
    "\n",
    "注：这一步我们已经做好了，不需要再下载，数据已经在 'data/aclImdb'目录。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir = 'data/aclImdb'\n",
    "train_dir = os.path.join(imdb_dir, 'train')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname),encoding='utf8')\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize the data\n",
    "\n",
    "\n",
    "Let's vectorize the texts we collected, and prepare a training and validation split.\n",
    "We will merely be using the concepts we introduced earlier in this section.\n",
    "\n",
    "Because pre-trained word embeddings are meant to be particularly useful on problems where little training data is available (otherwise, \n",
    "task-specific embeddings are likely to outperform them), we will add the following twist: we restrict the training data to its first 200 \n",
    "samples. So we will be learning to classify movie reviews after looking at just 200 examples...\n",
    "\n",
    "### 对数据进行分词\n",
    "利用本节前面介绍过的概念，我们对文本进行分词，并将其划分为训练集和验证集。\n",
    "\n",
    "因为预训练的词嵌入对训练数据很少的问题特别有用（否则，针对于具体任务的嵌入可能效果更好），所以我们又添加了以下限制：将训练数据限定为前 200 个样本。因此，你需要在读取 200 个样本之后学习对电影评论进行分类。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88582 unique tokens.\n",
      "Shape of data tensor: (25000, 100)\n",
      "Shape of label tensor: (25000,)\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen = 100  # We will cut reviews after 100 words（在 100 个单词后截断评论）\n",
    "training_samples = 200  # We will be training on 200 samples（在 200 个样本上训练）\n",
    "validation_samples = 10000  # We will be validating on 10000 samples（在 10 000 个样本上验证）\n",
    "max_words = 10000  # We will only consider the top 10,000 words in the dataset（只考虑数据集中前 10 000 个最常见的单词）\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=maxlen)\n",
    "\n",
    "labels = np.asarray(labels)\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "# Split the data into a training set and a validation set\n",
    "# But first, shuffle the data, since we started from data\n",
    "# where sample are ordered (all negative first, then all positive).\n",
    "# 将数据划分为训练集和验证集，但首先要打乱数据，因为一开始数据中的样本是排好序的（所有负面评论都在前面， 然后是所有正面评论）\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "x_train = data[:training_samples]\n",
    "y_train = labels[:training_samples]\n",
    "x_val = data[training_samples: training_samples + validation_samples]\n",
    "y_val = labels[training_samples: training_samples + validation_samples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the GloVe word embeddings\n",
    "\n",
    "\n",
    "Head to `https://nlp.stanford.edu/projects/glove/` (where you can learn more about the GloVe algorithm), and download the pre-computed \n",
    "embeddings from 2014 English Wikipedia. It's a 822MB zip file named `glove.6B.zip`, containing 100-dimensional embedding vectors for \n",
    "400,000 words (or non-word tokens). Un-zip it.\n",
    "\n",
    "### 下载 GloVe 词嵌入\n",
    "打开 https://nlp.stanford.edu/projects/glove ，下载 2014 年英文维基百科的预计算嵌入。这是 一个 822 MB 的压缩文件，文件名是 glove.6B.zip，里面包含 400 000 个单词（或非单词的标记） 的100 维嵌入向量。解压文件。\n",
    "\n",
    "备注：我们下载了，文件放在 'data/glove.6B.100d.txt'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-process the embeddings\n",
    "\n",
    "\n",
    "Let's parse the un-zipped file (it's a `txt` file) to build an index mapping words (as strings) to their vector representation (as number \n",
    "vectors).\n",
    "\n",
    "### 对嵌入进行预处理\n",
    "我们对解压后的文件（一个 .txt 文件）进行解析，构建一个将单词（字符串）映射为其向量表示（数值向量）的索引。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "glove_dir = 'data/'\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'),encoding='utf8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now let's build an embedding matrix that we will be able to load into an `Embedding` layer. It must be a matrix of shape `(max_words, \n",
    "embedding_dim)`, where each entry `i` contains the `embedding_dim`-dimensional vector for the word of index `i` in our reference word index \n",
    "(built during tokenization). Note that the index `0` is not supposed to stand for any word or token -- it's a placeholder.\n",
    "\n",
    "接下来，需要构建一个可以加载到 Embedding 层中的嵌入矩阵。它必须是一个形状为(max_words, embedding_dim) 的矩阵，对于单词索引（在分词时构建）中索引为 i 的单词， 这个矩阵的元素 i 就是这个单词对应的 embedding_dim 维向量。注意，索引 0 不应该代表任何单词或标记，它只是一个占位符。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dim = 100\n",
    "\n",
    "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if i < max_words:\n",
    "        if embedding_vector is not None:\n",
    "            # Words not found in embedding index will be all-zeros.\n",
    "            # 嵌入索引（embeddings_index） 中找不到的词，其嵌入向量全为 0\n",
    "            embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a model\n",
    "\n",
    "We will be using the same model architecture as before:\n",
    "\n",
    "### 定义模型\n",
    "我们将使用与前面相同的模型架构:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the GloVe embeddings in the model\n",
    "\n",
    "\n",
    "The `Embedding` layer has a single weight matrix: a 2D float matrix where each entry `i` is the word vector meant to be associated with \n",
    "index `i`. Simple enough. Let's just load the GloVe matrix we prepared into our `Embedding` layer, the first layer in our model:\n",
    "\n",
    "### 在模型中加载 GloVe 嵌入\n",
    "Embedding 层只有一个权重矩阵，是一个二维的浮点数矩阵，其中每个元素 i 是与索引 i 相关联的词向量。够简单。将准备好的 GloVe 矩阵加载到 Embedding 层中，即模型的第一层。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix])\n",
    "model.layers[0].trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Additionally, we freeze the embedding layer (we set its `trainable` attribute to `False`), following the same rationale as what you are \n",
    "already familiar with in the context of pre-trained convnet features: when parts of a model are pre-trained (like our `Embedding` layer), \n",
    "and parts are randomly initialized (like our classifier), the pre-trained parts should not be updated during training to avoid forgetting \n",
    "what they already know. The large gradient update triggered by the randomly initialized layers would be very disruptive to the already \n",
    "learned features.\n",
    "\n",
    "此外，需要冻结 Embedding 层（即将其 trainable 属性设为 False），其原理和预训练的卷积神经网络特征相同，你已经很熟悉了。如果一个模型的一部分是经过预训练的（如 Embedding 层），而另一部分是随机初始化的（如分类器），那么在训练期间不应该更新预训练的部分，以避免丢失它们所保存的信息。随机初始化的层会引起较大的梯度更新，会破坏已经学到的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate\n",
    "\n",
    "Let's compile our model and train it:\n",
    "\n",
    "### 训练模型与评估模型\n",
    "编译并训练模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s 4ms/step - loss: 1.6337 - acc: 0.5250 - val_loss: 0.7130 - val_acc: 0.5100\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.7565 - acc: 0.5800 - val_loss: 0.6910 - val_acc: 0.5418\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.5956 - acc: 0.6950 - val_loss: 1.1205 - val_acc: 0.4936\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.5335 - acc: 0.7350 - val_loss: 0.7134 - val_acc: 0.5362\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.4713 - acc: 0.8100 - val_loss: 0.7177 - val_acc: 0.5589\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 0s 2ms/step - loss: 0.1448 - acc: 0.9800 - val_loss: 1.3373 - val_acc: 0.4952\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.2545 - acc: 0.8800 - val_loss: 1.3110 - val_acc: 0.4960\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.1102 - acc: 0.9800 - val_loss: 0.8168 - val_acc: 0.5558\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0760 - acc: 0.9800 - val_loss: 1.5204 - val_acc: 0.5115\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0680 - acc: 0.9850 - val_loss: 0.7458 - val_acc: 0.5759\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))\n",
    "model.save_weights('pre_trained_glove_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot its performance over time:\n",
    "\t\n",
    "    绘制结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX9//HXh80QFlHAqiAE9yVsMaJUFBSLaF3qDmK/daXuW/1+i0uLxWL9udW1KrXWhQhFLRWtaF1wqxugggJVUAEDFCMighE0yef3x5mEIWSZhEnu5Ob9fDzmMXPvnLn3M3dmPvfcc889Y+6OiIjES4uoAxARkfRTchcRiSEldxGRGFJyFxGJISV3EZEYUnIXEYkhJfcYM7OWZrbOzHqks2yUzGxXM0t7/10zO8zMFidNf2RmB6VSth7rut/Mrqrv60VS0SrqAGQjM1uXNJkNbABKE9O/dPeCuizP3UuB9uku2xy4+x7pWI6ZnQ2c5u5DkpZ9djqWLVITJfcM4u4VyTVRMzzb3V+orryZtXL3ksaITaQ2+j5mFjXLNCFm9nsz+5uZTTKztcBpZjbQzN4ys6/NbIWZ3WFmrRPlW5mZm1lOYnpi4vnpZrbWzN40s151LZt4/ggz+9jM1pjZnWb2bzM7vZq4U4nxl2a2yMxWm9kdSa9taWZ/NLNVZvYJMLyG7XONmU2uNO9uM7s18fhsM1uQeD+fJGrV1S2r0MyGJB5nm9kjidjmAftWsd5PE8udZ2bHJOb3Bu4CDko0eX2ZtG2vTXr9uYn3vsrM/mFmO6SybeqyncvjMbMXzOwrM/uvmf1f0np+k9gm35jZLDPbsaomMDN7vfxzTmzPVxPr+Qq4xsx2M7MZiffyZWK7bZ30+p6J91iUeP52M8tKxLxXUrkdzKzYzDpX936lFu6uWwbegMXAYZXm/R74HjiasGNuC+wH7E84CtsZ+Bi4MFG+FeBATmJ6IvAlkA+0Bv4GTKxH2e2AtcCxiecuB34ATq/mvaQS45PA1kAO8FX5ewcuBOYB3YHOwKvha1vlenYG1gHtkpb9BZCfmD46UcaAQ4HvgD6J5w4DFictqxAYknh8M/AysA3QE5hfqezJwA6Jz+TURAw/Sjx3NvBypTgnAtcmHg9LxNgPyAL+BLyUyrap43beGlgJXAJsBXQEBiSeuxKYA+yWeA/9gG2BXStva+D18s858d5KgPOAloTv4+7AUKBN4nvyb+DmpPfzYWJ7tkuUPzDx3ARgfNJ6fgVMjfp32JRvkQegWzUfTPXJ/aVaXncF8FjicVUJ+96ksscAH9aj7JnAa0nPGbCCapJ7ijEekPT834ErEo9fJTRPlT93ZOWEU2nZbwGnJh4fAXxcQ9mngQsSj2tK7kuTPwvg/OSyVSz3Q+Cnice1JfeHgOuTnutIOM/SvbZtU8ft/HNgVjXlPimPt9L8VJL7p7XEcCIwM/H4IOC/QMsqyh0IfAZYYvp94Ph0/66a003NMk3P58kTZranmf0zcZj9DTAO6FLD6/+b9LiYmk+iVld2x+Q4PPwaC6tbSIoxprQuYEkN8QI8CoxMPD4VqDgJbWZHmdnbiWaJrwm15pq2VbkdaorBzE43szmJpoWvgT1TXC6E91exPHf/BlgNdEsqk9JnVst23glYVE0MOxESfH1U/j5ub2ZTzGxZIoYHK8Ww2MPJ+024+78JRwGDzCwX6AH8s54xCWpzb4oqdwO8j1BT3NXdOwK/JdSkG9IKQs0SADMzNk1GlW1JjCsISaFcbV01/wYcZmbdCc1GjyZibAs8DvyB0GTSCfhXinH8t7oYzGxn4B5C00TnxHL/k7Tc2rptLic09ZQvrwOh+WdZCnFVVtN2/hzYpZrXVffct4mYspPmbV+pTOX39/8Ivbx6J2I4vVIMPc2sZTVxPAycRjjKmOLuG6opJylQcm/6OgBrgG8TJ6R+2QjrfBrIM7OjzawVoR23awPFOAW41My6JU6u/bqmwu6+ktB08FfgI3dfmHhqK0I7cBFQamZHEdqGU43hKjPrZOE6gAuTnmtPSHBFhP3c2YSae7mVQPfkE5uVTALOMrM+ZrYVYefzmrtXeyRUg5q28zSgh5ldaGZtzKyjmQ1IPHc/8Hsz28WCfma2LWGn9l/CifuWZjaapB1RDTF8C6wxs50ITUPl3gRWAddbOEnd1swOTHr+EUIzzqmERC9bQMm96fsV8AvCCc77CDXXBpVIoKcAtxJ+rLsA7xFqbOmO8R7gReADYCah9l2bRwlt6I8mxfw1cBkwlXBS8kTCTioVYwlHEIuB6SQlHnefC9wBvJMosyfwdtJrnwcWAivNLLl5pfz1zxKaT6YmXt8DGJViXJVVu53dfQ3wE+AEwgncj4HBiadvAv5B2M7fEE5uZiWa284BriKcXN+10nurylhgAGEnMw14IimGEuAoYC9CLX4p4XMof34x4XP+3t3fqON7l0rKT16I1FviMHs5cKK7vxZ1PNJ0mdnDhJO010YdS1Oni5ikXsxsOOEwez2hK10JofYqUi+J8xfHAr2jjiUO1Cwj9TUI+JRwuD4c+JlOgEl9mdkfCH3tr3f3pVHHEwdqlhERiSHV3EVEYiiyNvcuXbp4Tk5OVKsXEWmSZs+e/aW719T1GEghuZvZA4TuS1+4e24VzxtwO+Gy8GLCpcnv1rbcnJwcZs2aVVsxERFJYma1XaUNpNYs8yA1jMRHGL9jt8RtNKFfsoiIRKjW5O7urxIu+qjOscDDHrwFdCofslRERKKRjhOq3dh08KBCqhlnxMxGJ8aKnlVUVJSGVYuISFXScUK1qoGXquxf6e4TCJc2k5+fv1mZH374gcLCQtavX5+GsKShZGVl0b17d1q3rm64FBGJWjqSeyGbjpjXnXApet0XVFhIhw4dyMnJIZynlUzj7qxatYrCwkJ69epV+wtEJBLpaJaZBvxPYjS5A4A17r6iPgtav349nTt3VmLPYGZG586ddXQlUg8FBZCTAy1ahPuCOv3lfd2k0hVyEjAE6GJmhYRR31oDuPu9wDOEbpCLCF0hz9iSgJTYM58+I5G6KyiA0aOhuDhML1kSpgFG1Xcc0Bqk0ltmpLvv4O6t3b27u//F3e9NJHYSvWQucPdd3L23u6vzujRbjVkzawoyZXtkQhxXX70xsZcrLg7zG4KGH0iyatUq+vXrR79+/dh+++3p1q1bxfT333+f0jLOOOMMPvrooxrL3H333RQ09199DJXXzJYsAfeNNbPm+lFnyvbIlDiWVjMcWnXzt1hUf9667777emXz58/fbF5NJk5079nT3SzcT5xYp5fXaOzYsX7TTTdtNr+srMxLS0vTt6Imqq6fVXPQs6d7SB+b3nr2jDqyaGTK9ohbHFTzR+eVb0225t6Ye+NFixaRm5vLueeeS15eHitWrGD06NHk5+ezzz77MG7cuIqygwYN4v3336ekpIROnToxZswY+vbty8CBA/niiy8AuOaaa7jtttsqyo8ZM4YBAwawxx578MYb4Q9ovv32W0444QT69u3LyJEjyc/P5/33398strFjx7LffvtVxOeJUT4//vhjDj30UPr27UteXh6LFy8G4Prrr6d379707duXqxvqeLCZavSaWYbLlO2RKXGMHw/Z2ZvOy84O8xtCk03ujd1+NX/+fM466yzee+89unXrxg033MCsWbOYM2cOzz//PPPnz9/sNWvWrGHw4MHMmTOHgQMH8sADD1S5bHfnnXfe4aabbqrYUdx5551sv/32zJkzhzFjxvDee+9V+dpLLrmEmTNn8sEHH7BmzRqeffZZAEaOHMlll13GnDlzeOONN9huu+146qmnmD59Ou+88w5z5szhV7/6VZq2jgD0qOavu6ubH3eZsj0yJY5Ro2DCBOjZE8zC/YQJDXMyFZpwcm/svfEuu+zCfvvtVzE9adIk8vLyyMvLY8GCBVUm97Zt23LEEUcAsO+++1bUnis7/vjjNyvz+uuvM2LECAD69u3LPvvsU+VrX3zxRQYMGEDfvn155ZVXmDdvHqtXr+bLL7/k6KOPBsJFR9nZ2bzwwguceeaZtG3bFoBtt9227htCqtXYNbNMlynbI1PigJDIFy+GsrJw31CJHZpwcm/svXG7du0qHi9cuJDbb7+dl156iblz5zJ8+PAq+323adOm4nHLli0pKSmpctlbbbXVZmXKm1dqUlxczIUXXsjUqVOZO3cuZ555ZkUcVXVXdHd1Y2xAjV0zy3SZsj0yJY7G1mSTe5R742+++YYOHTrQsWNHVqxYwXPPPZf2dQwaNIgpU6YA8MEHH1R5ZPDdd9/RokULunTpwtq1a3niifBH89tssw1dunThqaeeAsLFYcXFxQwbNoy//OUvfPfddwB89VVN48FJfTRmzawpyJTtkSlxNKYmm9yj3Bvn5eWx9957k5ubyznnnMOBBx6Y9nVcdNFFLFu2jD59+nDLLbeQm5vL1ltvvUmZzp0784tf/ILc3FyOO+449t9//4rnCgoKuOWWW+jTpw+DBg2iqKiIo446iuHDh5Ofn0+/fv344x//mPa4RSQzRPYfqvn5+V75zzoWLFjAXnvtFUk8maakpISSkhKysrJYuHAhw4YNY+HChbRqFdmfZ21Cn5VINMxstrvn11YuMzKFbGbdunUMHTqUkpIS3J377rsvYxK7iGQ+ZYsM1alTJ2bPnh11GCLSRDXZNncREamekruISAwpuYuIxJCSu4hIDCm5JxkyZMhmFyTddtttnH/++TW+rn379gAsX76cE088sdplV+76Wdltt91GcdKAOUceeSRff/11KqGLiGxCyT3JyJEjmTx58ibzJk+ezMiRI1N6/Y477sjjjz9e7/VXTu7PPPMMnTp1qvfyRKT5UnJPcuKJJ/L000+zYcMGABYvXszy5csZNGhQRb/zvLw8evfuzZNPPrnZ6xcvXkxubi4QhgYYMWIEffr04ZRTTqm45B/gvPPOqxgueOzYsQDccccdLF++nEMOOYRDDjkEgJycHL788ksAbr31VnJzc8nNza0YLnjx4sXstddenHPOOeyzzz4MGzZsk/WUe+qpp9h///3p378/hx12GCtXrgRCX/ozzjiD3r1706dPn4rhC5599lny8vLo27cvQ4cOTcu2FZHGlbH93C+9FKoYvnyL9OsHibxYpc6dOzNgwACeffZZjj32WCZPnswpp5yCmZGVlcXUqVPp2LEjX375JQcccADHHHNMtQNx3XPPPWRnZzN37lzmzp1LXl5exXPjx49n2223pbS0lKFDhzJ37lwuvvhibr31VmbMmEGXLl02Wdbs2bP561//yttvv427s//++zN48GC22WYbFi5cyKRJk/jzn//MySefzBNPPMFpp522yesHDRrEW2+9hZlx//33c+ONN3LLLbdw3XXXsfXWW/PBBx8AsHr1aoqKijjnnHN49dVX6dWrl8afEWmiVHOvJLlpJrlJxt256qqr6NOnD4cddhjLli2rqAFX5dVXX61Isn369KFPnz4Vz02ZMoW8vDz69+/PvHnzqhwULNnrr7/OcccdR7t27Wjfvj3HH388r732GgC9evWiX79+QPXDChcWFnL44YfTu3dvbrrpJubNmwfACy+8wAUXXFBRbptttuGtt97i4IMPplevXoCGBRZpqjK25l5TDbsh/exnP+Pyyy/n3Xff5bvvvquocRcUFFBUVMTs2bNp3bo1OTk5VQ7zm6yqWv1nn33GzTffzMyZM9lmm204/fTTa11OTeP/lA8XDGHI4KqaZS666CIuv/xyjjnmGF5++WWuvfbaiuVWjlHDAovEg2rulbRv354hQ4Zw5plnbnIidc2aNWy33Xa0bt2aGTNmsGTJkhqXc/DBB1f8CfaHH37I3LlzgTBccLt27dh6661ZuXIl06dPr3hNhw4dWLt2bZXL+sc//kFxcTHffvstU6dO5aCDDkr5Pa1Zs4Zu3boB8NBDD1XMHzZsGHfddVfF9OrVqxk4cCCvvPIKn332GaBhgUWaKiX3KowcOZI5c+ZU/BMSwKhRo5g1axb5+fkUFBSw55571riM8847j3Xr1tGnTx9uvPFGBgwYAIR/Verfvz/77LMPZ5555ibDBY8ePZojjjii4oRquby8PE4//XQGDBjA/vvvz9lnn03//v1Tfj/XXnstJ510EgcddNAm7fnXXHMNq1evJjc3l759+zJjxgy6du3KhAkTOP744+nbty+nnHJKyusRkcyhIX+lXvRZiUQj1SF/VXMXEYkhJXcRkRjKuOQeVTORpE6fkUjmy6jknpWVxapVq5Q8Mpi7s2rVKrKysqIORURqkFH93Lt3705hYSFFRUVRhyI1yMrKonv37lGHISI1yKjk3rp164orI0VEpP4yqllGRETSI6XkbmbDzewjM1tkZmOqeL6nmb1oZnPN7GUz0zG7iEiEak3uZtYSuBs4AtgbGGlme1cqdjPwsLv3AcYBf0h3oCIikrpUau4DgEXu/qm7fw9MBo6tVGZv4MXE4xlVPC8iIo0oleTeDfg8abowMS/ZHOCExOPjgA5m1rnygsxstJnNMrNZ6hEjItJwUknuVY3/Wrkj+hXAYDN7DxgMLANKNnuR+wR3z3f3/K5du9Y5WBERSU0qXSELgZ2SprsDy5MLuPty4HgAM2sPnODua9IVpIiI1E0qNfeZwG5m1svM2gAjgGnJBcysi5mVL+tK4IH0hikiInVRa3J39xLgQuA5YAEwxd3nmdk4MzsmUWwI8JGZfQz8CBjfQPGKiEgKUurn7u7PuPvu7r6Lu49PzPutu09LPH7c3XdLlDnb3Tc0ZNAiUrOCAsjJgRYtwn3iT8GkGcmo4QdEZMsVFMDo0VBcHKaXLAnTAKNGRReXNC4NPyASM1dfvTGxlysuDvOl+VByF4mZpUvrNl/iScldJGZ69KjbfIknJXeRmBk/HrKzN52XnR3mS/Oh5C4SM6NGwYQJ0LMnmIX7CRN0MrW5UW8ZkRgaNUrJvLlTzV1EJIaU3EVEYkjJXUQkhpTcRURiSMldRCSGlNxFRGJIyV1EJIaU3EVEYkjJXUQkhpTcRURiSMldRCSGlNxFRGJIyV1EJIaU3EVEYkjJXUQkhpTcRURiSMldRCSGlNxFRGJIyV1EJIaU3EVEYkjJXUQkhpTcJTYKCiAnB1q0CPcFBVFHJBKdVlEHIJIOBQUwejQUF4fpJUvCNMCoUdHFJRIV1dwlFq6+emNiL1dcHOaLNEdK7hILS5fWbb5I3KWU3M1suJl9ZGaLzGxMFc/3MLMZZvaemc01syPTH6pI9Xr0qNt8kbirNbmbWUvgbuAIYG9gpJntXanYNcAUd+8PjAD+lO5ARWoyfjxkZ286Lzs7zBdpjlKpuQ8AFrn7p+7+PTAZOLZSGQc6Jh5vDSxPX4gitRs1CiZMgJ49wSzcT5igk6nSfKXSW6Yb8HnSdCGwf6Uy1wL/MrOLgHbAYVUtyMxGA6MBeuh4WdJs1Cglc5FyqdTcrYp5Xml6JPCgu3cHjgQeMbPNlu3uE9w9393zu3btWvdoRUQkJakk90Jgp6Tp7mze7HIWMAXA3d8EsoAu6QhQRETqLpXkPhPYzcx6mVkbwgnTaZXKLAWGApjZXoTkXpTOQEVEJHW1Jnd3LwEuBJ4DFhB6xcwzs3Fmdkyi2K+Ac8xsDjAJON3dKzfdiIhII0lp+AF3fwZ4ptK83yY9ng8cmN7QRESkvnSFqohIDCm5i4jEkJK7iEgMKbmLiMSQkruISAwpuYuIxJCSu4hIDCm5i4jEkJK7iEgMKbmLiMSQkruISAwpuYuIxJCSu4hIDCm5i4jEkJK7iEgMKbmLiMSQkruISAwpucsWKyiAnBxo0SLcFxREHZGIpPQ3eyLVKSiA0aOhuDhML1kSpgFGjYouLpHmTjV32SJXX70xsZcrLg7zRSQ6Su6yRZYurdt8EWkcSu6yRXr0qNt8EWkcSu6yRcaPh+zsTedlZ4f5IhIdJXfZIqNGwYQJ0LMnmIX7CRN0MlUkauotI1ts1Cglc5FMo5q7iEgMKbmLiMSQkruISAwpuYuIxJCSu4hIDCm5i4jEUErJ3cyGm9lHZrbIzMZU8fwfzez9xO1jM/s6/aGKiEiqau3nbmYtgbuBnwCFwEwzm+bu88vLuPtlSeUvAvo3QKwiIpKiVGruA4BF7v6pu38PTAaOraH8SGBSOoKTmmkcdRGpTirJvRvwedJ0YWLeZsysJ9ALeKma50eb2Swzm1VUVFTXWCVJ+TjqS5aA+8Zx1JXgRQRSS+5WxTyvpuwI4HF3L63qSXef4O757p7ftWvXVGOUKmgcdRGpSSrJvRDYKWm6O7C8mrIjUJNMo9A46iJSk1SS+0xgNzPrZWZtCAl8WuVCZrYHsA3wZnpDlKpoHHURqUmtyd3dS4ALgeeABcAUd59nZuPM7JikoiOBye5eXZONpJHGUReRmqQ05K+7PwM8U2nebytNX5u+sKQ25UPsXn11aIrp0SMkdg29KyKg8dybNI2jLiLV0fADIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQykldzMbbmYfmdkiMxtTTZmTzWy+mc0zs0fTG6aIiNRFq9oKmFlL4G7gJ0AhMNPMprn7/KQyuwFXAge6+2oz266hAhYRkdqlUnMfACxy90/d/XtgMnBspTLnAHe7+2oAd/8ivWGKiEhdpJLcuwGfJ00XJuYl2x3Y3cz+bWZvmdnwqhZkZqPNbJaZzSoqKqpfxCIiUqtUkrtVMc8rTbcCdgOGACOB+82s02Yvcp/g7vnunt+1a9e6xioiIilKJbkXAjslTXcHlldR5kl3/8HdPwM+IiR7ERGJQCrJfSawm5n1MrM2wAhgWqUy/wAOATCzLoRmmk/TGaiIiKSu1uTu7iXAhcBzwAJgirvPM7NxZnZMothzwCozmw/MAP7X3Vc1VNAiIlKzlPq5u/sz7r67u+/i7uMT837r7tMSj93dL3f3vd29t7tPbsigo1ZQADk50KJFuC8oiDoiEZFN1drPXTZVUACjR0NxcZhesiRMA4waFV1cIiLJNPxAHV199cbEXq64OMwXEckUSu51tHRp3eaLiCT7+uvGWY+Sex316FG3+SIi5V54AXbfHR55pOHXpeReR+PHQ3b2pvOys8N8EZGqlJXBddfBsGHQtSvk5zf8OpXc62jUKJgwAXr2BLNwP2GCTqaKSNVWrYKf/hR++1s49VR45x3Ya6+GX696y9TDqFFK5iJSu7ffhpNOgpUr4d57Q886q2pAlwagmruISJq5w113wUEHQcuW8MYb8MtfNl5iByV3EZG0WrsWRo6Eiy6Cww+H2bNh330bPw4ld4mVDRuijkCas3nzYL/94LHH4A9/gCefhG23jSYWJXdp0r75BqZNC7WkPfeErCw4/3xYvz7qyKS5mTgRBgwI/dhffBHGjAlDlERFJ1SlSSkpgZkz4fnnw+2tt8K8tm1h8GA44AC4555wIuuxx2DnnaOOuHlbvRouuyzshMeOhb59o44o/davh0svhfvug4MPhsmTYYcdoo5KNXfJcO6wcCH86U9w3HHQpQv8+Mdw7bXhR/W//wsvvRSSyPTp8OCD4VD4008hLw+mTo36HTRfr7wSknlBQfiM+vULXQEXLYo6svT57DM48MCQ2H/961Bjz4TEDoC7R3Lbd999XaQqX37p/re/uZ99tnvPnu4hxYfH55zjPmVKKFOTzz5z32+/8LrLLnPfsKERAhd3D9v6yivdzdx328195kz3r74K87Kz3Vu2dB892r2wMOpIt8y0ae6dOrlvvbX7k0823nqBWZ5CjlVyl8itX+/+0kvhx5+fH5ICuHfs6P6zn7nffbf7xx+7l5XVfbkXXRSWdcAB7kuWNEz8stFHH4XPEMLOee3aTZ9fscL9ggvcW7d2z8pyv+KK2nfUmeaHH9x//evwHvPy3D/5pHHXr+QuGauszH3uXPdbbnEfPjzU5iDU6AYNcv/d79zfeCP8iNJhyhT3Dh3ct93W/Z//TM8yZVNlZe733x8+y223dX/iiZrLf/qp+89/HnbkHTu6jxu3+Y4gE61Y4T54cPi+jh7t/t13jR+DknsDW7XKfeXKqKNoOpYvd3/4YffTTnPffnuvaGrZYw/3Cy8Mh7Vr1jTc+j/+2L1v37DOK69M345DQs37+OPDth06tG7NLR9+GI7OwL1rV/fbbgtHXJno5ZfDd7dt2/BdjoqSewP44Qf3p592P/FE9zZtwqHlFVe4f/111JFlnnXr3J95JrR35+ZuTOZduriPGOH+l780fjNJcXFoswf3gw92X7ascdcfRy+84L7jjuG3cPPN7qWl9VvOW2+5H3po+Gx69HB/4IHM2QGXlrrfcIN7ixbuu+/u/sEH0caj5J5GCxa4/9//ue+ww8YEdckl7mecEQ4ru3Z1v+8+95KSqCONTkmJ+zvvuI8f7z5kSNj5gftWW4Xa3A03uM+eXf8ffzo9/HBoPthuu5CcpO7Wrw8VG3Dfc0/3d99Nz3Kff35jm/2ee7o/9ljdz7Wk01dfuR99dIjn5JPdv/kmuljKKblvoa+/dr/33nAirrw9+Oij3f/+9017XsyaFdqJIRz2z5gRWciR+O67cChdvuMr3w5XXOH+3HPu334bdYRVmzfPfa+9ws75d79r3jvmupo/371fv/BZn3de+j/jsrLQZr/XXmEd++4bvkuNneRnzXLPyQlHJXfcEe1OJpmSez2Uloaaw6mnhjP54L733u433RROpFSnrCx03evRI7zmuOMa/wx6Y1u/3v2uu8IhOYTa+sSJ7v/9b9SRpW7dunBSD9x/8hOdQ6lNWZn7n/4UfhtduoSugA2ppMT9r3/d+LsaMsT9zTcbdp3u4X3ee284+txpp8ZZZ13EMrlPnBj6OpuF+4kT67yIKn3yiftvfrPxS9Rfb6R7AAAKsklEQVSpU6iRvPNO3fbWxcXu110XDvnbtAndpRryJGEUNmxwv+ce9+7dw7YaNCh0Y2yqysrc//zn0Hy0447ur74adUSZaeVK96OOCp/54YfXXNlJt/XrQ815u+3C+o85puHavZN3+Icf7l5U1DDr2RKxS+4TJ27sMld+y86uf4JfuzbUCg4+OCzLLHyYkyZtefemZcvc/+d/wnJ/9KPQRaypH/Z//31IguU7wIEDw1FOphyqbqn33nPfddfQ/HbDDZlxbiBTTJ8evsdbbeV+++3RbZu1a91///vQddIs9LxK5xHyggXu++wTlj1uXOZ+B2KX3JOvVEy+9eyZ+jLKykLN7Iwz3Nu1C6/fdddwEvDzz+sUTkrefjskQXDv379p1gp/+CH0XOjVK7yPAQPCjz0uST3ZmjXuJ50U3udRR4Xurs3Zd9+5X3xx2B65ueHahEywalXo4JCV5d6qlfv554eutlti8mT39u1Dc9O//pWeOBtK7JJ7+VWLlW9mtb926dKwx9911/Ca9u3dzzrL/bXXGj5JlZW5FxRsbMY46aRwaXym++EH94ce2rjN9t03dAONY1JPVlbmfued4SRajx6hi15zNHfuxi6sF18czcU6tVm2zP3cc0OCb9vWfcyY0LulLjZs2HgV88CBDVPJS7fYJfe61tyLi90ffTScKCvfMQwZEhLWunV1WnVafPut+7XXhi/hVlu5X3VVZl6RV1ISdka77+4VPV+efDL+Sb2yd94J363WrUNTRHN5/2Vl4f1utVVoinnmmagjqt3ChaEThFkY52X8+NR+40uWuO+/v1eMP/T99w0fazrELrmn0uZeVhZqWueeGz7k8uQ/dmzm9F5ZujR8ESF0H3zwwcxo2ystDYem5d3PcnNDd7RMiC0qyX2cTzgh/herrVgRhoMob5Zqar2H5szZ+Hn96EfhCKy6AeOmTw/DJHTo4P74440b55aKXXJ3r763zIoV7jfeGLotQqgdn3aa+4svZm5yeuON0H4N4aKN11+PJo7S0vDlLj8E33vvMBZLpm63xlZWFrrCtmzpvssu6btYJ9NMmxbam7OyQnfHpnyk8u9/bxz/JScnHK2Xd2goKQk948zc+/QJw1I0NbFM7sk2bAg1y6OOCj88cP/xj0OPjqZSwyotDVdLlvcVHzGi8S7JLytznzp143gre+wRego19V49DeX11927dQvNFffe27STX7Jvvw3dfiFcmDR/ftQRpUdZmfuzz4ZRGyH0gnn0UffDDgvTp5+euRfY1Sa2yX3evHCCp3PnEP2OO4YTKf/5T70WlxHWrg21iayscPvNbxruvEBZWaillX/pd93V/ZFHlNRT8cUX7sOGhe126qmZec6kLt59N1ziD+GK4kwdsGtLlJaGI9Hyc0hZWWFco6Ystsn91lvDBUInnRRO9mTK4ELpsHix+ymnhE+lW7eQdNPVPFJWFrZX+bgdO+8c2vvjtP0aQ2lpuFCtRYuQGKMeRKo+SktDU1Pr1qFy9PzzUUfU8H74IRyZNsXPq7K0JndgOPARsAgYU8XzpwNFwPuJ29m1LbO+yX3Nmvj3P37ttdD1EMLZ/C3pjldWFsblKB8jp2fPcFFVU+kZkKlefDGctGvbNuwkm4rCwjCQW/kwGU3tjzIkjckdaAl8AuwMtAHmAHv75sn9rlRWWH7LxLFlMklpabiCtnzs89NOq1sf3LKykIAOPDC8fqedwsiV+ru59FmxInSvBfczz8z8Ntwnngg9RLKzw7mpuJw3aG5STe6p/EH2AGCRu3/q7t8Dk4Fj6/F3rVIHLVrA6afDxx/DlVfCY4/BHnvAuHFQXFzza195BQ45BIYOhcWL4e67w59Mjx4Nbdo0RvTNw/bbw/PPwzXXwAMPwAEHwEcfRR3V5tatg7PPhhNOgF694L33wrRZ1JFJQ0oluXcDPk+aLkzMq+wEM5trZo+b2U5VLcjMRpvZLDObVVRUVI9wm58OHeD662HBAjjySBg7FvbcEyZNCr39k73+ekjoQ4aEJHPHHeGf5s8/H7baKpLwY69VK7juOpg+HZYvh/x8+Nvfoo5qo5kzIS8v7HyuvBLeeAN23z3qqKQxmFfOEJULmJ0EHO7uZyemfw4McPeLksp0Bta5+wYzOxc42d0PrWm5+fn5PmvWrC1+A83NK6/ApZfC++/Dj38Mt90GJSUh6T//PGy3HYwZA+eeC23bRh1t8/L553DKKfDmm3DaabDbblBWBqWlm95XNa+hnpszB3bYAR55BAYPjnoLSTqY2Wx3z6+tXKsUllUIJNfEuwPLkwu4+6qkyT8D/y+VIKXuBg+GWbPgwQfhqqtgwIAwv0sXuOkmOO88aNcu0hCbrZ12CjvfK68MO93S0jDfLDSztWxZ830qZWoq26rV5vMHDgxNedtsE+22kcaXSs29FfAxMBRYBswETnX3eUlldnD3FYnHxwG/dvcDalquau5b7ptv4M47Q5PLuedC+/ZRRyTlSko2JnW1bUs6pa3m7u4lZnYh8Byh58wD7j7PzMYRztpOAy42s2OAEuArQu8ZaWAdO8LVV0cdhVSlVSrHxCINqNaae0NRzV1EpO5Srbmn0ltGRESaGCV3EZEYUnIXEYkhJXcRkRhSchcRiSEldxGRGFJyFxGJocj6uZtZEbAkkpWnTxfgy6iDyCDaHhtpW2xK22NTW7I9erp719oKRZbc48DMZqVyMUFzoe2xkbbFprQ9NtUY20PNMiIiMaTkLiISQ0ruW2ZC1AFkGG2PjbQtNqXtsakG3x5qcxcRiSHV3EVEYkjJXUQkhpTc68HMdjKzGWa2wMzmmdklUccUNTNraWbvmdnTUccSNTPrlPij+P8kviMDo44pSmZ2WeJ38qGZTTKzrKhjaixm9oCZfWFmHybN29bMnjezhYn7BvkTRCX3+ikBfuXuewEHABeY2d4RxxS1S4AFUQeRIW4HnnX3PYG+NOPtYmbdgIuBfHfPJfyb24hoo2pUDwLDK80bA7zo7rsBLyam007JvR7cfYW7v5t4vJbw4+0WbVTRMbPuwE+B+6OOJWpm1hE4GPgLgLt/7+5fRxtV5FoBbRP/x5wNLI84nkbj7q8S/no02bHAQ4nHDwE/a4h1K7lvITPLAfoDb0cbSaRuA/4PKIs6kAywM1AE/DXRTHW/mbWLOqiouPsy4GZgKbACWOPu/4o2qsj9yN1XQKgoAts1xEqU3LeAmbUHngAudfdvoo4nCmZ2FPCFu8+OOpYM0QrIA+5x9/7AtzTQYXdTkGhPPhboBewItDOz06KNqnlQcq8nM2tNSOwF7v73qOOJ0IHAMWa2GJgMHGpmE6MNKVKFQKG7lx/JPU5I9s3VYcBn7l7k7j8Afwd+HHFMUVtpZjsAJO6/aIiVKLnXg5kZoU11gbvfGnU8UXL3K929u7vnEE6UveTuzbZm5u7/BT43sz0Ss4YC8yMMKWpLgQPMLDvxuxlKMz7BnDAN+EXi8S+AJxtiJa0aYqHNwIHAz4EPzOz9xLyr3P2ZCGOSzHERUGBmbYBPgTMijicy7v62mT0OvEvoZfYezWgoAjObBAwBuphZITAWuAGYYmZnEXZ+JzXIujX8gIhI/KhZRkQkhpTcRURiSMldRCSGlNxFRGJIyV1EJIaU3EVEYkjJXUQkhv4/ZDQUQSVJaBUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4lNX1wPHvYRdZgoCoIIu4AgJCVPxJCy5VUBFElARwoVDEfWsrdZdKFaUqsbigBrcIIiouVbFVKlg3wmIQkIIKGEEJKCiCSOD8/jgTSEKWSTIz78zkfJ4nTzIzb973ZJKcuXPvufeKquKccy651Ag6AOecc5Hnyd0555KQJ3fnnEtCntydcy4JeXJ3zrkk5MndOeeSkCd3VyIRqSkiW0SkdSSPDZKIHCoiEa/9FZFTRWRVodvLReQ34RxbiWs9LiI3Vvb7yzjvnSLyZKTP64JTK+gAXGSIyJZCN+sD24GdoduXqGpWRc6nqjuBBpE+tjpQ1SMicR4RGQkMU9Xehc49MhLndsnPk3uSUNXdyTXUMhypqv8u7XgRqaWq+bGIzTkXe94tU02E3nY/LyJTReQnYJiInCAiH4nIJhFZJyIZIlI7dHwtEVERaRu6/Wzo8TdF5CcR+VBE2lX02NDjfUXkfyKyWUQeFJH/isjFpcQdToyXiMhKEflBRDIKfW9NEblfRDaKyBdAnzKen5tFZFqx+yaJyH2hr0eKyLLQz/NFqFVd2rlyRaR36Ov6IvJMKLYlQPcSrvtl6LxLROTs0P1HA/8AfhPq8tpQ6Lm9vdD3jw797BtFZKaIHBjOc1MeERkQimeTiLwrIkcUeuxGEVkrIj+KyOeFftYeIrIgdP93InJvuNdzUaCq/pFkH8Aq4NRi990J/Ar0w17U9wGOBY7H3sEdAvwPuCJ0fC1Agbah288CG4BUoDbwPPBsJY7dH/gJ6B967DpgB3BxKT9LODG+AjQG2gLfF/zswBXAEqAV0BSYY3/yJV7nEGALsG+hc68HUkO3+4WOEeBkYBvQOfTYqcCqQufKBXqHvp4A/AdoArQBlhY79nzgwNDvZEgohhahx0YC/ykW57PA7aGvTwvF2BWoBzwEvBvOc1PCz38n8GTo66NCcZwc+h3dGHreawMdgdXAAaFj2wGHhL6eB6SHvm4IHB/0/0J1/vCWe/Xyvqq+pqq7VHWbqs5T1Y9VNV9VvwQmA73K+P4ZqpqtqjuALCypVPTYs4BFqvpK6LH7sReCEoUZ412qullVV2GJtOBa5wP3q2quqm4E7i7jOl8Cn2EvOgC/Azapanbo8ddU9Us17wLvACUOmhZzPnCnqv6gqqux1njh605X1XWh38lz2AtzahjnBRgKPK6qi1T1F2AM0EtEWhU6prTnpixpwKuq+m7od3Q30Ah7kc3HXkg6hrr2vgo9d2Av0oeJSFNV/UlVPw7z53BR4Mm9evm68A0ROVJE/iki34rIj8BYoFkZ3/9toa+3UvYgamnHHlQ4DlVVrKVbojBjDOtaWIuzLM8B6aGvh2AvSgVxnCUiH4vI9yKyCWs1l/VcFTiwrBhE5GIR+TTU/bEJODLM84L9fLvPp6o/Aj8ALQsdU5HfWWnn3YX9jlqq6nLgeuz3sD7UzXdA6NDhQAdguYh8IiJnhPlzuCjw5F69FC8DfBRrrR6qqo2AW7Fuh2hah3WTACAiQtFkVFxVYlwHHFzodnmlms8Dp4Zavv2xZI+I7APMAO7CukxSgLfDjOPb0mIQkUOAh4FLgaah835e6LzllW2uxbp6Cs7XEOv++SaMuCpy3hrY7+wbAFV9VlVPxLpkamLPC6q6XFXTsK63vwMviki9KsbiKsmTe/XWENgM/CwiRwGXxOCarwPdRKSfiNQCrgaaRynG6cA1ItJSRJoCN5R1sKp+B7wPTAGWq+qK0EN1gTpAHrBTRM4CTqlADDeKSIrYPIArCj3WAEvgedjr3Eis5V7gO6BVwQByCaYCI0Sks4jUxZLsXFUt9Z1QBWI+W0R6h679J2yc5GMROUpETgpdb1voYyf2A1wgIs1CLf3NoZ9tVxVjcZXkyb16ux64CPvHfRRruUZVKIEOBu4DNgLtgYVYXX6kY3wY6xtfjA32zQjje57DBkifKxTzJuBa4GVsUHIQ9iIVjtuwdxCrgDeBpwudNwfIAD4JHXMkULif+l/ACuA7ESncvVLw/W9h3SMvh76/NdYPXyWqugR7zh/GXnj6AGeH+t/rAvdg4yTfYu8Ubg596xnAMrFqrAnAYFX9tarxuMoR6/J0LhgiUhPrBhikqnODjse5ZOEtdxdzItJHRBqH3trfglVgfBJwWM4lFU/uLgg9gS+xt/Z9gAGqWlq3jHOuErxbxjnnkpC33J1zLgkFtnBYs2bNtG3btkFd3jnnEtL8+fM3qGpZ5cNAgMm9bdu2ZGdnB3V555xLSCJS3kxrwLtlnHMuKXlyd865JOTJ3TnnkpAnd+ecS0Ke3J1zLgklVHLPyoK2baFGDfucVaEtn51zrvpImA2ys7Jg1CjYutVur15ttwGGVnkdPOecSy4J03K/6aY9ib3A1q12v3POuaISJrmvWVOx+51zrjorN7mLSKaIrBeRz8o4preILBKRJSLyXmRDNK1L2SCttPudc646C6fl/iS2LGuJRCQFeAjbqaUjcF5kQitq3DioX7/offXr2/3OOeeKKje5q+ocbGux0gwBXlLVNaHj10cotiKGDoXJk6FNGxCxz5Mn+2Cqc86VJBLVMocDtUXkP9hmxhNV9emSDhSRUcAogNaV6E8ZOtSTuXPOhSMSA6q1gO7AmcDpwC0icnhJB6rqZFVNVdXU5s3LXbHSOedcJUWi5Z4LbFDVn4GfRWQO0AX4XwTO7ZxzrhIi0XJ/BfiNiNQSkfrA8cCyCJzXOedcJZXbcheRqUBvoJmI5AK3AbUBVPURVV0mIm8BOcAu4HFVLbVs0jnnXPSVm9xVNT2MY+4F7o1IRM4556osYWaoOuecC58nd+ecS0Ke3J1zLgl5cnfOVQs//gg//xx0FLHjyd05l/RU4eST4byorHwVnxJmsw7nnKus+fPtQwTWrYMDDww6oujzlrtzLulNmQK1alkL/oUXgo4mNjy5O+eS2i+/wHPPwfnnQ5cuMG1a0BHFhid351xSmzkTNm2C3/8e0tLgww9h1aqgo4o+T+7OuaSWmWn7P5x0kiV3gOefDzamWPDk7pxLWmvWwL//DRdfDDVqQNu20KMHTJ0adGTR58ndOZe0nnrKBlEvumjPfenp8OmnsCzJ16715O6cS0q7dsGTT1p9e7t2e+4/7zxrxSf7wKond+dcUpozB778EoYPL3r/gQdC797WNaMaSGgx4cndOZeUpkyBRo1g4MC9H0tLgxUrYOHC2McVK57cnXNJ58cfYcYMS+L16+/9+Lnn2qSmZB5Y9eTunEs606fD1q17d8kU2G8/OP10K4nctSu2scVKucldRDJFZL2IlLl1nogcKyI7RWRQ5MJzzrmKmzIFjjoKjj++9GPS0+Hrr+GDD2IXVyyF03J/EuhT1gEiUhMYD8yKQEzOOVdpn39uCXv4cFsorDRnnw316iVv1Uy5yV1V5wDfl3PYlcCLwPpIBOWcc5X15JNQsyZccEHZxzVsCP362UJi+fkxCS2mqtznLiItgXOAR8I4dpSIZItIdl5eXlUv7ZxzReTn28SlM86AAw4o//j0dFi/HmbPjn5ssRaJAdUHgBtUdWd5B6rqZFVNVdXU5s2bR+DSzjm3x6xZ8O23tkhYOPr2tXLJZKyaiURyTwWmicgqYBDwkIgMiMB5nXOuQjIzoXlzOPPM8I6vVw/OOQdeegm2b49ubLFW5eSuqu1Uta2qtgVmAJep6swqR+accxWQlwevvWZ97bVrh/99aWmweTO89Vb0YgtCOKWQU4EPgSNEJFdERojIaBEZHf3wnHMuPFlZsGNH6bXtpTnlFGjWLPmqZsrdQ1VV08M9mapeXKVonHOuElStS+bYY6FTp4p9b+3aMGgQPP00/Pwz7LtvdGKMNZ+h6lwUbN2avDMf49GCBbB4ccVb7QXS0+139tprkY0rSOW23J1zFfPRR3DiiTaBpkULW4XwoIPsc0kfLVpUrI/Y7W3KFBscTQ+7n6Gonj2hZUurminYrSnReXJ3LsImTIDGjeHSS2HdOvtYs8aSfknTO0SswqO05F/wwnDAAZbAXFEFG2Cfcw6kpFTuHDVqwODB8OCD8MMP0KRJZGMMgid35yJozRp4+WX44x9h3Li9H9+xA777bk/SL/yxdq19zsmxY3aWMHOkSZOSE3/xjwYNov+zxotXXrGEHG5te2nS0uC+++z3V9VzxQNP7s5F0KRJ1hK//PKSH69dG1q1so+y7NwJGzaU/gKwbh3MnWuff/117+9v0MCSfPfuNtC4zz5V/9ni1ZQp0Lq17bhUFamp0L69dc14cnfO7fbzz/DYY9Y90Lp11c5Vs6b1xbdoAV27ln6cqrVaS0r+a9ZYed8xx8Cf/1y1eOLV11/D22/DzTdb10pViFjr/a677J1TixaRiTEontydi5CsLEu0V10Vu2uK2Nrk++0HHTvu/fiZZ1qyGjnSjkk2Tz9tL3AXXxyZ86WnW3faCy/AFVdE5pxB8VJI5yJAFTIyrJXcs2fQ0exx9902+/Kuu4KOJPJUrUumd2845JDInLNjR6uTT4YJTZ7cnYuAd96BJUvg6qvLXkM81o4+Gi680KpA1qwJOprImjsXvvgi8v3j6enw3/8m/vPlyd25CJg40coZBw8OOpK9jR1rn2+5Jdg4Ii0z09ZkP/fcyJ634Hf4/PORPW+seXJ3ropWroR//hNGj47POvTWrW0c4Jln4NNPg44mMn76yfrFS9sAuyrat4fjjkv8ZYA9uTtXRf/4h1W3XHpp0JGU7i9/sYlVf/lL0JFERnkbYFdVejosXAjLl0fn/LHgyd25KvjxR+seOP98qyuPV02awI03wptvJseuQ1OmwJFHQo8e0Tn/+efb2EkiD6x6cneuCp56yroIrr466EjKd+WVcPDBVvOeyIuaLV9uA57lbYBdFQcdBL16WXJXjc41os2Tu3OVtGuXVaH06GF9tPGuXj34618hO9v6qxNVuBtgV1VaGnz+eeKOU3hyd66S3nwTVqyI7aSlqho2zMojb7qp5GUL4l3BBth9+0a/G+zcc6FWrcQdWA1nJ6ZMEVkvIp+V8vhQEckJfXwgIl0iH6Zz8Scjw96+DxoUdCThq1nTJjZ98QVMnhx0NBX39tu2tEIs1n5p1gx+97vE7ZoJp+X+JNCnjMe/Anqpamfgr0AC/sk4VzHLllmiueyyxFuLvW9fm9U5dqwNCCeSKVMs6Ya7AXZVpafbZKYPP4zN9SKp3OSuqnOA78t4/ANV/SF08yOgnPXunEt8GRlQty6MGhV0JBUnAvfcY2vLT5gQdDTh27DBlvcdNgzq1InNNfv3t7GKRKyaiXSf+wjgzdIeFJFRIpItItl5Je1a4FwC+OEHW7BqyBCblZqIjj3Wyv3+/nf49tugownPc8/ZevixXI63USN7lzB9uvX3J5KIJXcROQlL7jeUdoyqTlbVVFVNbZ6o/xWu2nviCZtAkwjlj2UZN84GVe+4I+hIwpOZaevTH310bK+blmZLAL/3XmyvW1URSe4i0hl4HOivqhsjcU7n4lF+vs1I7dULuiR46cChh8Ill9ga9PE+E3PhQitJDGITjTPPtM1PEq1qpsrJXURaAy8BF6jq/6oeknPx67XXYPXqxCp/LMutt9ouTTfeGHQkZcvMtDGOym6AXRX77AMDBsCLLyZW+Wg4pZBTgQ+BI0QkV0RGiMhoERkdOuRWoCnwkIgsEpHsKMbrXKAmToQ2bWygLRnsvz/86U/w0ku2gXc8+uUX2wjlnHOC27g6PR02bYJZs4K5fmWIBlTAmZqaqtnZ/jrgEseiRbYZx7332gbYyWLLFuuiOfxw61eOp/XowQYzBw+2xHraacHE8OuvNmmqTx97oQmSiMxX1dTyjvMZqs6F6cEHbXnZESOCjiSyGjSA226zzS9efz3oaPY2ZYqtiXPKKcHFUKeOTVZ75RUbTE8EntydC0NenrXYLrwwuK6BaBo50lruY8bEV8lfbq612C+6yGbXBik93TZBj8cXwJJ4cncuDJMnw/bttrJiMqpdG/72N1i61NZuiReR3gC7Kn7zG+uaSZSqGU/uCW7qVOsucNGzYwc89JCtM9KhQ9DRRM/AgXD88dZFEw9dDwUbYPfqZbsjBa1mTev7f+MN23Q83nlyT2DLl1uL5qqrbINmFx0vvghr1yb+pKXyFCxL8M03trxC0N5/37YwDKK2vTRpaTa4+vLLQUdSPq+WSVCqcPLJVsHRtCns3AmLF9vgmIusE06wdU2WL4ca1aA51K+fDa5+8YX9bQVl+HB7YV23DvbdN7g4ClO1dxGHHRZcWaRXyyS5Z56B//wHxo+3zQtWr47/iSiJ6JNPrP77yiurR2IHuOsu211q3LjgYtiyxTYUOf/8+EnsYO9u0tLsnfL69UFHU7Zq8ueaXDZsgOuug//7P6ty6NnTks+DD1qLy0VORgY0bBgfA3qx0qmTVadMmgSrVgUTwwsvWGVKPHXJFEhPt3fKM2YEHUnZPLknoD//2QZ0Hn10T2vyb3+Ddu3snyEeBsOSwbp1NoHm97+31QGrk7Fj7W/rlluCuX5mJhxxhHWJxZtOnWxgPd6XAfbknmDee88qCP74R/sjK7DvvvD44zYAddttwcWXTB55xGq+r7gi6Ehir1UrG0DOyrJxnVj63/9sMDWaG2BXhYi13ufOha+/Djqa0nlyTyDbt8Po0dZCL6lFdfLJtsrffffBxx/HPr5ksn27Jfczz7Sp+dXRmDGQkgI3lLqId3Q8+aS9a4j2BthVkZZmn6dPDzaOsnhyTyD33GO7sT/0kE2DL+2Yli2t1bN9e2zjSybTptmAWbKXP5YlJcU20n77bfj3v2NzzZ0792yAfdBBsblmZRx6KKSmxveEJk/uCWLFCqteGDzYFi8qTaNGNpty2TLrN3UVp2qrP3boEOx6JvHg8suhdWtrve/aFf3rvf22zSkYPjz616qqtDSYP9/+N+ORJ/cEoAqXXmrrWd9/f/nH9+lj1R3jx8OCBVEPL+n897+2OcRVV8Vnn28s1asHd95pf0fPPx/96xVsgN2vX/SvVVWDB9vneB1Y9eSeALKyrK727rttbYtw3HefrdU9fHhibTAQDyZOtMXB4rnPN5aGDrVdp266KbpdfRs32qqLQ4fGbgPsqmjVytabmTrVGmDxxpN7nPv+e6tpP/54GywNV5MmNiCYk2MvCi48a9bY1PI//KH0cY3qpkYN+xv66isrv42W556zhkgidMkUSE+3LtDFi4OOZG/h7MSUKSLrReSzUh4XEckQkZUikiMi3SIfZvV1ww2W4CdPrvgMybPPhiFD7G11PP7xxaOHHrJW2GWXBR1JfDn9dKvG+utf4ccfo3ONzEzo1i2x9qYdNMgWFIvHrplw0sWTQBlDePQFDgt9jAIernpYDqyO9vHHreXeuXPlzlHQxTB8eHyt0x2Ptm61F9FzzrGt9NweBYuKbdhgnyNt4UKrp4/HGallad4cTj3Vknu8dc2Um9xVdQ7wfRmH9AeeVvMRkCIiYfYMu9L8+qvVtLdpU7VJSc2a2TTy+fNhwoTIxZeMsrLghx+qd/ljWbp3twqR++6zipZImjLF+tmD2AC7qtLTrcvqk0+CjqSoSPS5twQKz9PKDd23FxEZJSLZIpKdl5cXgUsnrwkTbOOESZOqvnDSoEFw7rlw++3WP+j2VlD+2LWrrdXjSnbnnfYO8I47InfO7dvthXXAANhvv8idN1YGDLBKtnireY9Eci+pWKzENyiqOllVU1U1tXnz5hG4dHL64gvr2xw0yGZIRkLBi8SIETZRxBX17ruwZIm12qt7+WNZ2re3d5RPPGET6iLhtddsXCnRumQKNG4MZ5xhpaLx9L8VieSeCxxc6HYrIMJv2qqPgsG82rWtJRkpLVrYCocffhgfGzHEm4wM6z8tmFbuSnfLLVZJ9Je/ROZ8mZlWVnjqqZE5XxDS0uDbb2HOnKAj2SMSyf1V4MJQ1UwPYLOqrovAeauladNslt7f/hb56ddDhsBZZ1m98sqVkT13IvviC2s9XnKJTdpxZWve3FYmnTnTJnxVxTffxM8G2FVx1ln2zjieumbCKYWcCnwIHCEiuSIyQkRGi8jo0CFvAF8CK4HHAC8iq6QffoBrr4Vjj7UZqZEmYrXvdepY90wsppMngn/8wxJLNJ7zZHXttXDAAZbkq1Il8vTT9neY6Ovl168P/fvbzlFxM2lQVQP56N69u7qiLrlEtWZN1YULo3udJ55QBdVJk6J7nUTw44+qjRqpDhkSdCSJ55FH7O9o5szKff+uXaqHHab6299GNq6gvPaaPR+vvx7d6wDZGkaO9RmqceKDD2z239VXW8VGNA0fDqedZq2uoHbaiRdPPWWTcq66KuhIEs+IEbahxpgxlZtD8d//2qJbiTQjtSynnWZzSuJlQpMn9ziwY4f19x58cGRLzEojAo89Zp//8If4m3wRK7t22UDq8cfbh6uYWrVsv9XPP7c69YqaMsU2dB80KPKxBaFOHSs5njkzPnZD8+QeB+67Dz77zMoVGzSIzTVbt4Z777V1up94IjbXjDdvvWUtR5+0VHkDBthWeLfdZnuehmvLFisdPP/82P3Nx0Jamv1sb7wRdCSe3AP31VfWWj/nnNgvczpqFPTuDddfD7m5sb12PMjIsFU2zz036EgSV8GyBOvWVax0d8aM+N0Auyp697ay43iomvHkHqCCmvaaNYOpPa9Rw9auyc+3bqHq1D3z+edWgnfZZYmxvGw869nTFqkbP97WnglHZiYcfjj83/9FN7ZYq1nT3o3885/RW2AtXJ7cA/TCC9Y1MG6cTeIIQvv2VlP/xhvw7LPBxBCEjAybMl6RZZRd6e66y7oj7ryz/GNXrLBF8eJ1A+yqSk+3JRVmzgw2Dk/uAdm0yfp6u3e3rcyCdOWVcOKJFs+6ajD9bNMmq5JJT7cJOa7qOnSwZP3QQ/Dll2UfW7AB9oUXxiS0mOvRwxb8C7pqxpN7QG66yTZgfvTR4Gfm1ahhg6pbt1o3RbJ3zxT8rD6QGll33GEVNLfcUvoxBRtg9+kT3xtgV4WIDaz+61/hd1NFgyf3AHz0ETz8sNVWd+8edDTmiCNsQ+2ZM627KFnt3GkzUn/72+jPJ6huWraEa66xHZVK27v3X/+yJQeSpba9NGlpNpY1Y0ZwMYgG1ExLTU3V7OzsQK4dpB07IDXVVsFbuhQaNgw6oj3y822A66uvLLZk7LKYOdMqk2bM8CqZaNi8GQ45xHZU+te/9n588GArv1271sY8kpWqdVW1aAH/+U9kzy0i81U1tbzjvOUeYxMn2r6mDz4YX4kd7C11Zqb9gybrjM2JE63Gv3//oCNJTo0bw803WwJ/++2ij33/vb24DhuW3IkdrGsmPd1Wifzmm2Bi8OQeQ6tW2WSP/v1t8kc86tQJbr3VBoOCHu2PtJwca0VdcYW9kLnouOwyaNvW9v8tvDhdIm6AXRVpadaCnz49mOt7co8RVUsqItZqj2c33GD90Zdeaq2tZJGRAfvsY2uiuOipW9dKIhctKjqZZ8oUOOaY6jPWcfjh1j0V1IQmT+4x8tJLNrHhr3+1NWTiWe3a9o+4YYMt7ZoMNmywrdwuvDAxt3JLNOnplsRvvtlqvj/91AZZq0urvUBaGsybZ3sGxJon9xjYvNlqyY85xj4ngq5dbbW/p5+Oj3UyqmryZPjll+QdS4g3NWrYjNVVq6z2vWAD7CFDgo4stgYPts9B1Lx7tUwMXHmlLQr28ce2EUei2L7dSjU3bbL9RRs3DjqiytmxA9q1s+qF4oN8Lrp+9ztrsYvAyScH1/8cpJ49rYG3eHFkzhfRahkR6SMiy0VkpYiMKeHx1iIyW0QWikiOiJxRmaCT0bx5ltivuCKxEjtY32lmps1a/dOfgo6m8l56ySoWvNUee+PH27jNxo3Jt0hYuNLTbdXXzz6L7XXD2WavJjAJ6At0ANJFpEOxw24GpqvqMUAa8FCkA01E+fm28uKBB4a35kY8Ou44WzXyscesvC0RTZwIhx5qO9S72OrWzbbQa9/eWvHV0aBB1k0V666ZcFruxwErVfVLVf0VmAYUrxJWoFHo68bA2siFmLgyMqxiICMDGjUq//h4dccdNvI/cqQtDpVI5s2DDz+0rrEaPsIUiMcesy6JoJfZCEqLFnDKKVY1E8te8HD+3FsCXxe6nRu6r7DbgWEikottmF3isKGIjBKRbBHJzsvLq0S4iWPNGqsXP+ssGDgw6GiqZp99rHtmzRobZE0kGRk2WSzRN2BOZLVq2d9QdZaWZguqxXKYMZzkXtKinMVff9KBJ1W1FXAG8IyI7HVuVZ2sqqmqmto8Gee2hxTUtKvaOibJsKzpiSdan/WkSTbrLhGsW2e7/QwfntjvnFziGzjQSoxjWfMeTnLPBQpXZrdi726XEcB0AFX9EKgHNItEgIlo5kx47TXrzmjTJuhoImfcOFs3ZMSI+NgjsjyPPmrjHolSfuqSV0oK9O1rjY3Cs3ajKZzkPg84TETaiUgdbMD01WLHrAFOARCRo7DknrT9LllZNr26Rg37nJW157GffrJk0qVL8i0pu+++tnPTypVlL+saD7Zvt5U3zzjDBlOdC1p6ui2YNndubK5XbnJX1XzgCmAWsAyrilkiImNF5OzQYdcDfxCRT4GpwMUaVAF9lGVlWQXM6tXW7bJ6td0uSPC33GK/wEcftbdhyeakk2D0aLj/flu6OF49/7ytl59sL7AucfXrB/Xrx65rxicxVVDbtpbQi2vTBl580UoHR4+2vulk9dNPtsBY/fqwcCHUqxd0REWp2pyCrVtt8lUyjHm45JCebkshr1tX+cafL/kbJWvWlHx/QQt+//1tT9Jk1rChTef//HPb4CPefPABzJ9vA8Ce2F08SU+3CV2xmDPiyb2CWrcu+f4mTWyadUZG4k7Tr4jTT7cZh/fcY4nANprIAAAVN0lEQVQ0nkycaANYF1wQdCTOFXX66fa3+dZb0b+WJ/cKGjfOuiMKq1cPtm2z0fBBg4KJKwh//7u9Uxk+3Nbpjgdff23LDfzhDzYA7Fw8qVvXat3vvz/61/LkXkFDh1qXRJs29pa/TRvrfxaxfvbq1A2QkmIDx4sXx09X1EMPWZ/75ZcHHYlzJWvfPjazpRNuP5rvvrMFeJo3t49mzWJflTJ0qH0AvPKK7ao0frytPFjd9Otnz8W4cbZOff36Vlte3seOHdE5btUq+30k0/wC5yoj4aplnn/epvIWlpKyJ9mH8xGp6o4tW2wZ2ZQU63dOxtLHcGzcCJ07WwloRdSqZR+1a+/5urSPcI+pU8eWSOhQfGk755JEuNUyCddyP/VUmD0b8vJK/vjiC1s3fcMGa8mVpEEDa/GH+2LQoEHJ3S233WZ9vM8/X30TO0DTprBsmVUShZuUa9SoXl1YzsVawiX3pk2hd+/yj1O1TSZKexEo+Fi3zjZOzsuzWY0lqVt374SfkmIzIEePhhNOiOiPmJAaNbKxB+dcfEi45B4uEStPbNLElqstj6p1s5T3YpCXB//7n31u3Rruuiv6P4tzzlVU0ib3ihKxyTkNG9riWOFQ9a4F51x88lLIKvDE7pyLV57cnXMuCXlyd865JOTJ3TnnkpAnd+ecS0Ke3J1zLgl5cnfOuSQUVnIXkT4islxEVorImFKOOV9ElorIEhF5LrJhupKUtZerc656K3cSk4jUBCYBvwNygXki8qqqLi10zGHAX4ATVfUHEdk/WgE7U7CX69atdrtgJyjYs2Klc676CqflfhywUlW/VNVfgWlA/2LH/AGYpKo/AKjq+siG6Yq76aY9ib3A1q12v3POhZPcWwJfF7qdG7qvsMOBw0XkvyLykYj0KelEIjJKRLJFJDsvL69yETug9L1cS7vfOVe9hJPcS5pkX3wR+FrAYUBvIB14XERS9vom1cmqmqqqqc2bN69orK6Q0vZyLe1+51z1Ek5yzwUOLnS7FVB8W4Zc4BVV3aGqXwHLsWTvoqSkvVzr17f7nXMunOQ+DzhMRNqJSB0gDXi12DEzgZMARKQZ1k3zZSQDdUWVtJfr5Mk+mOqcM+VWy6hqvohcAcwCagKZqrpERMYC2ar6auix00RkKbAT+JOqboxm4K7oXq7OOVdYWHXuqvqGqh6uqu1VdVzovltDiR0116lqB1U9WlWnRTNoF1+83t65+OObdbgq8Xp75+KTLz/gqsTr7Z2LT57cXZV4vb1z8cmTu6sSr7d3Lj55cndV4vX2zsUnT+6uSrze3rn45NUyrsq83t65+OMtd5c0vN7euT285e6SgtfbO1eUt9xdUvB6e+eK8uTukoLX2ztXlCd3lxS83t65ojy5u6Tg9fbOFeXJ3SUFr7d3riivlnFJw+vtndvDW+7OOZeEwkruItJHRJaLyEoRGVPGcYNEREUkNXIhOuecq6hyk7uI1AQmAX2BDkC6iHQo4biGwFXAx5EO0jnnXMWE03I/Dlipql+q6q/ANKB/Ccf9FbgH+CWC8TnnnKuEcJJ7S+DrQrdzQ/ftJiLHAAer6utlnUhERolItohk5+XlVThY55xz4QknuUsJ9+nuB0VqAPcD15d3IlWdrKqpqpravHnz8KN0zjlXIeEk91zg4EK3WwFrC91uCHQC/iMiq4AewKs+qOqcc8EJJ7nPAw4TkXYiUgdIA14teFBVN6tqM1Vtq6ptgY+As1U1OyoRO+ecK1e5yV1V84ErgFnAMmC6qi4RkbEicna0A3TOOVdxYc1QVdU3gDeK3XdrKcf2rnpYzjnnqsJnqDrnXBLy5O6cc0nIk7tzziUhT+7OOZeEPLk751wS8uTunHNJyJO7c84lIU/uzjmXhOJqm70dO3aQm5vLL7/4qsGJoF69erRq1YratWsHHYpzrpi4Su65ubk0bNiQtm3bIlLSYpQuXqgqGzduJDc3l3bt2gUdjismKwtuugnWrIHWrWHcON9ftrqJq26ZX375haZNm3piTwAiQtOmTf1dVhzKyoJRo2D1alC1z6NG2f2u+oir5A54Yk8g/ruKTzfdBFu3Fr1v61a731UfcZfcnXNVs2ZNxe53ySmhk3tWFrRtCzVq2Oeqvu3cuHEjXbt2pWvXrhxwwAG0bNly9+1ff/01rHMMHz6c5cuXl3nMpEmTyIrQe+SePXuyaNGiiJzLJYfWrSt2v0tOcTWgWhEF/YoFbz8L+hWh8gNHTZs23Z0ob7/9dho0aMAf//jHIseoKqpKjRolvy5OmTKl3OtcfvnllQvQuTCMG1f0fwOgfn2731UfCdtyj2W/4sqVK+nUqROjR4+mW7durFu3jlGjRpGamkrHjh0ZO3bs7mMLWtL5+fmkpKQwZswYunTpwgknnMD69esBuPnmm3nggQd2Hz9mzBiOO+44jjjiCD744AMAfv75Z84991y6dOlCeno6qamp5bbQn332WY4++mg6derEjTfeCEB+fj4XXHDB7vszMjIAuP/+++nQoQNdunRh2LBhEX/OXHCGDoXJk6FNGxCxz5Mne7VMdRNWy11E+gATgZrA46p6d7HHrwNGAvlAHvB7VV0d4ViLiHW/4tKlS5kyZQqPPPIIAHfffTf77bcf+fn5nHTSSQwaNIgOHToU+Z7NmzfTq1cv7r77bq677joyMzMZM2bMXudWVT755BNeffVVxo4dy1tvvcWDDz7IAQccwIsvvsinn35Kt27dyowvNzeXm2++mezsbBo3bsypp57K66+/TvPmzdmwYQOLFy8GYNOmTQDcc889rF69mjp16uy+zyWPoUM9mVd35bbcRaQmMAnoC3QA0kWkQ7HDFgKpqtoZmAHcE+lAi4t1v2L79u059thjd9+eOnUq3bp1o1u3bixbtoylS5fu9T377LMPffv2BaB79+6sWrWqxHMPHDhwr2Pef/990tLSAOjSpQsdO3YsM76PP/6Yk08+mWbNmlG7dm2GDBnCnDlzOPTQQ1m+fDlXX301s2bNonHjxgB07NiRYcOGkZWV5ZOQnEtC4XTLHAesVNUvVfVXYBrQv/ABqjpbVQs6ST4CWkU2zL2NG2f9iIVFs19x33333f31ihUrmDhxIu+++y45OTn06dOnxHrvOnXq7P66Zs2a5Ofnl3juunXr7nWMqlYovtKOb9q0KTk5OfTs2ZOMjAwuueQSAGbNmsXo0aP55JNPSE1NZefOnRW6nnMuvoWT3FsCXxe6nRu6rzQjgDdLekBERolItohk5+XlhR9lCYLsV/zxxx9p2LAhjRo1Yt26dcyaNSvi1+jZsyfTp08HYPHixSW+MyisR48ezJ49m40bN5Kfn8+0adPo1asXeXl5qCrnnXced9xxBwsWLGDnzp3k5uZy8sknc++995KXl8fW4gMYzrmEFk6fe0kzVUpsJorIMCAV6FXS46o6GZgMkJqaWrGmaQmC6lfs1q0bHTp0oFOnThxyyCGceOKJEb/GlVdeyYUXXkjnzp3p1q0bnTp12t2lUpJWrVoxduxYevfujarSr18/zjzzTBYsWMCIESNQVUSE8ePHk5+fz5AhQ/jpp5/YtWsXN9xwAw0bNoz4z+CcC46U9/ZfRE4AblfV00O3/wKgqncVO+5U4EGgl6quL+/Cqampmp2dXeS+ZcuWcdRRR1XoB0hW+fn55OfnU69ePVasWMFpp53GihUrqFUrvqpX/XfmXGyJyHxVTS3vuHAyxTzgMBFpB3wDpAFDil3sGOBRoE84id2Vb8uWLZxyyink5+ejqjz66KNxl9idc/Gr3D53Vc0HrgBmAcuA6aq6RETGisjZocPuBRoAL4jIIhF5NWoRVxMpKSnMnz+fTz/9lJycHE477bSgQ3JhivTMaecqI6ymoKq+AbxR7L5bC319aoTjci4hRWPmtHOVkbAzVJ2LR74io4sXntydiyBfkdHFC0/uzkWQr8jo4oUn90J69+6914SkBx54gMsuu6zM72vQoAEAa9euZdCgQaWeu3jpZ3EPPPBAkclEZ5xxRkTWfbn99tuZMGFClc/jyhfrmdPxzgeXg+PJvZD09HSmTZtW5L5p06aRnp4e1vcfdNBBzJgxo9LXL57c33jjDVJSUip9Phd7viLjHr7dX7DitnD6mmsg0ntQdO0KoZV2SzRo0CBuvvlmtm/fTt26dVm1ahVr166lZ8+ebNmyhf79+/PDDz+wY8cO7rzzTvr3L7LEDqtWreKss87is88+Y9u2bQwfPpylS5dy1FFHsW3btt3HXXrppcybN49t27YxaNAg7rjjDjIyMli7di0nnXQSzZo1Y/bs2bRt25bs7GyaNWvGfffdR2ZmJgAjR47kmmuuYdWqVfTt25eePXvywQcf0LJlS1555RX22WefUn/GRYsWMXr0aLZu3Ur79u3JzMykSZMmZGRk8Mgjj1CrVi06dOjAtGnTeO+997j66qsB21Jvzpw5PpM1DL4ioylrcNmfn+jzlnshTZs25bjjjuOtt94CrNU+ePBgRIR69erx8ssvs2DBAmbPns31119f5uJeDz/8MPXr1ycnJ4ebbrqJ+fPn735s3LhxZGdnk5OTw3vvvUdOTg5XXXUVBx10ELNnz2b27NlFzjV//nymTJnCxx9/zEcffcRjjz3GwoULAVvE7PLLL2fJkiWkpKTw4osvlvkzXnjhhYwfP56cnByOPvpo7rjjDsCWMF64cCE5OTm7lzWeMGECkyZNYtGiRcydO7fMFw3nivPB5WDFbcu9rBZ2NBV0zfTv359p06btbi2rKjfeeCNz5syhRo0afPPNN3z33XcccMABJZ5nzpw5XHXVVQB07tyZzp07735s+vTpTJ48mfz8fNatW8fSpUuLPF7c+++/zznnnLN7ZcqBAwcyd+5czj77bNq1a0fXrl2BspcVBltfftOmTfTqZUv/XHTRRZx33nm7Yxw6dCgDBgxgwIABAJx44olcd911DB06lIEDB9KqVdQX+3RJpHVr64op6f5Yy8qydwxr1tj1x41L/ncP3nIvZsCAAbzzzjssWLCAbdu27d4kIysri7y8PObPn8+iRYto0aJFicv8Fiay95prX331FRMmTOCdd94hJyeHM888s9zzlPUOoWC5YCh7WeHy/POf/+Tyyy9n/vz5dO/enfz8fMaMGcPjjz/Otm3b6NGjB59//nmlzu2qp3gZXI6nvv9YDjB7ci+mQYMG9O7dm9///vdFBlI3b97M/vvvT+3atZk9ezarS2qSFPLb3/529ybYn332GTk5OYAtF7zvvvvSuHFjvvvuO958c8/qyA0bNuSnn34q8VwzZ85k69at/Pzzz7z88sv85je/qfDP1rhxY5o0acLcuXMBeOaZZ+jVqxe7du3i66+/5qSTTuKee+5h06ZNbNmyhS+++IKjjz6aG264gdTUVE/urkLiZXA5XiaWxfpFJm67ZYKUnp7OwIEDi1TODB06lH79+pGamkrXrl058sgjyzzHpZdeyvDhw+ncuTNdu3bluOOOA2xXpWOOOYaOHTvutVzwqFGj6Nu3LwceeGCRfvdu3bpx8cUX7z7HyJEjOeaYY8rsginNU089tXtA9ZBDDmHKlCns3LmTYcOGsXnzZlSVa6+9lpSUFG655RZmz55NzZo16dChw+5dpZwLVzwMLsdL33+sB5jLXfI3WnzJ3+TgvzMX79q2Lbnvv00bqET7qNJq1LAWe3EisGtX+OcJd8lf75ZxziW1eOn7j/XsZU/uzrmkFi99/7F+kYm7PveC7eBc/AuqS8+5ioqHvv+C68eqJDOuknu9evXYuHEjTZs29QQf51SVjRs3Uq9evaBDcS5hxPJFJqzkLiJ9gIlATeBxVb272ON1gaeB7sBGYLCqrqpoMK1atSI3N5e8vLyKfqsLQL169Xxik3NxqtzkLiI1gUnA74BcYJ6IvKqqSwsdNgL4QVUPFZE0YDwwuKLB1K5dm3bt2lX025xzzhUTzoDqccBKVf1SVX8FpgH9ix3TH3gq9PUM4BTxfhXnnAtMOMm9JfB1odu5oftKPCa0ofZmoGnxE4nIKBHJFpFs73pxzrnoCSe5l9QCL14mEc4xqOpkVU1V1dTmzZuHE59zzrlKCGdANRc4uNDtVsDaUo7JFZFaQGPg+7JOOn/+/A0iUvYCLfGvGbAh6CDiiD8fRfnzsYc/F0VV5floE85B4ST3ecBhItIO+AZIA4YUO+ZV4CLgQ2AQ8K6WUwStqgnfdBeR7HCmAVcX/nwU5c/HHv5cFBWL56Pc5K6q+SJyBTALK4XMVNUlIjIWyFbVV4EngGdEZCXWYk+LZtDOOefKFladu6q+AbxR7L5bC339C3BeZENzzjlXWb62TNVMDjqAOOPPR1H+fOzhz0VRUX8+Alvy1znnXPR4y90555KQJ3fnnEtCntwrQUQOFpHZIrJMRJaIyNVBxxQ0EakpIgtF5PWgYwmaiKSIyAwR+Tz0N3JC0DEFSUSuDf2ffCYiU0WkWi0lKiKZIrJeRD4rdN9+IvIvEVkR+twk0tf15F45+cD1qnoU0AO4XEQ6BBxT0K4GlgUdRJyYCLylqkcCXajGz4uItASuAlJVtRNWTl3dSqWfBPoUu28M8I6qHga8E7odUZ7cK0FV16nqgtDXP2H/vMXX26k2RKQVcCbweNCxBE1EGgG/xeZ+oKq/quqmYKMKXC1gn9Ds9frsPcM9qanqHPaesV94scWngAGRvq4n9yoSkbbAMcDHwUYSqAeAPwMV2OY3aR0C5AFTQt1Uj4vIvkEHFRRV/QaYAKwB1gGbVfXtYKOKCy1UdR1YYxHYP9IX8OReBSLSAHgRuEZVfww6niCIyFnAelWdH3QscaIW0A14WFWPAX4mCm+5E0WoL7k/0A44CNhXRIYFG1X14Mm9kkSkNpbYs1T1paDjCdCJwNkisgpb6/9kEXk22JAClQvkqmrBO7kZWLKvrk4FvlLVPFXdAbwE/F/AMcWD70TkQIDQ5/WRvoAn90oIbUTyBLBMVe8LOp4gqepfVLWVqrbFBsreVdVq2zJT1W+Br0XkiNBdpwBLy/iWZLcG6CEi9UP/N6dQjQeYCylYbJHQ51cifYG42iA7gZwIXAAsFpFFoftuDK3B49yVQJaI1AG+BIYHHE9gVPVjEZkBLMCqzBZSzZYiEJGpQG+gmYjkArcBdwPTRWQE9gIY8bW5fPkB55xLQt4t45xzSciTu3POJSFP7s45l4Q8uTvnXBLy5O6cc0nIk7tzziUhT+7OOZeE/h8gi8reN4R8QgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The model quickly starts overfitting, unsurprisingly given the small number of training samples. Validation accuracy has high variance for \n",
    "the same reason, but seems to reach high 60s.\n",
    "\n",
    "Note that your mileage may vary: since we have so few training samples, performance is heavily dependent on which exact 200 samples we \n",
    "picked, and we picked them at random. If it worked really poorly for you, try picking a different random set of 200 samples, just for the \n",
    "sake of the exercise (in real life you don't get to pick your training data).\n",
    "\n",
    "We can also try to train the same model without loading the pre-trained word embeddings and without freezing the embedding layer. In that \n",
    "case, we would be learning a task-specific embedding of our input tokens, which is generally more powerful than pre-trained word embeddings \n",
    "when lots of data is available. However, in our case, we have only 200 training samples. Let's try it:\n",
    "\n",
    "模型很快就开始过拟合，考虑到训练样本很少，这一点也不奇怪。出于同样的原因，验证精度的波动很大，但似乎达到了接近 60%。\n",
    " \n",
    "注意，你的结果可能会有所不同。训练样本数太少，所以模型性能严重依赖于你选择的200个样本，而样本是随机选择的。如果你得到的结果很差，可以尝试重新选择 200 个不同的 随机样本，你可以将其作为练习（在现实生活中无法选择自己的训练数据）。\n",
    "\n",
    "你也可以在不加载预训练词嵌入、也不冻结嵌入层的情况下训练相同的模型。在这种情况下， 你将会学到针对任务的输入标记的嵌入。如果有大量的可用数据，这种方法通常比预训练词嵌入更加强大，但本例只有 200 个训练样本。我们来试一下这种方法：。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.6951 - acc: 0.4350 - val_loss: 0.6950 - val_acc: 0.5167\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.5028 - acc: 0.9800 - val_loss: 0.7054 - val_acc: 0.5069\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.2898 - acc: 0.9850 - val_loss: 0.7012 - val_acc: 0.5187\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.1183 - acc: 1.0000 - val_loss: 0.7166 - val_acc: 0.5156\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0524 - acc: 1.0000 - val_loss: 0.7150 - val_acc: 0.5288\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0261 - acc: 1.0000 - val_loss: 0.7249 - val_acc: 0.5260\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 0s 2ms/step - loss: 0.0141 - acc: 1.0000 - val_loss: 0.7211 - val_acc: 0.5389\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0082 - acc: 1.0000 - val_loss: 0.7390 - val_acc: 0.5267\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0049 - acc: 1.0000 - val_loss: 0.7283 - val_acc: 0.5393\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 0.0030 - acc: 1.0000 - val_loss: 0.7476 - val_acc: 0.5313\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X98FPW97/HXJ+Fn5KdJEE0gCRR/UoIxol5RsbYesQqt0irF0/qrtLZo7bX3XKrcalXa3tZ61OrpkdpaW1MoV2srPYqnUk6ptRWCGBA4CELACGJARBAEg5/7x8yGzbKbbMImmwzv5+Oxj8yP7858dpK8d/Y7szPm7oiISLTkZLsAERHJPIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMI9wsws18x2m9nQTLbNJjP7mJll/PxdM/ukmdXGja8xs3PSaduGdT1iZre29fki6eiW7QLkIDPbHTeaB+wDDoTjX3H3qtYsz90PAH0y3fZI4O4nZGI5ZnY9cJW7j4tb9vWZWLZIcxTunYi7N4ZruGd4vbs/n6q9mXVz94aOqE2kJfp77FzULdOFmNndZvZbM5ttZruAq8zsLDP7h5m9a2ZbzOwBM+setu9mZm5mpeH44+H8Z81sl5n93czKWts2nD/ezF4zs51m9hMz+5uZXZ2i7nRq/IqZrTOzHWb2QNxzc83sX81su5m9DlzUzPaZYWZzEqY9ZGb3hsPXm9nq8PW8Hu5Vp1pWnZmNC4fzzOzXYW0rgdOSrHd9uNyVZjYhnP5x4EHgnLDLa1vctr0j7vlfDV/7djP7vZkdm862ac12jtVjZs+b2Ttm9paZ/Uvcev5PuE3eM7NqMzsuWReYmb0Q+z2H23NRuJ53gBlmNsLMFoavZVu43frHPb8kfI314fz7zaxXWPNJce2ONbM9Zpaf6vVKC9xdj074AGqBTyZMuxvYD1xK8MbcGzgdOIPgU9gw4DVgWti+G+BAaTj+OLANqAS6A78FHm9D20HALmBiOO9/Ah8CV6d4LenU+AegP1AKvBN77cA0YCVQDOQDi4I/26TrGQbsBo6KW/bbQGU4fmnYxoBPAHuBUeG8TwK1ccuqA8aFw/cA/wUMBEqAVQltPw8cG/5OvhDWcEw473rgvxLqfBy4Ixy+MKxxNNAL+Dfgz+lsm1Zu5/7AVuAbQE+gHzAmnPdtoAYYEb6G0cDRwMcStzXwQuz3HL62BuAGIJfg7/F44AKgR/h38jfgnrjX82q4PY8K258dzpsFzIxbzy3AU9n+P+zKj6wXoEeKX0zqcP9zC8/7FvD/wuFkgf3vcW0nAK+2oe21wF/j5hmwhRThnmaNZ8bN/x3wrXB4EUH3VGzexYmBk7DsfwBfCIfHA6810/aPwNfD4ebCfVP87wL4WnzbJMt9Ffh0ONxSuD8GfC9uXj+C4yzFLW2bVm7nfwaqU7R7PVZvwvR0wn19CzVMApaEw+cAbwG5SdqdDWwALBx/Bbgs0/9XR9JD3TJdzxvxI2Z2opn9R/gx+z3gTqCgmee/FTe8h+YPoqZqe1x8HR78N9alWkiaNaa1LmBjM/UC/AaYHA5/AWg8CG1ml5jZS2G3xLsEe83NbauYY5urwcyuNrOasGvhXeDENJcLwetrXJ67vwfsAIri2qT1O2thOw8B1qWoYQhBwLdF4t/jYDOba2ZvhjX8MqGGWg8O3jfh7n8j+BQw1sxGAkOB/2hjTYL63LuixNMAHybYU/yYu/cDvkOwJ92ethDsWQJgZkbTMEp0ODVuIQiFmJZO1fwt8EkzKyboNvpNWGNv4Ang+wRdJgOA/0yzjrdS1WBmw4CfEnRN5IfL/e+45bZ02uZmgq6e2PL6EnT/vJlGXYma285vAMNTPC/VvPfDmvLipg1OaJP4+v4vwVleHw9ruDqhhhIzy01Rx6+Aqwg+Zcx1930p2kkaFO5dX19gJ/B+eEDqKx2wzj8CFWZ2qZl1I+jHLWynGucCN5tZUXhw7X8319jdtxJ0HTwKrHH3teGsngT9wPXAATO7hKBvON0abjWzARZ8D2Ba3Lw+BAFXT/A+dz3BnnvMVqA4/sBmgtnAdWY2ysx6Erz5/NXdU34SakZz2/lpYKiZTTOzHmbWz8zGhPMeAe42s+EWGG1mRxO8qb1FcOA+18ymEvdG1EwN7wM7zWwIQddQzN+B7cD3LDhI3dvMzo6b/2uCbpwvEAS9HAaFe9d3C/AlggOcDxPsubarMECvAO4l+GcdDiwj2GPLdI0/BRYAK4AlBHvfLfkNQR/6b+Jqfhf4JvAUwUHJSQRvUum4neATRC3wLHHB4+7LgQeAxWGbE4GX4p77J2AtsNXM4rtXYs+fT9B98lT4/KHAlDTrSpRyO7v7TuBTwOUEB3BfA84LZ/8I+D3Bdn6P4OBmr7C77cvArQQH1z+W8NqSuR0YQ/Am8zTwZFwNDcAlwEkEe/GbCH4Psfm1BL/n/e7+YitfuySIHbwQabPwY/ZmYJK7/zXb9UjXZWa/IjhIe0e2a+nq9CUmaRMzu4jgY/YHBKfSNRDsvYq0SXj8YiLw8WzXEgXqlpG2GgusJ/i4fhHwGR0Ak7Yys+8TnGv/PXfflO16okDdMiIiEaQ9dxGRCMpan3tBQYGXlpZma/UiIl3S0qVLt7l7c6ceA1kM99LSUqqrq7O1ehGRLsnMWvqWNqBuGRGRSFK4i4hEkMJdRCSCFO4iIhGkcBcRiaAWw93MfmFmb5vZqynmW3ibrXVmttzMKjJfpnRmVVVQWgo5OcHPqlbdxjtadXSGGlSH6gBavhMTcC5QQXgXniTzLya4Up4BZwIvpXOXkNNOO82l63v8cfe8PHc4+MjLC6YfaXV0hhpUR/TrIMUdtRIfad2uieDejanC/WFgctz4GuDYlpapcI+GkpKmf6yxR0nJkVdHZ6hBdUS/jnTDPRN97kU0vdVWHSnuymNmU8M7q1fX19dnYNWSbZtSXOIp1fQo19EZalAdqiMmE+Ge7DZlSa9G5u6z3L3S3SsLC1v89qx0AUNT3PQu1fQo19EZalAdqiMmE+FeR9P7SxYT3LhB2llnOEg0cybk5TWdlpcXTD/S6ugMNagO1dEonb4bmu9z/zRND6guTmeZ6nM/PJ3lIFGslpISd7PgZzZq6Cx1dIYaVEe06yDNPvcWr+duZrOBcUABwc1+bwe6h28M/25mBjxIcMOGPcA17t7iFcEqKyu9q144rKoKbrst6CsbOjR4553S1rtetlFpKWxMcvmgkhKore3YWkSk45jZUnevbKldi1eFdPfJLcx34OutqK1Lq6qCqVNhz55gfOPGYBw6NuA7y0EiEemc9A3VVrrttoPBHrNnTzC9I3WWg0Qi0jkp3Fups+wxd5aDRCLSOSncW6mz7DFPmQKzZgV97GbBz1mzOr7vX0Q6J4V7K3WmPeYpU4KDpx99FPxUsItIjMK9lbTHLCJdQdbuodqVTZmiMBeRzk177iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIiitcDezi8xsjZmtM7PpSeaXmNkCM1tuZv9lZsWZL1VERNLVYribWS7wEDAeOBmYbGYnJzS7B/iVu48C7gS+n+lCRUQkfensuY8B1rn7enffD8wBJia0ORlYEA4vTDJfREQ6UDrhXgS8ETdeF06LVwNcHg5/FuhrZvmJCzKzqWZWbWbV9fX1balXRETSkE64W5JpnjD+LeA8M1sGnAe8CTQc8iT3We5e6e6VhYWFrS5WRETS0y2NNnXAkLjxYmBzfAN33wxcBmBmfYDL3X1npooUEZHWSWfPfQkwwszKzKwHcCXwdHwDMysws9iyvg38IrNliohIa7QY7u7eAEwDngNWA3PdfaWZ3WlmE8Jm44A1ZvYacAwws53qFRGRNJh7Yvd5x6isrPTq6uqsrFtEpKsys6XuXtlSO31DVUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRlFa4m9lFZrbGzNaZ2fQk84ea2UIzW2Zmy83s4syXKiIi6Wox3M0sF3gIGA+cDEw2s5MTms0A5rr7qcCVwL9lulAREUlfOnvuY4B17r7e3fcDc4CJCW0c6BcO9wc2Z65EERFprXTCvQh4I268LpwW7w7gKjOrA54Bbky2IDObambVZlZdX1/fhnJFRCQd6YS7JZnmCeOTgV+6ezFwMfBrMztk2e4+y90r3b2ysLCw9dWKiEha0gn3OmBI3Hgxh3a7XAfMBXD3vwO9gIJMFCgiIq2XTrgvAUaYWZmZ9SA4YPp0QptNwAUAZnYSQbir30VEJEtaDHd3bwCmAc8BqwnOillpZnea2YSw2S3Al82sBpgNXO3uiV03IiLSQbql08jdnyE4UBo/7Ttxw6uAszNbmoiItJW+oSoiEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCEor3M3sIjNbY2brzGx6kvn/amavhI/XzOzdzJcqIiLp6tZSAzPLBR4CPgXUAUvM7Gl3XxVr4+7fjGt/I3BqO9QqIiJpSmfPfQywzt3Xu/t+YA4wsZn2k4HZmShORETaJp1wLwLeiBuvC6cdwsxKgDLgzynmTzWzajOrrq+vb22tIiKSpnTC3ZJM8xRtrwSecPcDyWa6+yx3r3T3ysLCwnRrFBGRVkon3OuAIXHjxcDmFG2vRF0yIiJZl064LwFGmFmZmfUgCPCnExuZ2QnAQODvmS1RRERaq8Vwd/cGYBrwHLAamOvuK83sTjObENd0MjDH3VN12YiISAdp8VRIAHd/BngmYdp3EsbvyFxZIiJyOPQNVRGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEUFrhbmYXmdkaM1tnZtNTtPm8ma0ys5Vm9pvMlikiIq3RraUGZpYLPAR8CqgDlpjZ0+6+Kq7NCODbwNnuvsPMBrVXwSIi0rJ09tzHAOvcfb277wfmABMT2nwZeMjddwC4+9uZLVNERFojnXAvAt6IG68Lp8U7HjjezP5mZv8ws4uSLcjMpppZtZlV19fXt61iERFpUTrhbkmmecJ4N2AEMA6YDDxiZgMOeZL7LHevdPfKwsLC1tYqIiJpSifc64AhcePFwOYkbf7g7h+6+wZgDUHYi4hIFqQT7kuAEWZWZmY9gCuBpxPa/B44H8DMCgi6adZnslAREUlfi+Hu7g3ANOA5YDUw191XmtmdZjYhbPYcsN3MVgELgf/l7tvbq2gREWmeuSd2n3eMyspKr66uzsq6RUS6KjNb6u6VLbXTN1RFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIR1C3bBYhI5n30Ebz1FuzaBUOHQu/e2a4oez74AGprYcMGyM2FwsKDj549s11d+1G4i3RRe/YEgbV+/cHH668HPzdsCEItZvBgKC0NHmVlTYeHDu3aIXfgALz5ZvCaY9sjfnjLltTP7dsXBg1qGviFhcmnFRZ2rTdJhbtIJ+Ue7H3Hh3b8IzG0+vSB4cPhxBPh4oth2LAgvDZuDPZca2th8WJ44gloaGj63OOOOzT0Y8NDhkCPHh3xipNzh3feOTS8Yz83boQPPzzYPicHiouD13/RRcHP2Otxh/r6po+33w5+btoES5cGw/HLi9enT/pvBIMGQV5eh2yipHQnJpEs2ru36d53fIhv2BDMjzELgnbYsEMfw4dDfn7QpiUHDsDmzcHyY6EfP/zGG0GbmJwcKCpKvedfXAzdDnM3ce/eYN2Je92x4ffea9q+oCBYd1nZwfCODR/um5E77NyZ+k0g2bT9+5MvKy8veehPmgRnnNG2+tK9E5PCXbqsjz4K9l5ff/3gY9OmIOB69oRevZr/2dp53bqlF57x3GHr1uR73q+/nnrvO1mAl5R0TPdJQwPU1SUP/g0bgnnxsZGbGwR8qj3/oqKg3Ztvpg7vxO3Qu3fq8C4rCz6RdBbuwbGNdN8I6uvhwQfhuuvatj6Fu0TCvn1BqMQHeOyR2K+cmxsEiVkwfd++gz8z8Week9Pym0Ls50cfBd0F69cfuvcd6zJIFuIFBa1/A+lo+/cHAZ9qz3/z5qbbO/ammNh1MmRI6vA+5pjOvx3ayj145LTxXMV0w1197pJ1O3cmD+/XXw+6COKDIi8vCMUTTgj6lYcPP/gYOhS6dz90+e7B3mhi4McPpzMvnWm7d8P27cE6R4yAf/qnpkHeUXvf7alHj4NvRsns2xd8goqF/oYNwfT48E71uzoSmHXMG5fCvQuLHWjaty/YY4w92rpH0F5i3SexrojEx/btTdsXFgZheM45TcN7+PC27dGZBUHSvXvn+jgfVT17Bm9sI0Zku5IjW5cL92XLgiP+xx138DFoUPCRPGo++ijor42d7bBx48FHbPz99w99Xo8eTcO+d++m4+0xrWfPQ/u/47tP4rsmcnKCPbfhw+Hyy5uG97Bh0K9fR21hkejqcuH+zDMwY0bTaTk5wXm88YGf7JGf37n2ahsagoNMiYEdG9606dCj8EcfHXy0P/54+NSnguGjjgq6BD74IAjR2HCq8R07giBONj/VKWCt1bt3ENYjRgSno8WCe/jwoOZsnlonciTocgdUGxqCvdnNm5t/bNt26HO7d4djjz0Y9vHD8Y+BAzPTJxbre0wW3hs3Bgel4k85g+BNqqTk4KO0tOl4e3crHDjQNOzTfcPYty/4BBXbAx88OLoHxLq6Dz/8kLq6Oj6IPxotnU6vXr0oLi6me8LBicgeUO3WLTgjInZ6VSr79gVfAEkW/Fu2wJo1sHBhsBebqGfPlj8FHHdc8CkgPqwTAzzx9K7YlytKSoL+5MTwHjo06OLIptzc4JPAUUdltw5pP3V1dfTt25fS0lJM78Cdkruzfft26urqKCsra9My0gp3M7sIuB/IBR5x9x8kzL8a+BHwZjjpQXd/pE0VZUjPngdDszl79wYhnOoTwPLlMH9+cB5rS3r0CAK6pATGjz80vIuKjtwzBKTz+OCDDxTsnZyZkZ+fT319fZuX0WK4m1ku8BDwKaAOWGJmT7v7qoSmv3X3aW2uJEt6927+tK6YXbsOfRNoaGga3oMHd64+fZFUFOyd3+H+jtLZcx8DrHP39eEK5wATgcRwj7S+fYPH8cdnuxIRkZals59ZBLwRN14XTkt0uZktN7MnzGxIsgWZ2VQzqzaz6sP5uCEiHaeqKviEmpMT/KyqOrzlbd++ndGjRzN69GgGDx5MUVFR4/j+VBdpSXDNNdewZs2aZts89NBDVB1usV1YOnvuyT4bJJ5iMw+Y7e77zOyrwGPAJw55kvssYBYEZ8u0slYR6WBVVTB1anB5YQhOFJg6NRieMqVty8zPz+eVV14B4I477qBPnz5861vfatLG3XF3clL0cz766KMtrufrX/962wqMiHT23OuA+D3xYmBzfAN33+7u+8LRnwGnZaY8Ecmm2247GOwxe/YE0zNt3bp1jBw5kq9+9atUVFSwZcsWpk6dSmVlJaeccgp33nlnY9uxY8fyyiuv0NDQwIABA5g+fTrl5eWcddZZvP322wDMmDGD++67r7H99OnTGTNmDCeccAIvvvgiAO+//z6XX3455eXlTJ48mcrKysY3nni33347p59+emN9sVPIX3vtNT7xiU9QXl5ORUUFtbW1AHzve9/j4x//OOXl5dzWHhsrDemE+xJghJmVmVkP4Erg6fgGZnZs3OgEYHXmShSRbNm0qXXTD9eqVau47rrrWLZsGUVFRfzgBz+gurqampoa/vSnP7Fq1aGH+nbu3Ml5551HTU0NZ511Fr/4xS+SLtvdWbx4MT/60Y8a3yh+8pOfMHjwYGpqapg+fTrLli1L+txvfOMbLFmyhBUrVrBz507mz58PwOTJk/nmN79JTU0NL774IoMGDWLevHk8++yzLF68mJqaGm655ZYMbZ3WaTHc3b0BmAY8RxDac919pZndaWYTwmY3mdlKM6sBbgKubq+CRaTjDB3auumHa/jw4Zx++umN47Nnz6aiooKKigpWr16dNNx79+7N+PHjATjttNMa954TXXbZZYe0eeGFF7jyyisBKC8v55RTTkn63AULFjBmzBjKy8v5y1/+wsqVK9mxYwfbtm3j0ksvBYIvHeXl5fH8889z7bXX0ju8bdPRRx/d+g2RAWmd5+7uzwDPJEz7Ttzwt4FvZ7Y0Ecm2mTOb9rlDcGXOmTPbZ31HxX17bu3atdx///0sXryYAQMGcNVVVyX9Vm2PuGtZ5Obm0pB4m6lQz/BynPFt0vmG/p49e5g2bRovv/wyRUVFzJgxo7GOZKcrununONVUZ2WLSEpTpsCsWcH3OMyCn7Nmtf1gamu899579O3bl379+rFlyxaee+65jK9j7NixzJ07F4AVK1Yk/WSwd+9ecnJyKCgoYNeuXTz55JMADBw4kIKCAubNmwcEXw7bs2cPF154IT//+c/ZG14t75133sl43enocpcfEJGONWVKx4R5ooqKCk4++WRGjhzJsGHDOPvsszO+jhtvvJEvfvGLjBo1ioqKCkaOHEn//v2btMnPz+dLX/oSI0eOpKSkhDPi7o9XVVXFV77yFW677TZ69OjBk08+ySWXXEJNTQ2VlZV0796dSy+9lLvuuivjtbeky104TEQOz+rVqznppJOyXUan0NDQQENDA7169WLt2rVceOGFrF27lm6He1PYDEn2u4rshcNERDJl9+7dXHDBBTQ0NODuPPzww50m2A9XNF6FiEgbDBgwgKVLl2a7jHahA6oiIhGkcBcRiSCFu4hIBCncRUQiSOEuIh1q3Lhxh3wh6b777uNrX/tas8/r06cPAJs3b2bSpEkpl93SKdb33Xcfe+K+cnvxxRfz7rvvplN6l6JwF5EONXnyZObMmdNk2pw5c5g8eXJazz/uuON44okn2rz+xHB/5plnGDBgQJuX11npVEiRI9jNN0OSK9weltGjIbzSblKTJk1ixowZ7Nu3j549e1JbW8vmzZsZO3Ysu3fvZuLEiezYsYMPP/yQu+++m4kTJzZ5fm1tLZdccgmvvvoqe/fu5ZprrmHVqlWcdNJJjV/5B7jhhhtYsmQJe/fuZdKkSXz3u9/lgQceYPPmzZx//vkUFBSwcOFCSktLqa6upqCggHvvvbfxqpLXX389N998M7W1tYwfP56xY8fy4osvUlRUxB/+8IfGC4PFzJs3j7vvvpv9+/eTn59PVVUVxxxzDLt37+bGG2+kuroaM+P222/n8ssvZ/78+dx6660cOHCAgoICFixYkLlfAgp3Eelg+fn5jBkzhvnz5zNx4kTmzJnDFVdcgZnRq1cvnnrqKfr168e2bds488wzmTBhQsoLcf30pz8lLy+P5cuXs3z5cioqKhrnzZw5k6OPPpoDBw5wwQUXsHz5cm666SbuvfdeFi5cSEFBQZNlLV26lEcffZSXXnoJd+eMM87gvPPOY+DAgaxdu5bZs2fzs5/9jM9//vM8+eSTXHXVVU2eP3bsWP7xj39gZjzyyCP88Ic/5Mc//jF33XUX/fv3Z8WKFQDs2LGD+vp6vvzlL7No0SLKysra5fozCneRI1hze9jtKdY1Ewv32N6yu3PrrbeyaNEicnJyePPNN9m6dSuDBw9OupxFixZx0003ATBq1ChGjRrVOG/u3LnMmjWLhoYGtmzZwqpVq5rMT/TCCy/w2c9+tvHKlJdddhl//etfmTBhAmVlZYwePRpIfVnhuro6rrjiCrZs2cL+/fspKysD4Pnnn2/SDTVw4EDmzZvHueee29imPS4L3KX63DN9L0cRyY7PfOYzLFiwgJdffpm9e/c27nFXVVVRX1/P0qVLeeWVVzjmmGOSXuY3XrK9+g0bNnDPPfewYMECli9fzqc//ekWl9PcdbZilwuG1JcVvvHGG5k2bRorVqzg4YcfblxfsksAd8RlgbtMuMfu5bhxI7gfvJejAl6k6+nTpw/jxo3j2muvbXIgdefOnQwaNIju3buzcOFCNm7c2Oxyzj333MabYL/66qssX74cCC4XfNRRR9G/f3+2bt3Ks88+2/icvn37smvXrqTL+v3vf8+ePXt4//33eeqppzjnnHPSfk07d+6kqKgIgMcee6xx+oUXXsiDDz7YOL5jxw7OOuss/vKXv7BhwwagfS4L3GXCvSPv5Sgi7W/y5MnU1NQ03gkJYMqUKVRXV1NZWUlVVRUnnnhis8u44YYb2L17N6NGjeKHP/whY8aMAYK7Kp166qmccsopXHvttU0uFzx16lTGjx/P+eef32RZFRUVXH311YwZM4YzzjiD66+/nlNPPTXt13PHHXfwuc99jnPOOadJf/6MGTPYsWMHI0eOpLy8nIULF1JYWMisWbO47LLLKC8v54orrkh7PenqMpf8zckJ9tgTmcFHH2WwMJGI0yV/u47DueRvl9lz7+h7OYqIdGVdJtxnzgzu3RivPe/lKCLSlXWZcM/mvRxFoiZb3bGSvsP9HXWp89yzdS9HkSjp1asX27dvJz8/v91Px5O2cXe2b99Or1692ryMLhXuInL4iouLqauro74xqklEAAADKUlEQVS+PtulSDN69epFcXFxm5+vcBc5wnTv3r3xm5ESXV2mz11ERNKncBcRiSCFu4hIBGXtG6pmVg80f+GIzq8A2JbtIjoRbY+DtC2a0vZo6nC2R4m7F7bUKGvhHgVmVp3O14CPFNoeB2lbNKXt0VRHbA91y4iIRJDCXUQkghTuh2dWtgvoZLQ9DtK2aErbo6l23x7qcxcRiSDtuYuIRJDCXUQkghTubWBmQ8xsoZmtNrOVZvaNbNeUbWaWa2bLzOyP2a4l28xsgJk9YWb/Hf6NnJXtmrLJzL4Z/p+8amazzaztlzrsYszsF2b2tpm9GjftaDP7k5mtDX8ObI91K9zbpgG4xd1PAs4Evm5mJ2e5pmz7BrA620V0EvcD8939RKCcI3i7mFkRcBNQ6e4jgVzgyuafFSm/BC5KmDYdWODuI4AF4XjGKdzbwN23uPvL4fAugn/eouxWlT1mVgx8Gngk27Vkm5n1A84Ffg7g7vvd/d3sVpV13YDeZtYNyAM2Z7meDuPui4B3EiZPBB4Lhx8DPtMe61a4HyYzKwVOBV7KbiVZdR/wL4BuVQ7DgHrg0bCb6hEzOyrbRWWLu78J3ANsArYAO939P7NbVdYd4+5bINhRBAa1x0oU7ofBzPoATwI3u/t72a4nG8zsEuBtd1+a7Vo6iW5ABfBTdz8VeJ92+tjdFYT9yROBMuA44Cgzuyq7VR0ZFO5tZGbdCYK9yt1/l+16suhsYIKZ1QJzgE+Y2ePZLSmr6oA6d499knuCIOyPVJ8ENrh7vbt/CPwO+B9ZrinbtprZsQDhz7fbYyUK9zaw4MaTPwdWu/u92a4nm9z92+5e7O6lBAfK/uzuR+yembu/BbxhZieEky4AVmWxpGzbBJxpZnnh/80FHMEHmENPA18Kh78E/KE9VqLb7LXN2cA/AyvM7JVw2q3u/kwWa5LO40agysx6AOuBa7JcT9a4+0tm9gTwMsFZZss4gi5FYGazgXFAgZnVAbcDPwDmmtl1BG9+n2uXdevyAyIi0aNuGRGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQi6P8DpZXynsTQLtMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X+clXWd9/HXm18CQmAwpoIwoIiiIuLE2kqBZi2kghkVOFa6eU+2mpZttyTWKnc81sw7jW7uanJ1u3OKvHUttkx2V9nUrZQBCQUiUEFGSEcSUEFl4LN/XGeYM8OZmTM/z+Hi/Xw8zuNc13W+57o+5wy8r+t8r1+KCMzMLF16FLoAMzPrfA53M7MUcribmaWQw93MLIUc7mZmKeRwNzNLIYe75SSpp6Q3JI3ozLaFJOlESZ1+7K+k8yVtyhpfL+n9+bRtx7LuknRje9/fwny/IemfO3u+Vji9Cl2AdQ5Jb2SN9gfeBvZlxj8XEVVtmV9E7AMGdHbbw0FEjO2M+Ui6ErgsIqZmzfvKzpi3pZ/DPSUi4kC4ZrYMr4yI/2iuvaReEVHXHbWZWfdzt8xhIvOz+2eSfirpdeAySe+T9HtJOyRtk7RQUu9M+16SQlJpZvzezOu/lvS6pN9JGtXWtpnXp0v6k6Sdkr4r6b8kXd5M3fnU+DlJGyW9Jmlh1nt7SrpD0nZJzwHTWvh+bpK0uMm0RZK+nRm+UtK6zOd5LrNV3dy8aiRNzQz3l/TjTG1rgLNyLPf5zHzXSJqRmX468H+A92e6vF7N+m5vznr/VZnPvl3SzyUdm8930xpJF2fq2SHpUUljs167UdJWSbsk/THrs54taWVm+suSvpXv8qwLRIQfKXsAm4Dzm0z7BvAOcBHJSr0f8F7gr0h+wY0G/gRck2nfCwigNDN+L/AqUAb0Bn4G3NuOtkcDrwMzM69dD+wFLm/ms+RT4y+AQUAp8Jf6zw5cA6wBhgNDgMeSf/I5lzMaeAM4MmverwBlmfGLMm0EnAfsAcZnXjsf2JQ1rxpgamb4duA/gaOAkcDaJm0/ARyb+ZtcmqnhPZnXrgT+s0md9wI3Z4Y/nKlxAtAX+L/Ao/l8Nzk+/zeAf84Mn5Kp47zM3+jGzPfeGzgV2Awck2k7ChidGV4OzMkMDwT+qtD/Fw7nh7fcDy9PRMS/RsT+iNgTEcsj4smIqIuI54FKYEoL778/IqojYi9QRRIqbW17IbAqIn6Ree0OkhVBTnnW+I8RsTMiNpEEaf2yPgHcERE1EbEduLWF5TwPPEuy0gH4ELAjIqozr/9rRDwfiUeBR4CcO02b+ATwjYh4LSI2k2yNZy/3vojYlvmb/IRkxVyWx3wByoG7ImJVRLwFzAWmSBqe1aa576Yls4ElEfFo5m90K/AukpVsHcmK5NRM194Lme8OkpX0GElDIuL1iHgyz89hXcDhfnjZkj0i6WRJv5L0Z0m7gPnA0Bbe/+es4d20vBO1ubbHZdcREUGypZtTnjXmtSySLc6W/ASYkxm+lGSlVF/HhZKelPQXSTtItppb+q7qHdtSDZIul/SHTPfHDuDkPOcLyec7ML+I2AW8BgzLatOWv1lz891P8jcaFhHrgS+T/B1eyXTzHZNpegUwDlgv6SlJH8nzc1gXcLgfXpoeBvgDkq3VEyPiXcDXSbodutI2km4SACSJxmHUVEdq3AYcnzXe2qGaPwPOz2z5ziQJeyT1A+4H/pGky2Qw8G951vHn5mqQNBr4HvB5YEhmvn/Mmm9rh21uJenqqZ/fQJLun5fyqKst8+1B8jd7CSAi7o2Ic0i6ZHqSfC9ExPqImE3S9fa/gQck9e1gLdZODvfD20BgJ/CmpFOAz3XDMn8JTJR0kaRewHVASRfVeB/wRUnDJA0BbmipcUS8DDwB3AOsj4gNmZeOAPoAtcA+SRcCH2xDDTdKGqzkPIBrsl4bQBLgtSTruStJttzrvQwMr9+BnMNPgc9KGi/pCJKQfTwimv0l1IaaZ0iamln2V0j2kzwp6RRJ52aWtyfz2EfyAT4laWhmS39n5rPt72At1k4O98Pbl4HPkPzH/QHJlmuXygToJ4FvA9uBE4CnSY7L7+wav0fSN/4Myc6++/N4z09IdpD+JKvmHcCXgAdJdkrOIllJ5eMfSH5BbAJ+Dfy/rPmuBhYCT2XanAxk91P/O7ABeFlSdvdK/fsfJukeeTDz/hEk/fAdEhFrSL7z75GseKYBMzL970cAt5HsJ/kzyS+FmzJv/QiwTsnRWLcDn4yIdzpaj7WPki5Ps8KQ1JOkG2BWRDxe6HrM0sJb7tbtJE2TNCjz0/5rJEdgPFXgssxSxeFuhTAZeJ7kp/004OKIaK5bxszawd0yZmYp5C13M7MUKtiFw4YOHRqlpaWFWryZ2SFpxYoVr0ZES4cPAwUM99LSUqqrqwu1eDOzQ5Kk1s60BtwtY2aWSg53M7MUcribmaWQw93MLIUc7mZmKeRwNzNLIYe7mVkKFew4dzOzw8Wbb8Kf/gTr1yfPF14IEyd27TId7mZmnWDfPnjxxSTAsx9/+hPUZN0+RYKjj3a4m9kh7M034bnn4K23kkA7+mjo37/QVXXMX/7SOLjrhzduhLezrm06eDCMHQvnnQcnnZQMjx0LJ54I/fp1fZ0OdzPrkF27kgDfuPHgx9atB7fv378h6EtKWh4uKYEjjuj+z/TOO8lnaroFvn49vPpqQ7teveCEE5LQ/shHkuf6IC8pSbbSC8Xhbmateu213OG9cSO88krjtscem2yd/s3fJM/1W6q1tUnbV15pGH7pJVi1Khneuzf3sgcNan0lUD88dGgSuPmIgG3bDt4CX78eXngB9mfd/fWYY5LA/uhHG7bAx46F0lLo3dwdbgvM4W5mRMD27c0H+PbtjdsPH56E9owZDQF+4onJVuyAAe1b/s6duVcA2cMbN8LvfpeM72/m1ttDhuReAQwdmnyO7B2br7/e8L5+/ZKt7rPOgksvbdgCP+mkZAVzqHG4m3XA7t1JYLz6auPHrl3Qp0/SpdC3b/Joy/ARR0DPnp1ba0RDQG7cCBs2NA7wnTsb2kowYkQS2LNmNYT3mDEwenTn9xlLSR/14MHJMlqzf3/S993ayuDZZ5Ph+pVT/ecaOxYuv7xhC/ykk5IVVo8UHRzucDfLePvt3EHd0mPPnq6rp3fv9q8c6offeqtxf/gbbzTMv0ePpFvhxBPh7LMbb4GPGlWYvu589eiRbIkPHQqnnNJ6+7q65O81aFD37MwsBg5365C3304O/9qyJekz7dmz8aNXr5bH823T1h1Te/cmW3ZtCers4Gtq0KCGMDnuOBg/vmG86WPIkKT93r1JuL79dvLcFcO7djXf5p13ku9y9OgksD/wgcYBPnJk8uvicNCrV9JvfjhxuFuLdu2CzZubf/z5z91TR48e+a0QpGTnX3YXQ1MDBjQO45NPbjmohwxp306zPn3gyCPb/5k7qr5POk1dDZY/h/thLCLZYs0O602bGo/v2NH4PX36JH2WI0cmh36NHJk8RoxIugL27Ut+Au/b1/BoOp5Pm/a8Z9++JNAGD245qPv2LcjX3e0c6oe3vMJd0jTgO0BP4K6IuLXJ63cA52ZG+wNHR8Tgziy0GNT329XWJs9vvdXQv9mv38HD/folW3yFOtZ1377kOOOWtryb9hkPHNgQ2Oec0zBc/3jPexwaZoeCVsNdUk9gEfAhoAZYLmlJRKytbxMRX8pq/wXgzC6olaoqmDcv6eMdMQIWLIDy8vbPb/fuhrDOfjSdVj/+2mttX4Z0cOC3tDJoy7T65x07cgd3TU2yQso2dGgS0uPGwfTpDaFdWpo8Dx5c2BMvzKxz5LPlPgnYGBHPA0haDMwE1jbTfg7wD51TXoOqKqioSAIZkvCqqEiGy8sbHyebT2DX1jbMq6levZIQrD8pYsKEZLh+vH64X79kB9aePQ07seqHW5qW/dqbbyZHaORqn30qcz569Eh29o0cCX/91wdvdY8YUdg+YDPrPvmE+zBgS9Z4DfBXuRpKGgmMAh5t5vUKoAJgxIgRbSp03ryDw3j3brjiCvjKV5Lwbu4Mt379GgK5pCQ5dCo7pJsGd7Fsve7f33D0Q3MrjT17GrpShg8v3rPlzKx75RPuuWIummk7G7g/IvblejEiKoFKgLKysubmkdOLL+aevndvsmMv15Z1/eNQvVBRjx7JiqlfPzjqqEJXY2aHknzCvQY4Pmt8OJDjckBAEu5Xd7SoXEaMSLpimho5Eu66qyuWaGZ26MrnuIflwBhJoyT1IQnwJU0bSRoLHAX8rnNLTCxYcPAWeP/+yXQzM2us1XCPiDrgGmApsA64LyLWSJovaUZW0znA4ohoU3dLvsrLobIy2VKXkufKyo4dLWNmllbqoixuVVlZWVRXVxdk2WZmhypJKyKirLV2Ph3FzCyFHO5mZinkcDczSyGHu5lZCjnczcxSyOFuZpZCDnczsxRyuJuZpZDD3cwshRzuZmYp5HA3M0shh7uZWQo53M3MUsjhbmaWQg53M7MUcribmaWQw93MLIXyCndJ0yStl7RR0txm2nxC0lpJayT9pHPLNDOztujVWgNJPYFFwIeAGmC5pCURsTarzRjgq8A5EfGapKO7qmAzM2tdPlvuk4CNEfF8RLwDLAZmNmnzP4BFEfEaQES80rllmplZW+QT7sOALVnjNZlp2U4CTpL0X5J+L2larhlJqpBULam6tra2fRWbmVmr8gl35ZgWTcZ7AWOAqcAc4C5Jgw96U0RlRJRFRFlJSUlbazUzszzlE+41wPFZ48OBrTna/CIi9kbEC8B6krA3M7MCyCfclwNjJI2S1AeYDSxp0ubnwLkAkoaSdNM835mFmplZ/loN94ioA64BlgLrgPsiYo2k+ZJmZJotBbZLWgssA74SEdu7qmgzM2uZIpp2n3ePsrKyqK6uLsiyzcwOVZJWRERZa+18hqqZWQo53M3MUsjhbmaWQg53M7MUcribmaWQw93MLIUc7mZmKeRwNzNLIYe7mVkKOdzNzFLI4W5mlkIOdzOzFHK4m5mlkMPdzCyFHO5mZinkcDczSyGHu5lZCjnc26GqCkpLoUeP5LmqqtAVmZk1lle4S5omab2kjZLm5nj9ckm1klZlHld2fqnFoaoKKipg82aISJ4rKhzwZlZcWg13ST2BRcB0YBwwR9K4HE1/FhETMo+7OrnOojFvHuze3Xja7t3JdDOzYpHPlvskYGNEPB8R7wCLgZldW1bxevHFtk03MyuEfMJ9GLAla7wmM62pj0laLel+ScfnmpGkCknVkqpra2vbUW7hjRjRtulmZoWQT7grx7RoMv6vQGlEjAf+A/hRrhlFRGVElEVEWUlJSdsqLRILFkD//o2n9e+fTDczKxb5hHsNkL0lPhzYmt0gIrZHxNuZ0R8CZ3VOecWnvBwqK2HkSJCS58rKZLqZWbHolUeb5cAYSaOAl4DZwKXZDSQdGxHbMqMzgHWdWmWRKS93mJtZcWs13COiTtI1wFKgJ3B3RKyRNB+ojoglwLWSZgB1wF+Ay7uwZjMza4Uimnafd4+ysrKorq4uyLLNzA5VklZERFlr7XyGqplZCjnczcxSyOFuZpZCDnczsxRyuJuZpZDD3cwshRzuZmYp5HA3M0shh7uZWQo53M3MUsjhbmaWQg53M7MUcribmaWQw93MLIUc7mZmKeRwNzNLIYe7mVkK5RXukqZJWi9po6S5LbSbJSkktXqXEDMz6zqthruknsAiYDowDpgjaVyOdgOBa4EnO7tIMzNrm3y23CcBGyPi+Yh4B1gMzMzR7n8BtwFvdWJ9ZmbWDvmE+zBgS9Z4TWbaAZLOBI6PiF92Ym1mZtZO+YS7ckyLAy9KPYA7gC+3OiOpQlK1pOra2tr8qzQzszbJJ9xrgOOzxocDW7PGBwKnAf8paRNwNrAk107ViKiMiLKIKCspKWl/1WZm1qJ8wn05MEbSKEl9gNnAkvoXI2JnRAyNiNKIKAV+D8yIiOouqdjMzFrVarhHRB1wDbAUWAfcFxFrJM2XNKOrCzQzs7brlU+jiHgIeKjJtK8303Zqx8syM7OO8BmqZmYp5HA3M0shh7uZWQo53M3MUsjhbmaWQg53M7MUcribmaWQw93MLIUc7mZmKeRwNzNLIYe7mVkKOdzNzFLI4W5mlkIOdzOzFHK4m5mlkMPdzCyFHO5mZinkcDczS6G8wl3SNEnrJW2UNDfH61dJekbSKklPSBrX+aWamVm+Wg13ST2BRcB0YBwwJ0d4/yQiTo+ICcBtwLc7vVIzM8tbPlvuk4CNEfF8RLwDLAZmZjeIiF1Zo0cC0XklmplZW+UT7sOALVnjNZlpjUi6WtJzJFvu1+aakaQKSdWSqmtra9tTr2WpqoLSUujRI3muqip0RWZWLPIJd+WYdtCWeUQsiogTgBuAm3LNKCIqI6IsIspKSkraVqk1UlUFFRWweTNEJM8VFQ54M0vkE+41wPFZ48OBrS20Xwxc3JGirHXz5sHu3Y2n7d6dTDczyyfclwNjJI2S1AeYDSzJbiBpTNboBcCGzivRcnnxxbZNN7PDS6/WGkREnaRrgKVAT+DuiFgjaT5QHRFLgGsknQ/sBV4DPtOVRRuMGJF0xeSabmbWargDRMRDwENNpn09a/i6Tq7LWrFgQdLHnt01079/Mt3MzGeoHqLKy6GyEkaOBCl5rqxMppuZ5bXlbsWpvNxhbma5ecvdzCyFHO5mZinkcDczSyGHu5lZCjnczcxSyOFuZpZCDnczsxRyuJuZpZDD3cwshRzuZmYp5HA3M0shh7uZWQo53M3MUsjhbmaWQg53M7MUyivcJU2TtF7SRklzc7x+vaS1klZLekTSyM4v1czM8tVquEvqCSwCpgPjgDmSxjVp9jRQFhHjgfuB2zq7UDMzy18+W+6TgI0R8XxEvAMsBmZmN4iIZRFRfzfP3wPDO7dMMzNri3zCfRiwJWu8JjOtOZ8Fft2RoszMrGPyuYeqckyLnA2ly4AyYEozr1cAFQAjRozIs0QzM2urfLbca4Djs8aHA1ubNpJ0PjAPmBERb+eaUURURkRZRJSVlJS0p14zM8tDPuG+HBgjaZSkPsBsYEl2A0lnAj8gCfZXOr9MMzNri1bDPSLqgGuApcA64L6IWCNpvqQZmWbfAgYA/1/SKklLmpmdmZl1g3z63ImIh4CHmkz7etbw+Z1cl5mZdYDPUDUzSyGHu5lZCjnczcxSyOFuZpZCDnczsxRyuJuZpZDD3cwshRzuZmYp5HA3M0shh7uZWQo53M3MUsjhbmaWQg53M7MUcrhbh1VVQWkp9OiRPFdVFboiM8vrkr9mzamqgooK2J25Pfrmzck4QHl54eoyO9x5y906ZN68hmCvt3t3Mt3MCsfhbh3y4ottm25m3cPhbh0yYkTbpptZ98gr3CVNk7Re0kZJc3O8/gFJKyXVSZrV+WVasVqwAPr3bzytf/9kupkVTqs7VCX1BBYBHwJqgOWSlkTE2qxmLwKXA3/fkWL27t1LTU0Nb731VkdmY92kb9++fOITw4HezJuXdMWMGJEEu3emmhVWPkfLTAI2RsTzAJIWAzOBA+EeEZsyr+3vSDE1NTUMHDiQ0tJSJHVkVtbFIoLt27dTU1NDefkoh7lZkcmnW2YYsCVrvCYzrc0kVUiqllRdW1t70OtvvfUWQ4YMcbAfAiQxZMgQ/8oyK1L5hHuupI32LCwiKiOiLCLKSkpKci/MwX7I8N/KrHjlE+41wPFZ48OBrV1TjpmZdYZ8wn05MEbSKEl9gNnAkq4tKz+dfdr79u3bmTBhAhMmTOCYY45h2LBhB8bfeeedvOZxxRVXsH79+hbbLFq0iKpOOkd/8uTJrFq1qlPmZWbp0eoO1Yiok3QNsBToCdwdEWskzQeqI2KJpPcCDwJHARdJuiUiTu3KwrvitPchQ4YcCMqbb76ZAQMG8Pd/3/gAoIggIujRI/d68Z577ml1OVdffXX7CjQzy1Nex7lHxEMRcVJEnBARCzLTvh4RSzLDyyNieEQcGRFDujrYoXtPe9+4cSOnnXYaV111FRMnTmTbtm1UVFRQVlbGqaeeyvz58w+0rd+SrqurY/DgwcydO5czzjiD973vfbzyyisA3HTTTdx5550H2s+dO5dJkyYxduxYfvvb3wLw5ptv8rGPfYwzzjiDOXPmUFZW1uoW+r333svpp5/Oaaedxo033ghAXV0dn/rUpw5MX7hwIQB33HEH48aN44wzzuCyyy7r9O/MzArrkL1wWHef9r527Vruuecevv/97wNw66238u53v5u6ujrOPfdcZs2axbhx4xq9Z+fOnUyZMoVbb72V66+/nrvvvpu5cw86B4yI4KmnnmLJkiXMnz+fhx9+mO9+97scc8wxPPDAA/zhD39g4sSJLdZXU1PDTTfdRHV1NYMGDeL888/nl7/8JSUlJbz66qs888wzAOzYsQOA2267jc2bN9OnT58D08wsPQ7Zyw9092nvJ5xwAu9973sPjP/0pz9l4sSJTJw4kXXr1rF27dqD3tOvXz+mT58OwFlnncWmTZtyzvuSSy45qM0TTzzB7NmzATjjjDM49dSWfww9+eSTnHfeeQwdOpTevXtz6aWX8thjj3HiiSeyfv16rrvuOpYuXcqgQYMAOPXUU7nsssuoqqqid+/ebfouzKz4HbLh3t2nvR955JEHhjds2MB3vvMdHn30UVavXs20adNyHu/dp0+fA8M9e/akrq4u57yPOOKIg9pEtO1o0+baDxkyhNWrVzN58mQWLlzI5z73OQCWLl3KVVddxVNPPUVZWRn79u1r0/LMrLgdsuFeXg6VlTByJEjJc2Vl95z2vmvXLgYOHMi73vUutm3bxtKlSzt9GZMnT+a+++4D4Jlnnsn5yyDb2WefzbJly9i+fTt1dXUsXryYKVOmUFtbS0Tw8Y9/nFtuuYWVK1eyb98+ampqOO+88/jWt75FbW0tu5vuwDCzQ9oh2+cOSZAX4rT3iRMnMm7cOE477TRGjx7NOeec0+nL+MIXvsCnP/1pxo8fz8SJEznttNMOdKnkMnz4cObPn8/UqVOJCC666CIuuOACVq5cyWc/+1kiAkl885vfpK6ujksvvZTXX3+d/fv3c8MNNzBw4MBO/wzdraoKX+PGLENt/fnfWcrKyqK6urrRtHXr1nHKKacUpJ5iU1dXR11dHX379mXDhg18+MMfZsOGDfTqVVzr42L5mzU9NBaSbrru+jVn1l0krYiIstbaFVdS2AFvvPEGH/zgB6mrqyMi+MEPflB0wV5MWjo01uFuhyOnRZEaPHgwK1asKHQZhwzfEcqssUN2h6pZNt8Ryqwxh7ulgu8IZdaYw91SoZCHxpoVI4e7pUZ5OWzaBPv3J8+FCPbOvlKpWXs53LNMnTr1oBOS7rzzTv7u7/6uxfcNGDAAgK1btzJrVu77g0+dOpWmh342deeddzY6megjH/lIp1z35eabb+b222/v8HysZfWHY27eDBENVyp1wFshONyzzJkzh8WLFzeatnjxYubMmZPX+4877jjuv//+di+/abg/9NBDDB48uN3zs+7VnVcqNWtN0R4K+cUvQmffg2LCBMhcaTenWbNmcdNNN/H2229zxBFHsGnTJrZu3crkyZN54403mDlzJq+99hp79+7lG9/4BjNnzmz0/k2bNnHhhRfy7LPPsmfPHq644grWrl3LKaecwp49ew60+/znP8/y5cvZs2cPs2bN4pZbbmHhwoVs3bqVc889l6FDh7Js2TJKS0uprq5m6NChfPvb3+buu+8G4Morr+SLX/wimzZtYvr06UyePJnf/va3DBs2jF/84hf069ev2c+4atUqrrrqKnbv3s0JJ5zA3XffzVFHHcXChQv5/ve/T69evRg3bhyLFy/mN7/5Dddddx2Q3FLvscceS8WZrF2lmA7H9Nm65i33LEOGDGHSpEk8/PDDQLLV/slPfhJJ9O3blwcffJCVK1eybNkyvvzlL7d4ca/vfe979O/fn9WrVzNv3rxGx6wvWLCA6upqVq9ezW9+8xtWr17Ntddey3HHHceyZctYtmxZo3mtWLGCe+65hyeffJLf//73/PCHP+Tpp58GkouYXX311axZs4bBgwfzwAMPtPgZP/3pT/PNb36T1atXc/rpp3PLLbcAySWMn376aVavXn3gssa33347ixYtYtWqVTz++OMtrjSseA7HdPeQQRFvube0hd2V6rtmZs6cyeLFiw9sLUcEN954I4899hg9evTgpZde4uWXX+aYY47JOZ/HHnuMa6+9FoDx48czfvz4A6/dd999VFZWUldXx7Zt21i7dm2j15t64okn+OhHP3rgypSXXHIJjz/+ODNmzGDUqFFMmDABaPmywpBcX37Hjh1MmTIFgM985jN8/OMfP1BjeXk5F198MRdffDEA55xzDtdffz3l5eVccsklDB8+PJ+v8LC1YEHuSyB09+GYxXS2brH8giiWOrqTt9ybuPjii3nkkUdYuXIle/bsOXCTjKqqKmpra1mxYgWrVq3iPe95T87L/GaTdNC0F154gdtvv51HHnmE1atXc8EFF7Q6n5Z+IdRfLhhavqxwa371q19x9dVXs2LFCs466yzq6uqYO3cud911F3v27OHss8/mj3/8Y7vmfbgolsMxi6V7qFh+QRRLHfW1dNfRVHmFu6RpktZL2ijpoFsJSTpC0s8yrz8pqbSzC+0uAwYMYOrUqfzt3/5tox2pO3fu5Oijj6Z3794sW7aMzZs3tzifD3zgAwdugv3ss8+yevVqILlc8JFHHsmgQYN4+eWX+fWvf33gPQMHDuT111/POa+f//zn7N69mzfffJMHH3yQ97///W3+bIMGDeKoo47i8ccfB+DHP/4xU6ZMYf/+/WzZsoVzzz2X2267jR07dvDGG2/w3HPPcfrpp3PDDTdQVlbmcM9DMRyOWSzdQ8Wyg7lY6ujulUyr3TKSegKLgA8BNcBySUsiIvsC458FXouIEyXNBr4JfLIrCu4Oc+bM4ZJLLml05Ex5eTkXXXQRZWVlTJgwgZNPPrnFeXz+85/niiuuYPz48UyYMIFJkyYByV2VzjzzTE7rQ9KTAAAE0UlEQVQ99dSDLhdcUVHB9OnTOfbYYxv1u0+cOJHLL7/8wDyuvPJKzjzzzBa7YJrzox/96MAO1dGjR3PPPfewb98+LrvsMnbu3ElE8KUvfYnBgwfzta99jWXLltGzZ0/GjRt34K5SVtyKpXuoWH5BFEsd3d5dFhEtPoD3AUuzxr8KfLVJm6XA+zLDvYBXyVxOuLnHWWedFU2tXbv2oGlW3Pw3K0733hsxcmSElDzfe2/31zByZESyjdr4MXLk4VmHlLsOqW3zAaqjldyOiLy6ZYYBW7LGazLTcraJiDpgJzCk6YwkVUiqllRdW1ub7/rHzNqoGLqHiuV6P8VSR3d3l+UT7gfvFYSme/jyaUNEVEZEWUSUlZSU5FOfmR2iimUHc7HU0d0rmXwOhawBjs8aHw5sbaZNjaRewCDgL+0pKDK3g7PiFwW6i5cdOgp1K8xirKN++d11SGY+W+7LgTGSRknqA8wGljRpswT4TGZ4FvBotON/ft++fdm+fbtD4xAQEWzfvp2+ffsWuhSzQ0Z3dpe1uuUeEXWSriHZadoTuDsi1kiaT9KxvwT4J+DHkjaSbLHPbk8xw4cPp6amBvfHHxr69u3rE5vMilRR3SDbzMxalu8Nsn2GqplZCjnczcxSyOFuZpZCBetzl1QLtHyBluI3lORsXEv4+2jg76Ixfx+NdeT7GBkRrZ4oVLBwTwNJ1fns2Dhc+Pto4O+iMX8fjXXH9+FuGTOzFHK4m5mlkMO9YyoLXUCR8ffRwN9FY/4+Guvy78N97mZmKeQtdzOzFHK4m5mlkMO9HSQdL2mZpHWS1ki6rtA1FZqknpKelvTLQtdSaJIGS7pf0h8z/0beV+iaCknSlzL/T56V9FNJh82lRCXdLekVSc9mTXu3pH+XtCHzfFRXLNvh3j51wJcj4hTgbOBqSeMKXFOhXQesK3QRReI7wMMRcTJwBofx9yJpGHAtUBYRp5FcWbZdV409RP0zMK3JtLnAIxExBngkM97pHO7tEBHbImJlZvh1kv+8TW89eNiQNBy4ALir0LUUmqR3AR8guQw2EfFOROwobFUF1wvol7mRT38OvtlPakXEYxx846KZwI8ywz8CLu6KZTvcO0hSKXAm8GRhKymoO4H/CewvdCFFYDRQC9yT6aa6S9KRhS6qUCLiJeB24EVgG7AzIv6tsFUV3HsiYhskG4rA0V2xEId7B0gaADwAfDEidhW6nkKQdCHwSkSsKHQtRaIXMBH4XkScCbxJF/3sPhRk+pNnAqOA44AjJV1W2KoODw73dpLUmyTYqyLiXwpdTwGdA8yQtAlYDJwn6d7CllRQNUBNRNT/krufJOwPV+cDL0REbUTsBf4F+OsC11RoL0s6FiDz/EpXLMTh3g5K7uD9T8C6iPh2oesppIj4akQMj4hSkh1lj0bEYbtlFhF/BrZIGpuZ9EFgbQFLKrQXgbMl9c/8v/kgh/EO5ozse05/BvhFVyyk1XuoWk7nAJ8CnpG0KjPtxoh4qIA1WfH4AlCVuaH888AVBa6nYCLiSUn3AytJjjJ7msPoUgSSfgpMBYZKqgH+AbgVuE/SZ0lWfh/vkmX78gNmZunjbhkzsxRyuJuZpZDD3cwshRzuZmYp5HA3M0shh7uZWQo53M3MUui/ARcrEELFzVPBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Validation accuracy stalls in the low 50s. So in our case, pre-trained word embeddings does outperform jointly learned embeddings. If you \n",
    "increase the number of training samples, this will quickly stop being the case -- try it as an exercise.\n",
    "\n",
    "Finally, let's evaluate the model on the test data. First, we will need to tokenize the test data:\n",
    "\n",
    "验证精度停留在 50% 多一点。因此，在本例中，预训练词嵌入的性能要优于与任务一起学习的嵌入。如果增加样本数量，情况将很快发生变化，你可以把它作为一个练习。\n",
    "\n",
    "最后，我们在测试数据上评估模型。首先，你需要对测试数据进行分词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dir = os.path.join(imdb_dir, 'test')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(test_dir, label_type)\n",
    "    for fname in sorted(os.listdir(dir_name)):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname),encoding='utf8')\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "x_test = pad_sequences(sequences, maxlen=maxlen)\n",
    "y_test = np.asarray(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's load and evaluate the first model:\n",
    "\n",
    "接下来，加载并评估第一个模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 1s 59us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.744873430480957, 0.57604]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights('pre_trained_glove_model.h5')\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an appalling test accuracy of 57%. Working with just a handful of training samples is hard!\n",
    "\n",
    "测试精度达到了令人震惊的 57% ！只用了很少的训练样本，得到这样的结果很不容易。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}