使用带Attention的RNN进行机器翻译.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 目录\n",
    "1. [背景](#1)<br>\n",
    "2. [载入数据文件](#2)<br>\n",
    "3. [Seq2Seq模型](#3)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;3.1 [编码器](#3.1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;3.2 [解码器](#3.2)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3.2.1 [简单的解码器](#3.1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3.2.2 [Attention解码器](#3.2)<br>\n",
    "4. [训练](#4)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;4.1 [准备训练数据](#4.1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;4.2 [训练模型](#4.2)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;4.2 [绘制结果](#4.3)<br>\n",
    "5. [评估](#5)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;5.1 [执行训练和评估过程](#5.1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;5.2 [Attention过程可视化](#5.2)<br>\n",
    "6. [更多的尝试](#6)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"1\"></div>\n",
    "\n",
    "# 1 背景"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在本案例中，我们训练一个带Attention机制的循环神经网络用于将中文翻译成英文。\n",
    "\n",
    "一些简单的翻译结果示例如下：\n",
    "```\n",
    "[> 输入 ，= 目标 ，< 输出]\n",
    "\n",
    "> 他不是傻子。\n",
    "= he is not stupid .\n",
    "< he is no fool .\n",
    "```\n",
    "该循环神经网络的编码器部分将输入序列压缩成矢量，解码器部分将该矢量展开为新的序列，即完成了翻译过程。同时，为提升模型性能，我们引入Attention机制使得解码器可以重点关注输入序列的某一部分。\n",
    "\n",
    "中文翻译比较困难的一点在于，与英文、法文等使用空格作为词语分割的语言不同，中文的分词是一个比较复杂的问题，在这里我们使用`jieba`进行分词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们首先导入之后需要的包。\n",
    "\n",
    "+ `unicode_literals`代表将模块中显式出现的所有字符串转为unicode类型，防止在循环时以字节为单位而不是以字符为单位。\n",
    "+ `print_function`和`division`代表使用python 3.x版本的`print()`函数和除法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "import unicodedata\n",
    "import string\n",
    "import re\n",
    "import random\n",
    "import jieba\n",
    "\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "import torch.nn as nn\n",
    "\n",
    "# use_cuda = torch.cuda.is_available()\n",
    "use_cuda = [] # cpu运行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"2\"></div>\n",
    "\n",
    "# 2 载入数据文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该数据（[原始下载路径](http://www.manythings.org/anki/)）包含了19577条中文-英文的翻译句子对。 每一行代表了一个翻译对，并使用制表符tab作为中文（繁体字）和英文的分割。\n",
    "\n",
    "```\n",
    "Do it now.    現在就做。\n",
    "```\n",
    "我们使用独热矢量(one-hot vector)表示每一个单词，即除了单个单词（在单词的索引处）以外所有值都为零的巨大矢量。每一个语言中有非常多的单词，为简单起见，我们删除掉部分数据，只使用有限的单词构成的句子。\n",
    "\n",
    "我们需要每个单词的唯一索引作为网络的输入和目标。为了跟踪所有词，并且标记序列的起始和结束。我们将构建一个`Lang`的类，它具有词→索引（`word2index`）和索引→词（`index2word`）的字典，以及每个单词的计数`word2count`用于以后替换罕见的词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用来确定句子序列开始与终止\n",
    "SOS_token = 0 # 序列起始标记\n",
    "EOS_token = 1 # 序列终止标记\n",
    "\n",
    "class Lang:\n",
    "    \"\"\"\n",
    "    包含每个语言单词数的统计信息，\n",
    "    name:语言的名字简写, english -> eng, chinese -> chi\n",
    "    addSentence:统计句子数量\n",
    "    addWord:统计词语数量\n",
    "    \"\"\"\n",
    "    def __init__(self, name):\n",
    "        self.name = name\n",
    "        self.word2index = {}\n",
    "        self.word2count = {}\n",
    "        self.index2word = {0: \"SOS\", 1: \"EOS\"}\n",
    "        self.n_words = 2  # 包含SOS 和 EOS, 所以一开始词语数量为2\n",
    "\n",
    "    def addSentence(self, sentence):\n",
    "        # 满足条件为中文，需要使用jieba分词，否则使用空格分词\n",
    "        if len(sentence.split(' ')) == 1: \n",
    "            for word in list(jieba.cut(sentence)):\n",
    "                self.addWord(word)\n",
    "        else:\n",
    "            for word in sentence.split(' '):\n",
    "                self.addWord(word)\n",
    "\n",
    "    def addWord(self, word):\n",
    "        if word not in self.word2index:\n",
    "            # 词 -> 索引\n",
    "            self.word2index[word] = self.n_words\n",
    "            self.word2count[word] = 1\n",
    "            # 索引 -> 词\n",
    "            self.index2word[self.n_words] = word \n",
    "            self.n_words += 1\n",
    "        else:\n",
    "            self.word2count[word] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这些文件都是Unicode格式，为了简化我们将Unicode字符转换为ASCII格式，并使所有内容都变成小写字母，修剪大部分标点符号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unicode转换成ASCII格式\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "    )\n",
    "\n",
    "# 小写, 修剪并且去除非字母字符\n",
    "def normalizeString(s):\n",
    "    s = unicodeToAscii(s.lower().strip())\n",
    "    s = re.sub(r\"([.!?])\", r\" \\1\", s) # 为了能够单独将符号分割出来\n",
    "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们首先将文件分割成多行，然后将行分成中文-英文对。这些样本都是英文→中文的排列方式，而我们想从中文→英文的演绎，所以添加了`reverse`来反转翻译对。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readLangs(lang1, lang2, reverse=False):\n",
    "    \"\"\"\n",
    "    读取翻译原始文件，并返回输入实例和输出实例及翻译对\n",
    "    输入参数：源语言 lang1，目标语言 lang2，是否反转翻译对 reverse\n",
    "    输出参数：输入序列 input_lang，输出序列 output_lang，翻译对 pairs\n",
    "    \"\"\"\n",
    "    print(\"Reading lines...\")\n",
    "\n",
    "    # 读取文件并分割为多行\n",
    "    lines = open('./input/%s-%s.txt' % (lang1, lang2), encoding='utf-8').\\\n",
    "    read().strip().split('\\n')\n",
    "\n",
    "    # 分割每一行翻译对并规整字符串\n",
    "    pairs = [[normalizeString(l.split('\\t')[0]), l.split('\\t')[1]]\\\n",
    "         for l in lines]\n",
    "\n",
    "    # 反转翻译对，并创建输入输出序列\n",
    "    if reverse:\n",
    "        pairs = [list(reversed(p)) for p in pairs]\n",
    "        input_lang = Lang(lang2)\n",
    "        output_lang = Lang(lang1)\n",
    "    else:\n",
    "        input_lang = Lang(lang1)\n",
    "        output_lang = Lang(lang2)\n",
    "    return input_lang, output_lang, pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于有很多例句，我们希望快速训练，所以我们将数据集裁剪为相对简短的句子，这里的英文句子最大长度`MAX_LENGTH`设置10个字符（包括结尾标点符号）。\n",
    "\n",
    "然后我们过滤翻译对，只保留翻译成形如\"I am\"或\"He is\"等的句子（并且考虑缩写情况\"I'm\"和\"He's\"等）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 10\n",
    "\n",
    "eng_prefixes = (\n",
    "    \"i am \", \"i m \",\n",
    "    \"he is\", \"he s \",\n",
    "    \"she is\", \"she s\",\n",
    "    \"you are\", \"you re \",\n",
    "    \"we are\", \"we re \",\n",
    "    \"they are\", \"they re \"\n",
    ")\n",
    "\n",
    "def filterPair(p):\n",
    "    \"\"\"\n",
    "    单个翻译对过滤，过滤条件：分词后，中文词数小于MAX_LENGTH，\n",
    "    英文单词数小于MAX_LENGTH，\n",
    "    且以eng_prefixes中的元素开始\n",
    "    \"\"\"\n",
    "    return len(list(jieba.cut(p[0]))) < MAX_LENGTH and \\\n",
    "        len(p[1].split(' ')) < MAX_LENGTH and \\\n",
    "        p[1].startswith(eng_prefixes)\n",
    "\n",
    "def filterPairs(pairs):\n",
    "    \"\"\"\n",
    "    对所有的翻译对进行过滤，并返回一个嵌套列表\n",
    "    \"\"\"\n",
    "    return [pair for pair in pairs if filterPair(pair)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "整个准备数据的完整过程是：\n",
    "\n",
    "+ 读取文本文件并拆分成行，将行拆分成翻译对\n",
    "+ 使文本规整，按照长度和内容进行过滤\n",
    "+ 从成对的句子中做出单词列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading lines...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /tmp/jieba.cache\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 19577 sentence pairs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading model cost 1.232 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed to 1379 sentence pairs\n",
      "Counting words...\n",
      "Counted words:\n",
      "chi 1731\n",
      "eng 1209\n",
      "我們|全體|為|日本|祈禱|。 we are all praying for japan .\n"
     ]
    }
   ],
   "source": [
    "def prepareData(lang1, lang2, reverse=False):\n",
    "    # 读数据\n",
    "    input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse) \n",
    "    print(\"Read %s sentence pairs\" % len(pairs))\n",
    "    # 过滤数据\n",
    "    pairs = filterPairs(pairs) \n",
    "    print(\"Trimmed to %s sentence pairs\" % len(pairs))\n",
    "    print(\"Counting words...\")\n",
    "    for pair in pairs:\n",
    "        input_lang.addSentence(pair[0])\n",
    "        output_lang.addSentence(pair[1])\n",
    "    print(\"Counted words:\")\n",
    "    print(input_lang.name, input_lang.n_words)\n",
    "    print(output_lang.name, output_lang.n_words)\n",
    "    return input_lang, output_lang, pairs\n",
    "\n",
    "input_lang, output_lang, pairs = prepareData('eng', 'chi', True)\n",
    "# 随机选择样本进行展示\n",
    "sample = random.choice(pairs) \n",
    "print('|'.join(jieba.cut(sample[0])), sample[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"3\"></div>\n",
    "\n",
    "# 3 Seq2Seq模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "递归神经网络（RNN）是一个按照序列运行的网络，它将自己的输出当做后续步骤的输入。\n",
    "\n",
    "Seq2Seq网络或编码器-解码器网络是由两个称为编码器和解码器的RNN组成的模型。 编码器读取输入序列并输出单个矢量（上下文向量context），解码器读取该矢量以产生输出序列。\n",
    "\n",
    "与使用单个RNN的序列预测不同，每个输入对应于一个输出，Seq2Seq模型将我们从序列长度和顺序中解放出来，这使得它成为两种语言之间进行翻译的理想工具。\n",
    "\n",
    "考虑“我|對|這個|故事|感興趣|”→“i am interested in this story”这个句子。输入句子中的大部分单词在输出句子中具有直接翻译，但是顺序稍有不同，例如，“故事感兴趣”和“interested in this story”。并且输入句子相较于输出还少一个词。直接从输入词序列中产生正确的翻译将是困难的。\n",
    "\n",
    "使用Seq2Seq模型，编码器会创建一个单一的上下文矢量，在理想的情况下，将输入序列的“含义”编码为这个上下文矢量——$N$维上下文空间中的单个点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"3.1\"></div>\n",
    "\n",
    "## 3.1 编码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seq2Seq网络的编码器是一个RNN，它为输入句子中的每个单词输出一些值。对于每个输入，编码器输出一个向量和一个隐藏状态，并将隐藏状态作为下一轮迭代的输入。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![png](http://cookdata.cn/media/note_images/encoder-network_1560754383753_5d14.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 编码器\n",
    "class EncoderRNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, n_layers=1):\n",
    "        super(EncoderRNN, self).__init__()\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        # word embedding 词向量维度即为 input_size × hidden_size\n",
    "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
    "        # 编码器就是一个RNN-GRU\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size)\n",
    "\n",
    "    # 向前传播\n",
    "    def forward(self, input, hidden):\n",
    "        # 展开为向量\n",
    "        output = self.embedding(input).view(1, 1, -1) \n",
    "        for i in range(self.n_layers):\n",
    "            output, hidden = self.gru(output, hidden)\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        result = Variable(torch.zeros(1, 1, self.hidden_size))\n",
    "        if use_cuda:\n",
    "            return result.cuda()\n",
    "        else:\n",
    "            return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"3.2\"></div>\n",
    "\n",
    "## 3.2 解码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器是另一个RNN，它接收编码器的输出向量作为输入并输出得到翻译序列。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"3.2.1\"></div>\n",
    "\n",
    "## 3.2.1 简单的解码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在最简单的Seq2Seq解码器中，我们只使用编码器的最后一个输出。这个最后的输出即是上下文向量，因为它从整个序列编码得到。该上下文向量被用作解码器的初始隐藏状态。\n",
    "\n",
    "在解码的每一步，解码器都被赋予一个输入令牌（input token）和隐藏状态。初始输入令牌是标记序列开始的`<SOS>`，第一个隐藏状态是上下文向量（编码器的最后隐藏状态）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![png](http://cookdata.cn/media/note_images/decoder-network_1560754751098_5d14.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 解码器\n",
    "class DecoderRNN(nn.Module):\n",
    "    def __init__(self, hidden_size, output_size, n_layers=1):\n",
    "        super(DecoderRNN, self).__init__()\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        # words embedding\n",
    "        self.embedding = nn.Embedding(output_size, hidden_size)\n",
    "        # 解码器也是一个RNN-GRU\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size)\n",
    "        self.out = nn.Linear(hidden_size, output_size)\n",
    "        # 经过log softmax激活函数\n",
    "        self.softmax = nn.LogSoftmax()\n",
    "\n",
    "    # 向前传播\n",
    "    def forward(self, input, hidden):\n",
    "        # 展开为向量\n",
    "        output = self.embedding(input).view(1, 1, -1)\n",
    "        for i in range(self.n_layers):\n",
    "            output = F.relu(output)\n",
    "            output, hidden = self.gru(output, hidden)\n",
    "        output = self.softmax(self.out(output[0]))\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        result = Variable(torch.zeros(1, 1, self.hidden_size))\n",
    "        if use_cuda:\n",
    "            return result.cuda()\n",
    "        else:\n",
    "            return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以尝试训练和观察这个模型的结果，但为了节省时间，我们将直接引入带注意机制的解码器网络。\n",
    "\n",
    "使用带注意力机制的解码器Seq2Seq模型，其编码器是没有变化的，依旧是一个RNN-GRU，不用更改。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"3.2.2\"></div>\n",
    "\n",
    "## 3.2.2 Attention解码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果仅在编码器和解码器之间传递上下文向量，则该单个向量承担编码整个句子的负担。\n",
    "\n",
    "注意机制允许解码器网络在解码器自身输出的每一步“聚焦”编码器输出的不同部分。首先我们计算一组注意力权重。这些将被乘以编码器输出矢量以创建加权组合。结果（在代码中称为`attn_applied`）应该包含关于输入序列的特定部分的信息，从而帮助解码器选择正确的输出字。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![png](http://cookdata.cn/media/note_images/attention_1560754808346_5d14.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用解码器的输入和隐藏状态作为输入，利用另一前馈层`attn`来计算注意力权重。由于在训练数据中有各种大小的句子，为了实际创建和训练这个层，我们必须选择一个最大的句子长度（输入长度，用于编码器的输出）。最大长度的句子将使用所有的注意权重，而较短的句子将只使用前几个。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![png](http://cookdata.cn/media/note_images/attention-decoder-network_1560754824456_5d14.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 在Seq2Seq模型中，我们要使用双向循环神经网络来计算输入的隐状态。这是因为如果只是用单向循环神经网络，在计算注意力权重参数$\\alpha_{ij}$时，会造成前后文信息的丢失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 带注意力机制的解码器\n",
    "class AttnDecoderRNN(nn.Module):\n",
    "    def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1, max_length=MAX_LENGTH):\n",
    "        super(AttnDecoderRNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout_p = dropout_p\n",
    "        self.max_length = max_length\n",
    "\n",
    "        # word embedding 词向量\n",
    "        self.embedding = nn.Embedding(self.output_size, self.hidden_size)\n",
    "        # 在带注意力机制的解码器中，我们要使用双向RNN模型\n",
    "        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)\n",
    "        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)\n",
    "        self.dropout = nn.Dropout(self.dropout_p)\n",
    "        \n",
    "        self.gru = nn.GRU(self.hidden_size, self.hidden_size)\n",
    "        self.out = nn.Linear(self.hidden_size, self.output_size)\n",
    "\n",
    "    def forward(self, input, hidden, encoder_output, encoder_outputs):\n",
    "        embedded = self.embedding(input).view(1, 1, -1)  # 展开为向量\n",
    "        embedded = self.dropout(embedded)\n",
    "\n",
    "        # 使用另外一个前馈神经网络 self.attn 计算注意力权重参数\n",
    "        attn_weights = F.softmax(\n",
    "            self.attn(torch.cat((embedded[0], hidden[0]), 1)))\n",
    "        # 注意力参数与编码器得到的隐状态做加权和\n",
    "        attn_applied = torch.bmm(attn_weights.unsqueeze(0),\n",
    "                                 encoder_outputs.unsqueeze(0))\n",
    "\n",
    "        output = torch.cat((embedded[0], attn_applied[0]), 1)\n",
    "        output = self.attn_combine(output).unsqueeze(0)\n",
    "\n",
    "        # 向前传播\n",
    "        for i in range(self.n_layers):\n",
    "            output = F.relu(output)\n",
    "            output, hidden = self.gru(output, hidden)\n",
    "\n",
    "        # log-softmax激活函数 --> 与一般的解码器是相同的\n",
    "        output = F.log_softmax(self.out(output[0]))\n",
    "        return output, hidden, attn_weights\n",
    "\n",
    "    def initHidden(self):\n",
    "        result = Variable(torch.zeros(1, 1, self.hidden_size))\n",
    "        if use_cuda:\n",
    "            return result.cuda()\n",
    "        else:\n",
    "            return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"4\"></div>\n",
    "\n",
    "# 4 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"4.1\"></div>\n",
    "\n",
    "## 4.1 准备训练数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了便于训练，我们将需要一一对应的输入tensor（输入句子中的词的索引）和目标tensor（目标句中的词的索引）。在创建这些向量时，我们会将`<EOS>`标记附加到两个语言的序列中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def indexesFromSentence(lang, sentence):\n",
    "    if len(sentence.split(' ')) == 1:\n",
    "        words = list(jieba.cut(sentence)) # jieba中文分词\n",
    "    else:\n",
    "        words = sentence.split(' ')\n",
    "    return [lang.word2index[word] for word in words] # 空格英文分词\n",
    "\n",
    "def variableFromSentence(lang, sentence):\n",
    "    indexes = indexesFromSentence(lang, sentence)\n",
    "    indexes.append(EOS_token)\n",
    "    result = Variable(torch.LongTensor(indexes).view(-1, 1))\n",
    "    if use_cuda:\n",
    "        return result.cuda()\n",
    "    else:\n",
    "        return result\n",
    "\n",
    "def variablesFromPair(pair):\n",
    "    input_variable = variableFromSentence(input_lang, pair[0])\n",
    "    target_variable = variableFromSentence(output_lang, pair[1])\n",
    "    return (input_variable, target_variable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"4.2\"></div>\n",
    "\n",
    "## 4.2 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在训练时，我们将待翻译的序列输入编码器，并跟踪每个输出和最新的隐藏状态。然后解码器被赋予`<SOS>`令牌作为其第一个输入，而编码器的最后一个隐藏状态作为解码器的第一个隐藏状态。\n",
    "\n",
    "在解码器训练时，我们将使用“导师驱动(teaching forcing)”机制，即将实际目标输出用作下一个输入，而不是将解码器的猜测用作下一个输入。使用导师驱动目的是希望损失能更快地收敛，但是使用这个机制训练出来的网络在预测阶段可能表现出不稳定性。\n",
    "\n",
    "使用导师驱动得到的网络可以学习到连贯的语法，但离正确的翻译还是有一段距离。直观地说，它已经学会了代表输出语法，并且可以在导师告诉它的前几个单词时“拾取”意义，但是从翻译的角度来看，它还没有很好的习得如何去创造句子。\n",
    "\n",
    "由于PyTorch的`autograd`给了我们很大的自由选择空间，我们可以很方便的使用if语句选择使用导师驱动的网络或者非导师驱动的网络。可以使用`teacher_forcing_ratio`来指定使用导师驱动的程度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "teacher_forcing_ratio = 0.5\n",
    "\n",
    "\n",
    "def train(input_variable, target_variable, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):\n",
    "    encoder_hidden = encoder.initHidden()\n",
    "\n",
    "    # 权值初始化\n",
    "    encoder_optimizer.zero_grad()\n",
    "    decoder_optimizer.zero_grad()\n",
    "\n",
    "    input_length = input_variable.size()[0]\n",
    "    target_length = target_variable.size()[0]\n",
    "\n",
    "    encoder_outputs = Variable(torch.zeros(max_length, encoder.hidden_size))\n",
    "    encoder_outputs = encoder_outputs.cuda() if use_cuda else encoder_outputs\n",
    "\n",
    "    loss = 0\n",
    "\n",
    "    for ei in range(input_length):\n",
    "        encoder_output, encoder_hidden = encoder(\n",
    "            input_variable[ei], encoder_hidden)\n",
    "        encoder_outputs[ei] = encoder_output[0][0]\n",
    "\n",
    "    # SOS_token作为解码器的第一个输入\n",
    "    decoder_input = Variable(torch.LongTensor([[SOS_token]]))\n",
    "    decoder_input = decoder_input.cuda() if use_cuda else decoder_input\n",
    "\n",
    "    decoder_hidden = encoder_hidden\n",
    "\n",
    "    use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False\n",
    "\n",
    "    if use_teacher_forcing:\n",
    "        # 导师驱动：使用目标输出作为下一个隐层的输入\n",
    "        for di in range(target_length):\n",
    "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
    "                decoder_input, decoder_hidden, encoder_output, encoder_outputs)\n",
    "            # 损失值\n",
    "            loss += criterion(decoder_output, target_variable[di])\n",
    "            decoder_input = target_variable[di]\n",
    "\n",
    "    else:\n",
    "        # 非导师驱动: 使用预测值作为下一个隐层的输入\n",
    "        for di in range(target_length):\n",
    "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
    "                decoder_input, decoder_hidden, encoder_output, encoder_outputs)\n",
    "            topv, topi = decoder_output.data.topk(1)\n",
    "            ni = topi[0][0]\n",
    "\n",
    "            decoder_input = Variable(torch.LongTensor([[ni]]))\n",
    "            decoder_input = decoder_input.cuda() if use_cuda else decoder_input\n",
    "\n",
    "            # 损失值\n",
    "            loss += criterion(decoder_output, target_variable[di])\n",
    "            if ni == EOS_token:\n",
    "                break\n",
    "\n",
    "    # 向后传播\n",
    "    loss.backward()\n",
    "\n",
    "    # 梯度更新\n",
    "    encoder_optimizer.step()\n",
    "    decoder_optimizer.step()\n",
    "\n",
    "    return loss.item() / target_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了能够实时观察训练的进度，我们定义两个辅助函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def asMinutes(s):\n",
    "    # 记录分钟\n",
    "    m = math.floor(s / 60)\n",
    "    # 记录秒\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "def timeSince(since, percent):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    es = s / (percent)\n",
    "    rs = es - s\n",
    "    return '%s - %s' % (asMinutes(s), asMinutes(rs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总结整个训练过程如下：\n",
    "\n",
    "+ 启动一个计时器\n",
    "+ 初始化优化器optimizers和损失函数criterion\n",
    "+ 创建用于训练的翻译对\n",
    "+ 不断计算损失以绘制图形\n",
    "\n",
    "然后我们将模型代入训练函数`train`，设定学习率为0.01，使用SGD进行优化，并且实时输出训练进度（训练数据的完成百分比，时间耗费，剩余时间估计）和平均损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def trainIters(encoder, decoder, n_iters, print_every=100, plot_every=100, learning_rate=0.01):\n",
    "    start = time.time()\n",
    "    plot_losses = []\n",
    "    print_loss_total = 0\n",
    "    plot_loss_total = 0 \n",
    "\n",
    "    # 优化器\n",
    "    encoder_optimizer = optim.SGD(encoder.parameters(), lr=learning_rate)\n",
    "    decoder_optimizer = optim.SGD(decoder.parameters(), lr=learning_rate)\n",
    "    training_pairs = [variablesFromPair(random.choice(pairs))\n",
    "                      for i in range(n_iters)]\n",
    "    # NLL损失函数\n",
    "    criterion = nn.NLLLoss() \n",
    "\n",
    "    for iter in range(1, n_iters + 1):\n",
    "        training_pair = training_pairs[iter - 1]\n",
    "        input_variable = training_pair[0]\n",
    "        target_variable = training_pair[1]\n",
    "\n",
    "        # 损失值\n",
    "        loss = train(input_variable, target_variable, encoder,\n",
    "                     decoder, encoder_optimizer, decoder_optimizer, criterion)\n",
    "        print_loss_total += loss\n",
    "        plot_loss_total += loss\n",
    "\n",
    "        if iter % print_every == 0:\n",
    "            print_loss_avg = print_loss_total / print_every\n",
    "            print_loss_total = 0\n",
    "            print('%s (%d %d%%) %.4f' % (timeSince(start, iter / n_iters),\n",
    "                                         iter, iter / n_iters * 100, print_loss_avg))\n",
    "\n",
    "        if iter % plot_every == 0:\n",
    "            plot_loss_avg = plot_loss_total / plot_every\n",
    "            plot_losses.append(plot_loss_avg)\n",
    "            plot_loss_total = 0\n",
    "\n",
    "    showPlot(plot_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"4.3\"></div>\n",
    "\n",
    "## 4.3 绘制结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了直观的看出训练过程中损失值的变化情况。我们使用`matplotlib`完成绘图，同时使用训练时保存的损失值`plot_losses`数组。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "def showPlot(points):\n",
    "    plt.figure()\n",
    "    fig, ax = plt.subplots()\n",
    "    # loc控制刻度间隔\n",
    "    loc = ticker.MultipleLocator(base=0.2)\n",
    "    ax.yaxis.set_major_locator(loc)\n",
    "    plt.plot(points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"5\"></div>\n",
    "\n",
    "# 5 评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估与训练大体相同，但没有目标，因此我们只是将解码器的预测反馈给每一步。 每次预测一个单词，我们将它添加到输出字符串，如果它预测`<EOS>`令牌，我们就停止输出新的字符。我们也存储了解码器的注意力输出以供稍后对注意力进行分析。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate(encoder, decoder, sentence, max_length=MAX_LENGTH):\n",
    "    input_variable = variableFromSentence(input_lang, sentence)\n",
    "    input_length = input_variable.size()[0]\n",
    "    encoder_hidden = encoder.initHidden()\n",
    "\n",
    "    encoder_outputs = Variable(torch.zeros(max_length, encoder.hidden_size))\n",
    "    encoder_outputs = encoder_outputs.cuda() if use_cuda else encoder_outputs\n",
    "\n",
    "    # 这里与训练模型时是类似的\n",
    "    for ei in range(input_length):\n",
    "        encoder_output, encoder_hidden = encoder(input_variable[ei],\n",
    "                                                 encoder_hidden)\n",
    "        encoder_outputs[ei] = encoder_outputs[ei] + encoder_output[0][0]\n",
    "\n",
    "    # SOS_token作为解码器的第一个输入\n",
    "    decoder_input = Variable(torch.LongTensor([[SOS_token]]))  \n",
    "    decoder_input = decoder_input.cuda() if use_cuda else decoder_input\n",
    "\n",
    "    decoder_hidden = encoder_hidden\n",
    "\n",
    "    decoded_words = []\n",
    "    decoder_attentions = torch.zeros(max_length, max_length)\n",
    "\n",
    "    for di in range(max_length):\n",
    "        decoder_output, decoder_hidden, decoder_attention = decoder(\n",
    "            decoder_input, decoder_hidden, encoder_output, encoder_outputs)\n",
    "        decoder_attentions[di] = decoder_attention.data\n",
    "        topv, topi = decoder_output.data.topk(1)\n",
    "        ni = topi[0][0]\n",
    "        # 如果得到EOS_token，停止输出\n",
    "        if ni == EOS_token:\n",
    "            decoded_words.append('<EOS>')\n",
    "            break\n",
    "        else:\n",
    "            decoded_words.append(output_lang.index2word[int(ni)])\n",
    "\n",
    "        decoder_input = Variable(torch.LongTensor([[ni]]))\n",
    "        decoder_input = decoder_input.cuda() if use_cuda else decoder_input\n",
    "\n",
    "    return decoded_words, decoder_attentions[:di + 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以从训练集中随机挑选句子进行评估，并打印输入、目标和输出以对模型做一些分析："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    > 中文语句\n",
    "    = 目标翻译\n",
    "    < 模型翻译\n",
    "\"\"\"\n",
    "def evaluateRandomly(encoder, decoder, n=10):\n",
    "    for i in range(n):\n",
    "        pair = random.choice(pairs)\n",
    "        print('>', pair[0])\n",
    "        print('=', pair[1])\n",
    "        output_words, attentions = evaluate(encoder, decoder, pair[0])\n",
    "        output_sentence = ' '.join(output_words)\n",
    "        print('<', output_sentence)\n",
    "        print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"5.1\"></div>\n",
    "\n",
    "# 5.1 执行训练和评估过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前面定义了非常多的函数，并未开始任何的训练过程，虽然看起来像是额外的工作，但这更容易运行多个实验。\n",
    "\n",
    "现在，我们可以正式初始化一个网络并开始训练。\n",
    "\n",
    "需要注意的是，输入的句子被严重过滤（从19577缩减为1379）。对于这个小数据集，我们可以使用256个隐藏节点和一个`GRU`层（门控单元）的相对较小的网络。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:22: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:33: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1m 44s (- 22m 33s) (5000 7%) 2.9993\n",
      "3m 20s (- 20m 2s) (10000 14%) 2.0066\n",
      "5m 2s (- 18m 27s) (15000 21%) 1.1277\n",
      "6m 41s (- 16m 42s) (20000 28%) 0.5095\n",
      "8m 15s (- 14m 52s) (25000 35%) 0.1953\n",
      "9m 58s (- 13m 17s) (30000 42%) 0.0975\n",
      "11m 38s (- 11m 38s) (35000 50%) 0.0691\n",
      "13m 10s (- 9m 52s) (40000 57%) 0.0523\n",
      "14m 47s (- 8m 13s) (45000 64%) 0.0446\n",
      "16m 19s (- 6m 31s) (50000 71%) 0.0464\n",
      "17m 56s (- 4m 53s) (55000 78%) 0.0456\n",
      "19m 29s (- 3m 14s) (60000 85%) 0.0403\n",
      "21m 7s (- 1m 37s) (65000 92%) 0.0379\n",
      "22m 48s (- 0m 0s) (70000 100%) 0.0384\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD6CAYAAAC1W2xyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XecnFXZ//HPtb1vks2mk0ZIIYEAKZQQSgiYQAAR5UGwAI8/1McGqCg+giioKKJUQVQUUEQfQYNACIQWQpaShBTS26a3zSbb+57fHzPbZmezs8m90/b7fr14cc89Z2euvZNcc+bc5zrHnHOIiEj8SIh0ACIi4i0ldhGROKPELiISZ5TYRUTijBK7iEicUWIXEYkzSuwiInFGiV1EJM4osYuIxJmkSLxp37593fDhwyPx1iIiMWvp0qVFzrn8ztpFJLEPHz6cJUuWROKtRURilpltC6WdhmJEROKMEruISJwJObGb2c1mtqCD5/qa2TtmtsrM7vEuPBER6aqQEruZDQOuO0KTm4CXgInAbDMbfeyhiYjI0Qi1x/4AcNsRnp8BvOacawTeBs4/1sBEROTodJrYzewaYAWw5gjN8oAS/3Ep0CfI69xoZkvMbMmBAweOJlYREQlBKD32OcAFwLPAJDP7epA2RUCu/zjX/7gN59zjzrnJzrnJ+fmdTsMUEZGj1Glid85d45w7G7gaWOqcezhIs9eBi8wsATgXeNPbMH3W7y3jvlfXc7C8pjteXkQkLnR5uqOZnWVm3wk4/SBwMbASeMk5t8mL4AJtPlDOQ29soqi8tjteXkQkLoRceeqcKwRm+h8uDniuCJjuXVjBJSf6Podq6xu7+61ERGJWTBUopST5E3uDEruISEdiKrEnJxqgHruIyJGEMt0xycz+z8zeNbMnOmiTaWZz/W1+6X2YPqn+HnudeuwiIh0Kpcf+SWCFc24aMNDMTgnS5lrgPX+b8WY2zssgmzSNsSuxi4h0LJSbp68AL5tZEtALXwFSoBogw8wMSAO6ZdqKbp6KiHQulHns5c65SuBdYJ9zbkuQZs8As4G1wDrn3ObABl5UnurmqYhI50IZY88zs1TgLKC3mQVbB+Y24DHn3Figj5mdFdjAi8rTFPXYRUQ6FcoY+7eBzzjnGoBKID1Im2yg2n9cA2R5E15bKc03T113vLyISFwIJbE/AtxgZgXAQWC9mf0qSJuv+tuk41tiwHMtY+wN3fHyIiJxodObp865XfiW5W3tOwFtCoFp3oUVnHrsIiKdi80CJd08FRHpUEwldt08FRHpnCeVp/52t/r3PZ1nZinehtn8HiQnGnUNjVTXNfCHd7bQ0KhhGRGR1jypPDWzkcB459x0YB4wxNswW6QkJlBb38iDr2/k7pfW8sKKXd31ViIiMcmrytML8M1xXwjsAx7yLsS2kpMSqGto5GCFr7i1XjdSRUTa8KryNB844Jw7B19v/ezABl7teZqalMAHhYf410e+nnpmashLyouI9AheVZ6WAuv9x1uAwYENvNrzNDstmbV7Wr40JNhRv5SISFzyqvJ0KTDFfzwKX3LvFr3Sk9s8rtEMGRGRNjypPHXOFQBFZvYhsN4594H3ofr0ymib2KvrVIUqItKaJ5Wn/nZf9SqoI8lRj11E5IhiqkAJICctILHXKbGLiLQWc4m9aTmBG6aNAKBGC4KJiLThWeWpv+3NZrbAu/Da++q5x3PnpSdy+5xxJBi8t6WYf3y4g4qa+u58WxGRmOHVnqeY2TDgOg9jC+q4PhlcN20EZkZSYgKLNhVx63Mrue35Vd391iIiMSGUxP4K8OtOKk8BHsC3k1LYtF4MbEtReTjfWkQkanlSeWpm1wArgDUdvY5Xlacd0Uq+IiI+XlWezsG3XsyzwCQz+3pgA68qTzvinNaMEREBjypPnXPXOOfOBq4GljrnHvY2zOAunTio+VjL94qI+Hi152lEPPBfLfdxG9RjFxEBPKw89bctBGYee1ihSWi1AlijeuwiIkAMFih1RD12ERGfuEnsjf5ZMU+/t429JdWRDUZEJII8qTw1nyfN7D0ze8E/5z2sGhoduw5Xcfu/P+Yrf1ka7rcXEYkaXlWeTgOSnHNnADnARR7GGJL6xkaqan3LCpRU1YX77UVEooZXlaf78FWeAtR6FFtIZk8YAEB5TT0VNb4FwVIS42aESUSkyzypPHXObXTOfWBmVwApwPzANt1VefrQZ0/ly+eMpLqukeJK32dKcpL2yxORnsurylPM7DLgW8Cl/mKmNrqr8jQpMYH87FSA5pum6rGLSE/mSeWpmQ0Avgtc4pwr8zbEzmWl+u7V7vEn9mQldhHpwbyqPP0iMBCYb2aLzOwGj+M8okx/Yn/w9Y0ApCQpsYtIz+VJ5alz7hfALzyMq0sCe+hmGmMXkZ4rLrq2fTJT2jyuqdN2eSLSc8VFYp86og8Lbjm3+XG1EruI9GBeVZ6mmdmLZrbCzJ62CIyFjOqXxdIfzuT8MflU12nXDRHpubyqPP0csNM5NxHoDVzoYYwhy8tKJTc9mSr12EWkB/Oq8nQG8Jr/+A0g6Fz3cMhJT6a0WksKiEjP5UnlKZAHlPiPS4E+gQ26e8/TJn0yUyipqqNem6CKSA/lVeVpEZDrP871P26ju/c8bdInMwXn4FevbuCzj7/H+r1hr5cSEYkoTypPgddpWdFxBvCmN+F1Xe8M39THx97eTMGWg3zi/oU8VVCoPVFFpMfwqvL0r8BgM1sJFONL9BHRKyO53bk75q7mH0t2RCAaEZHw86rytAaY42FcR214XmbQ8+XV9WGOREQkMuKiQKm14/pksPYns+jnX/GxiVYZEJGeIu4SO0B6SmK7c4crNQVSRHqGUGbFdLqfqZllmtlcf3XqL7sn1GPz8JubeLqgkKpaFS+JSHwLpcceyn6m1wLv+atTx5vZOA9j9Mztc1fzv/9aFekwRES6VSiJPZT9TGuADP8aMWlHaBdxz3+0i1dX7410GCIi3SaUytNO9zMFngFmA2uBdc65zYENwlV52uRIuyjd+PTSbn9/EZFICenmaWf7mQK3AY8558YCfczsrMAG4ao8bfLEdVO4ftrwbn8fEZFoE8rN01D2M80Gqv3HNUCWN+EdvTEDsvnRpePJSet0qr6ISFwJpcceuJ/pfwepPH0E+Kq/OjWdCFaeBiq47YJIhyAiElahVJ52up+pc64Q3+yZqJOZmsRPLh/PHXNXt3tu6bZiGhp9OzCJiMSLuCxQCvSFM4e3O7dwwwGufLSAq35XEP6ARES6UY9I7MF84YkPIh2CiEi38KTy1N/uVjN7x8zmmVmK96F6Y+a4/pEOQUSkW3lSeWpmI4HxzrnpwDxgiKdReigvM2o/c0REPOFV5ekF+HZXWghMB7Z6EJunpp/QF0AbXYtI3AtlVsxGgE4qT/OBA865y/xTHs8G3mndwMxuBG4EGDp06DGG3XW//8JkDlXW8rf3t4f9vUVEwsmrytNSYL3/eAswOLBBuCtPA6UlJzIwN51vXHACz954BsPyMpqfu/OF9lMhRURilVeVp0uBKf7jUfiSe1RKTkzgjJF51NY3Np/78+LCyAUkIuIxTypPnXMFQJGZfQisd85F/VzC+66aSHpyy4Ycn350MWt2l0YwIhERb5hzLuxvOnnyZLdkyZKwv2+gh17fyH2vbWh+fO7ofJ68YWoEIxIR6ZiZLXXOTe6sXY8tUAKoa2z7oZabnhyhSEREvNOjE3t9Q2Obxy+s2M3zy3ZGKBoREW94Vnnqb3uzmS3wNsTuUxeQ2AFu+ccKXlq5JwLRiIh4w6s9TzGzYcB13oXW/eoagt9f+Nozy8IciYiId7yqPMXf5rZjjiiM6ht9PfbvzRrLwNy05vPH52dGKiQRkWPmSeWpmV0DrADWdPQ6ka48DWbMgBwATh6SS0ZKy9TH0f2zIxWSiMgxC2nfuBAqT+cAQ4FPAGPM7OvOuYdbN3DOPQ48Dr7pjscUtUc+d/pQJg3tzYmDcshIabkUlbVaT0ZEYlenib1V5eks51xFsDbOuWv8bYcDfwhM6tHKzDhxkK/Xnphgzecra+sjFZKIyDELpcfeuvIU4E/AOOfcd7ozsHCzlrxOZW0D+8uqSTQjLys1ckGJiBwFT/Y8bdW2EJh5jDFFRFpSyxj76t2lTP3p62SmJLL6J7MiGJWISNf16AKl1rLT2n/GVfjH2hsaHTX1GncXkdigxO6XFSSxN7nhzx9yyo9fIxLr6oiIdJUnladdqU6NVtmpHYf89oYDVNU1sHZPR6sWi4hED68qT0OqTo1mpw3rDcBv/mtih21KqurCFY6IyFELpWcdSuVpqNWpUevyUwYzZkA2YwfkcPPfVwRtU639UkUkBnTaY3fObXTOfXCkytNQ2sSCsf5K1NZ+Pm9t8/H1f/6Qv32gPVNFJLp5tedpp23M7EYzW2JmSw4cOHAsMYfV795uu8vfD/61KkKRiIiExpM9T0NpE+nNrL2Sk6bNOEQkunmy52mQNjd4HWi06JWRTGOjpj2KSPTq0XuedmTbwQr+31NL2LCvHIALT+zPa2v2tWnzjy+fydQRfSIRnoj0UNrz9BgMy8vk9BF5zY+vOHVwuzYLN8TOfQIR6VlirpAoXL47awyDe6dz4/SRJLRa+bFJkFMiIlHBq8rTNDN70cxWmNnTZhbzaS8nLZmvnHt80KQOviV/RUSikVeVp58DdjrnJgK9gQu9CzE6NeX1jfvK2LRfSw2ISPTwas/TGcBr/uM3gPOPMa6o8/Z3z2uzGUddg2+/1At/s5CZv14YqbBERNrxpPIUyANK/MelQNxNFxmWl0nvjJY57I+8uZl6f3IXEYkmXlWeFgG5/uNc/+PA14jJytPWqgL2Qv2gsDhCkYiIdMyTylPgdVrG3mcAbwY2iIfK06qARcBW7ixpPq6tV+9dRKKDV5WnfwUGm9lKoBhfoo87TQWnj157GgDbDlY2P3eoMiYXtRSROOTJnqfOuRpgjldBRbuLxg9gZN9MSqtb1mcvKq+hf05aBKMSEfFR5WkXPHD1KZw7Op/EBCM7PZnSVhtv3PrPlYz54bwIRici4qPK0y64/JTBXH6Kb3mBnLQkSqvrm59bvbs0UmGJiLShHvtRygnosTdp0MqPIhJhoU53TDaz/xzh+Uwzm2tm75rZL70LL3rlpCVTVF7T7nzgzBkRkXALZbpjOrCUIy8TcC3wnnNuGjDezMZ5FF/UyklPoqzVUEyTypr250REwimUytMq59zJwM4jNKsBMvyLf6URoxtad0V+VmrQ8xW16rGLSGR5Ncb+DDAbWAusc85tDmwQD5WnrR3XJyPo+eeX7eTxhZtZ1ap4SUQknLxK7LcBjznnxgJ9zOyswAbxUHna2pDe6UHPP/TGJn728joufXhRmCMSEfHxKrFnA9X+4xogy6PXjVod9dhFRCKty4ndzEYEWVLgEeCrZlYApBOnSwq0lpOWzDu3ns/A3DQ+PWlI0Dab9pcTiT1lRaRn02bWHnhtzT7+31PBf58Hrj6luahJRORYaDPrMBqZn9nhc6pIFZFwU2L3wFD/ePvUEe33F9HOqCISbp5Unvrb3Gpm75jZPDNL8Sa82JCcmEDhPZfwvVlj2z+pzC4iYeZJ5amZjQTGO+emA/OA4HcT41zrrfOaGEZlbT2NWkNGRMLEq8rTC4DeZrYQmA5s9Si+mNIns/0XlbLqOk68Yz6Pvt2uZktEpFt4NcaeDxxwzp2Dr7d+dmCDeKs8DSYnrX2Pvenm6fzVe8Mdjoj0UF4l9lJgvf94C9Bufl+8VZ4Gk5DQfkB91S7f0gLD8jqeOSMi4iWvEvtSYIr/eBS+5C60rM+eqJuoIhImnlSeOucKgCIz+xBY75z7wKsA40VJkE05RES6Q8hb4znnRvn/vxX4TpDnv+phXDFrTP9s1u8ra3deiV1EwkUFSh770/VTuOTkge3OlwbZlENEpDsosXtsUK90vveJ9oVKJVV1rN9bxr7S6iA/JSLiHc8qT/3tbjazBcceVmwb2CuNi08awIvfaJn1WVJVxyfuX8is+xdGMDIR6Qk6HWP3V56+D4zupN0w4DogPiepd0FyYgK/vXYSAF88cxjLd5awYsdhAA5VaqxdRLqXV5WnAA/g20lJWvnx5RP4TAfrtYuIdAdPxtjN7BpgBbDmCG3ivvK0IznpLRWpZmjzDRHpVl7dPJ2Db72YZ4FJZvb1wAY9ofK0I1mpic3HzkFxRW0EoxGReOdJYnfOXeOcOxu4GljqnHvYi9eNF/UNvh56apLvck+6ewGvfLxHKz6KSLfwas9TOYJzx+Rz3VnD+e21pzWf+8pflvHPZZ3dthAR6bqQE3vrylPnXLvKU/9zhc65mV4FFy9SkxK587LxTBic2+b8rf9cqWEZEfGcCpTCqG9Wartz9y/YEIFIRCSeKbGHUWKCccuFbcsBNK9dRLzmSeWp+TxpZu+Z2QtmFvLiYj1NZmrbS7O/tFoLhImIpzzZ8xSYBiQ5584AcoCLvAkv/qQktb3k728tZuKPX2XRxiIAauobePq9bZoxIyJHzavK0334Kk8BdDfwCFITWy55Vqve+5aicgAefWszt//7Y+au2BX22EQkPng1j32jc+4DM7sCSAHmB7bpyZWnrTX12Ptlp/KNGaOazyeYb4ulw/4x98MaexeRo+TZzVMzuwz4FnCpc64h8PmeXHnaWlNiT05MoF9OyyyZ0uo6FqzZx+FK3xeeBg3FiMhR8uQmp5kNAL4LzHLOVXjxmvEqxT8Uk5Ro9M9Oaz6/61AVv3xlffPjRq0nIyJHyavK0y8CA4H5ZrbIzG7wJLo41NRjT0wwememNJ/fU9J2A46GxrCGJSJxxJM9T51zvwB+4W1o8al5KCYhgeF5mfTNSqWovIbdh6vatFOPXUSOlgqUwiy51VBMekoiS344k1OO68W2g5Vt2jUtHCYi0lVK7BGS1Gra44TBOVTVtb3fXFmnza9F5Oh4VXmaZmYvmtkKM3vazD93T9qp9w+eJye0XKKpI/LatauqbTexSEQkJF5Vnn4O2Omcmwj07qRtj9a0pMCIvpnN56YO79OuXUlVHV/76zIue3hR2GITkfjQ6c1T51wVcLKZbTpCsxnAc/7jN4DzgVePPbz4M2FwLo9eexrnjenXfG5Ablq7dvNX76W6TlNjRKTrvBpjzwNK/MelQLsuqCpPW8w+aSDpKYltzv39xjPaPG6d1Cf8aD4/f3ltWGITkdjnVWIvApp2kcj1P25DladHdvrI9uPsTcpr6vndwi1hjEZEYplXif11WlZ0nAG86dHr9ljBbj/X1OuGqoh0zqvK078Cg81sJVCML9HLUfro9gsZlJve7vzekmqcc/z0pTWs2lkS5CdFRLyrPK0B5ngbWs8z92vT2Li/nN6ZKQzvm8GugGrUZ97f3jwk81TBNtbfPTsSYYpIlFOBUhSZeFwvPj1pCACj8rPaPd96nL2mXjNmRCQ4JfYo9bUZo/if846PdBgiEoNCKVDqtKrUzDLNbK6ZvWtmv+yeUHuWftlpzDl5EADnj9EsIhEJXSg99lCqSq8F3nPOTQPGm9k4D2PssU4clMOCW87hvqtOiXQoIhJDQknsM4DX/MdNVaWBaoAMf28+De176plR/bLpk5nCwCDVqSIiwYSS2DutKgWeAWYDa4F1zrnNgQ1UeXpsXv7m9EiHICIxIpTE3mlVKXAb8JhzbizQx8zOCmygytNj0zszhVdvPqfNuaXbDrG1SDsRikhboST2UKpKs4Gmvd1qgPZz9eSYje6f3ebxlY8u5vxfvRWZYEQkaoWS2AOrSjcHqTx9BPiqmRUA6ajyVEQkYkJZtjdYVWlg5WkhMM27sKQjZhC4Hern//g+/3PeKM48vuOFxESk51CBUoxJT05sd+6djUV89vfvRSAaEYlGSuwx5oJx/Tt8rrHRcahCM01FejpPKk/97W41s3fMbJ6ZpXgfqgDc++mTWXDLOZw0OLfdc3MeWsSpd71GdV0D1XVa4lekp/Kk8tTMRgLjnXPTgXnAEE+jlGZpyYmM6pfNsLwMAK47azhXnDoYgDV7SgGY8tMFTLl7AcUVtazZXRqxWEUkMryqPL0A6G1mC4HpwFZvwpOO5KQnA3B8fiYXjOvX5rmy6nrKauq56ncFXPzgO7jAu60iEte8qjzNBw44587B11s/O7CBKk+99c0ZJ3D+mHwumziYnLTkoG027S8H4EBZTThDE5EI86rytBRY7z/eAgwObKDKU28NyE3jT9dPJTcjmekn9D1i28KDlWGKSkSigVeVp0uBKf7jUfiSu4SJmXHJSQM7fL5Qyw6I9CieVJ465wqAIjP7EFjvnPvA+1DlSO7+5ATOHBm8QOnW51aydNshXvl4L6XVdWGOTETCzSJxY23y5MluyZIlYX/feLeksJhPP1YQ9LmrJg/hH0t2cv6YfP50/dQwRyYiXjCzpc65yZ21U4FSHEkLUpXaZE+Jb422JdsOhSscEYkQJfY4EkpiL6uuD1c4IhIhnlWe+tvebGYLvA1RQpWW3PLHedXkIdz3mYnNj7drZoxIj+HVnqeY2TDgOu9Ck65KTvT9cfbNSuGXn57IlZNaCoBrGxqbjytrW3rtJZV13PL35RRsPhi+QEWkW3lVeQrwAL6dlCRCcv3VqN+5aMwR2y3edJDDlb7Fwib+5FWe/2iXVocUiSOdrsdO+8rTdlnDzK4BVgBrOnoRM7sRuBFg6NChXQ5UOpeWnEjhPZe0OfeNGaP48+LCNmPrX3rKNyPpx5eNb9P2UEUtvTO1fptIrPOq8nQOvvVingUmmdnXAxuo8jQyvn3RGP76pdObH7ceh//RC6vbtL3t+VVhi0tEuo8nlafOuWucc2cDVwNLnXMPexeiHKshvTOaj0f27Xg72oMVWlNGJB54teepRLHeGb6x98smDmJkfiYAnzqt3XI+rNtbxraDvuUHnHPsLalu10ZEop8qT3uIsuo60pITKa+u58mCQr4x4wRKq+q4f8EGnizY1qbturtm8cdFW7l3/nrmfWs64wbmRCZoEWlDlafSRnZaMsmJCfTOTOGmmaNJTDB6Z6bw48sntGs76/6F3Dvft1jnksLicIcqIsdIiV3aab3M7+1zV7NUyxCIxJQjJvZQqk7N50kze8/MXjCzUKZQShR58LOnBj1/zmjf7KU75n7MHXM/blPYBL7lgK98dDGbD5R3e4wiErrOeuyhVJ1OA5Kcc2cAObTMoJEYcdnEQXzGX6V6xamDuWnmCRTecwlPXj+F2RMGsHp3KU8VbGPu8t1tfu57/uWA39+i4RqRaNJZYg+l6nQfvqpTgFqP4pIwu+Wi0Zw5Mo/bZo/lppmjAd8GHqcO7dXc5snFhRQWVTDz12/z6FubeX+rL6EH9uS3Haxg7vJd4QteRNroLLF3ut+pc26jc+4DM7sCSAHmB3sh7Xka3QbmpvO3G8+gX05am/ODe7XMgV+3t4yvPbOMTfvL+cUr65rPb9pfztzlu5o3zb7qdwV869nlVNT4En51XQMHy9vOkb/578u5d/46RMR7nY2Hh1J1ipldBnwLuNQ51xCsjXPuceBx8E13PKpoJewG5LZN9Kt3l7Zr8+yHO3j2wx1kpSaRm57MvlJfEl+3t4yRfTM59S7fl77Cey7BOccT7xbyr498PfrvfmKsp/GWVNZxuKqWYXmZnr6uSCzprMfeadWpmQ0Avgtc4pwr8zY8ibSTh+Ty5XNHcuelJ9I3y7eOTOtlCYbltfTo//vJJW12cLry0cV88rfvNj/eX1bNsu2HuevF9ksKNTQ6zvnlm/z9w+0ALN1WzIOvb+xyvHMefodz732ryz8nEk86S+yhVJ1+ERgIzDezRWZ2QzfEKRGSnJjAbbPHcd20EfzKv7772AE5FN5zCYX3XEL/7LR2P3PS4Nzm422tpk6+sHx3h3uu7i+rZntxJd97bhWNjY4rHy3g169toKrW9wVwR3ElP5r7cdCx++q6li+JO4qrAJqHgWLZ7sNVbNinvpJ03RETu3Ouxjk3xzl3snPu8865rc657wS0+YVzbpRz7mz/f090b8gSKU3rzEwb1bJp9mdPP65du7lfm8aMsf2aH//vxeMAuPultVz/pw/btP3NaxsY/v2XOPPnb7S8zw9ebj4u9C9xMOO+t3iyYBvfenZ5m5//wztbGHv7K7y4cjcNjS0jfNuL224ssq+0mjfX7w/tF40SZ93zBhf9ZmGkwzgqjY2uzZ+HhJcKlCRkQ/MyWHDLudzsnzUDcMWpQ/jgBxew+Pszms8lJBi//8JkHvvcaZw2tFfQdWmaPNDJcMvizQdZUlhMXUNLkvjV/PWc/rMFlFTVcfdLawH4+jMfsXBjy035lTsP8/k/vs+lDy3ia88s4/Sfvc71f/qQt9bvZ39Z0zaBdRyq8E3kOlhew67DVZx053x+9rLvNZfvOMxNz35EZW091XUN1Dc04pyjrLqOv7y3jcZWiau0uo7NB8pZuu0QwZbpuO/V9fx83toj/q4d6SxBbthXxq7DVazbW9rmJvXyHYf5/cItXXqvj3eVcO/8dUF/h2C2H6yksKii3fnP/fF9Lnt4UZfeO5r9fuEWnl+2M+hzFTX1vLsp6O3HiNFaMeKZQxW11Dc68rNT2z332Nub2VFcyeYD5bzXhXnvCQYd5bWJx/VixY7DRxXrijsuYuJPXmVYXgZPXDeFKx55l9JWa9Z//ONP8PAbm3js7c186tTBvLZmH6cM7UXB5oMkJRrVdY3MOXkgD19zGgAXP/AOa/b4bix/56LR5GWlcv6YfnxYWMzbGw7wz6W+pNB6vfw9JVU4BylJCfz7o118/sxhpCYlUlJZx1sb9jd/O1n0vfPJz07lj4u2Mu34vgzpnU6fzBR+Pm8dK3Yc5v2txQzMTWNPSTW56cksv+NCzIzh338JgFV3XsSGfWXkZ6WxvbiSU4f2IjM1if9bsoOSqjqunzaCxARf7eG421+hqq6B5XdcSK+MFNbtLWVonwxq6xt5c/1+PnnKYJbvOMwVv11M36wUisp9H4x/+e/TOev4PBISjO0HKznnXt/tuHV3zWJHcSVpyYlkpibxs5fXct6YfM6Z+VonAAAKA0lEQVQcmUdeViq19Y2s2HmYzzxWQGKCcenJAzl9ZB5XnDqYF5bvZuaJ/emTmUJDo6Ng80HOGNmHpMTg/dFVO0sY1CuNvCzf3z/nHE01lY+8uYmTBucyflAOjy/cws0XjmZ/aQ21DQ2M6pfd7rUOV9biHM37EzRdyx9eMo4vTR+Jc477F2zkvDH5PPb2Zuav3scb3z6XkflZlFXXUVhUyUlDcqmuayApwZpjrqlvIDWp472JOxPqWjFHTOxmlgb8EzgOWAl8wQX8QChtAimx92wllXWUVNXRLyeVfaXVJJhR3+hYvLmI5MQE7pj7MWMH5HD9tOHthl4CjR2QTVF5TXOCAd/N3eq6Rr4/eyyj8rOaNxbpDldNHsKMsf34yl+WhdR+6og+jOmfzerdJSzb3vZD6a7LxzNuYA7PvL+d5z9quZdw3VnD+fPiwjZt+2SmUFwRvGwkJy2JcQNzmusMgrlgbD9eX+cbmspOS+Kuyydw099brvWlEwdx6nG9uPulNfTOSOGg/70yUhKprA068Y3zx+TzxbOGU1hUwZ3/8d0gH9I7nZ2HqoK2X3DLOTxVsI2nAhahAxjcK51dh9v/3DcvOIFbLvR9Y1y1s4QHXt/ANacP5YY/+/6M+2al8KnThvD4wi0kJRiv3HQOD72xsbm4bvaEAcz7eG+b19z684t5ceUeMlMT+f5zq9hf5vvWk5+dys0zRzOoVxrXtRpCfPKGqf5vNuvbxdf0AQvwg4vH8rOX13HxSQPIy0ylrLqOjfvLufK0Idxw9oig16QzXiX2LwGTnXNfMbMXgQedc692tU0gJXYJRUOjY+7yXdQ3OK6cNIRDlbU8+tZmrjh1MPcv2EDhwUpe+ubZlFXXU1nTwP88s5Szju/LzHH9Ka6oZdaEAYBvyGXu8t0UbDnISyv3NL/+uaPzKa6oZdWuko5C4MyRecw8sT8fbT/Ei61+NppkpiRSUdtAr4xkkhISKCqP/Lr6eZktHwZeyk5NIic9OWjSP1pH+lbotZSkBB7//CTOG9Ov88ZBeJXYnwGec849Z2a3APnOudu62iaQErscq3r/5twdfS0/ktLqOnYUV3J8fhZ1DY088/528rJSOXtUX5ZtP8RZx+exvbiSP7yzldkTBjD7pIE459hTUk3frFT+vXwXnzhxAH9ctIWiilo+2n6Yuz85gc0Hypk1YQA1dY18sLWY6aP7ctd/1rBs+yEG987g9BF9cM6Rm5ECzrFiZwn/XLqTqyYP4dZZY3mqYBtbiyo4XFnL8flZlFbXUbD5IAcrajFg3MAcJgzO4cyRfTlUWcvra/cx5+RBnHF8HksKizljZB7/WbG7+b4DwGcmDWHcwBzOGZ3Pb9/cxMTjepGXlcLDb2zi5CG5vLOxiFtnjWFHcRU5aUmM7p/NNX94n2F5GRzXO4Mzj8/j1TX7+Mll40lPSWy+mfvTKyYwcUgvvvm3j9hSVME3Z4xi/b4yVuwoIT0lkT9fP4Xdh6tZvLmIAblp7C+t4eqpx/Gvj3YxICeNJxcXsmJnCWbwxHVTOH9MP15YsZtv/u2jNn9Wl00cxK+vmsiTBdu479X1DM/LbB7y6pedSl1DI4er6vjVpydyyckD+cG/VvH8sl3MHNefBWv3Nb/O6P5Z7CiuIjU5gf7ZaazfV9b8gRho6vA+zJowgJ8ETMudNX4Ar6z29fa/cu7xlFT5/h7915TjmDA4lx/+exUrd5RQ1sGMrCG907n7kxOOOqmDd4l9PnCvc26Bv2c+xTn35a628bdrvefppG3b2n/9EulpDlXU0isjuXksOFB1XQPJiQnNY+ChKCqvoW9W+/scodqwr4wRfTNJDvKh+ea6/UwZ0Yes1JbaxsZGR0IX4muyv7SaBucYmJvefO7lVXuoqm3g3DH59M5ICfp71zc0UlpdT3ZaErX1jWwtqmBCqym2TWPra3aX8tyynVw6cRCnHNeL2vpGkhKsTaxl1XVs2FfOpGG9Kauu4z8r9vDZqcdhZhQWVbBi52HOOSGf5TsOc96YfN7ecIBJw3qTnZbc4e9VVF5DcUUtB8pqOGlILhv2ljGoVzqDeqV3+DOh8iqx/xV43t8b/zbQxzn3v11tE0g9dhGRrvNqo41OK09DbCMiImHiReVpYJvXvQ9TRERCdcRFwJxzNcCcgNOBlafB2oiISISo8lREJM4osYuIxBkldhGROKPELiISZ5TYRUTiTERWdzSzA8DRlp72pYMt+qKU4u0+sRQrxFa8sRQrxFa8xxLrMOdcfmeNIpLYj4WZLQml8ipaKN7uE0uxQmzFG0uxQmzFG45YNRQjIhJnlNhFROJMLCb2xyMdQBcp3u4TS7FCbMUbS7FCbMXb7bHG3Bi7iIgcWSz22EVE5AhiJrGbWZqZvWhmK8zsaetoZ4IIMbNkM/uP/7hdrNEUvz+eJ83sPTN7wcyyojVeM0sys/8zs3fN7Ilov7b+mG82swVm1tfM3jGzVWZ2j/+5duciGOcsM9tpZov8/02MgWt7q//6zTOzftF6fc3svFbXdYeZfTGc1zZmEjvwOWCnc24i0Bu4MMLxNDOzdGApLTEFizWa4p8GJDnnzgBygBuCxBYt8X4SWOGcmwYMBL4exbFiZsOA6/wPbwJeAiYCs81sdAfnIulR59zZzrmzgSlE97UdCYx3zk0H5gH3E6XX1zn3VqvruhLIJYzXNpYS+wzgNf/xG8D5EYylDedclXPuZGCn/1SwWKMp/n3AA/7jWuBOojfeV4Bfm1kS0As4LUhc0RIr+K5r056/M4DXnHONwNu0ijXgXCRdaWYfmNlzwAVE97W9AOhtZguB6cAIovz6mlkGMAo4gzBe21hK7HlA03bypUCfCMbSmWCxRk38zrmNzrkPzOwKIAXft42ojNc5V+6cqwTexfeBFLXX1syuAVYATbsgR22sfpuB251zU/F9G/oU0R1vPnDAOXcOMASYSnTHC75e+OuE+e9CLCX2InxfZ/D/P5rLh4PFGlXxm9llwLeAS4H9RGm8ZpZnZqnAWfi+rk4IEldUxIpvw5kLgGeBSfhKx6M1VvDteLbAf1wINBLd8ZYC6/3HW/DFHM3xgu/f14uEOSfEUmKPpb1Vg8UaNfGb2QDgu8AlzrmyDmKLlni/DXzGOdcAVAI/DRJXVMTqnLvGP6Z6Nb5vQY8AF5lZAnBu61gDzkXKLcDV/lgm4LvWUXlt/Zbiuw8AvuGN9UTx9fXfDD0f3zBLWP+NxVJij6W9VYPFGk3xfxHfV+/5ZrYISA4SW7TE+whwg5kVAAeBP0ZxrIEeBC7Gd/PsJefcpg7ORcrDwPXA+8C/iPJr65wrAIrM7EN8Sf0LRPf1nQKsds5VE+acoAIlEZE4E0s9dhERCYESu4hInFFiFxGJM0rsIiJxRoldRCTOKLGLiMQZJXYRkTjz/wETz9NMe1ZPdAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 隐藏节点\n",
    "hidden_size = 256\n",
    "\n",
    "# 编码器与带注意力机制的解码器\n",
    "encoder1 = EncoderRNN(input_lang.n_words, hidden_size)\n",
    "attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, 1, dropout_p=0.1)\n",
    "\n",
    "if use_cuda:\n",
    "    encoder1 = encoder1.cuda()\n",
    "    attn_decoder1 = attn_decoder1.cuda()\n",
    "\n",
    "# 正式训练\n",
    "trainIters(encoder1, attn_decoder1, 70000, print_every=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们来看看翻译的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 我仍是这里的老板。\n",
      "= i m still the boss here .\n",
      "< i m still the boss here . <EOS>\n",
      "\n",
      "> 你一直忙。\n",
      "= you re always busy .\n",
      "< you re always busy . <EOS>\n",
      "\n",
      "> 他不是很擅長數學。\n",
      "= he is not very good at mathematics .\n",
      "< he is not very good at mathematics . <EOS>\n",
      "\n",
      "> 她中文有进步。\n",
      "= she is progressing in chinese .\n",
      "< she is progressing in chinese . <EOS>\n",
      "\n",
      "> 她既没钱，也不出名。\n",
      "= she s neither rich nor famous .\n",
      "< she s neither rich nor famous . <EOS>\n",
      "\n",
      "> 我太忙了，無法幫助他。\n",
      "= i m too busy to help him .\n",
      "< i m too busy to help him . <EOS>\n",
      "\n",
      "> 他是我弟弟的一個朋友。\n",
      "= he s a friend of my brother s .\n",
      "< he s a friend of my brother s . <EOS>\n",
      "\n",
      "> 我来自东京。\n",
      "= i m from tokyo .\n",
      "< i m from tokyo . <EOS>\n",
      "\n",
      "> 他很容易遲到。\n",
      "= he is apt to be late .\n",
      "< he is apt to be late . <EOS>\n",
      "\n",
      "> 他不是美国人。\n",
      "= he isn t an american .\n",
      "< he is not an american . <EOS>\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:22: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:33: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    }
   ],
   "source": [
    "evaluateRandomly(encoder1, attn_decoder1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"5.2\"></div>\n",
    "\n",
    "## 5.2 Attention过程可视化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention机制的一个有用的特性是其高度可解释的输出。\n",
    "\n",
    "由于它用于对输入序列的特定编码器输出进行加权，所以我们可以看出在每个时间步网络关注的位置。\n",
    "\n",
    "通过`plt.matshow(attentions)`，将Attention输出显示为矩阵，其中列是输入步骤，行是输出步骤："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:22: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:33: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f1fd0977630>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARUAAAEACAYAAACd9eLKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAACx9JREFUeJzt3V+MZnV9x/H3BxdYFsXyz4iANIS0GpqWhIVaFqxAFU1lW9I0MdgEy8UkJCbGNppw0cte6B0JpIY0TUxTY9I2RGuLm6I2CllYh7TQViUKYthCDIuN2FKXP357sUM6WalzZuZ75uw5vl/JJg97hud8zwTe+5tnz/P8UlVIUpeTph5A0rIYFUmtjIqkVkZFUiujIqmVUZHUapKoJNmd5AtJHknyl0kyxRzblWM+neTBJJ9PsmvqmbYjyUeT3Df1HFuV5ONJvpbk3iSnTD3PViQ5PcnnkjyQ5JNTz7MVU61U/gA4XFW/BpwJvHuiObZrH7Crqt4BnAG8Z+J5tizJRcCHpp5jq5JcDFxaVdcA9wIXTDzSVn0QeLCq9gGXJnn71ANt1lRRuQ74x7XHXwaunWiO7fo+cMfa4xenHKTBHcDtUw+xDdcDZyb5KnAN8N2J59mqo8CetdX7bmb439VUUTkb+OHa4+eBsyaaY1uq6ttVdSjJTcApwIGpZ9qKJDcDjwDfmHqWbTgXeLaq3smxVcrVE8+zVZ8B3gd8E/hWVT0+8TybNlVUjgBvXHv8xrV/nqUk+4GPADdW1StTz7NF7+fYn/SfBS5P8uGJ59mK54HH1h4/AZw/4SzbcTvwqap6G3BWkqumHmizporKl/i/1x+uA74y0RzbkuTNwMeA366qH009z1ZV1c1VdTXwAeDhqrpz6pm24GHgirXHl3AsLHP0BuDHa4+PAq+fcJYtmSoqfwWcn+RR4Acci8wc3QKcBxxIcn+SW6ce6OdVVR0EjiT5OvBYVR2aeqYtugu4LclB4DRm+P9GfJeypE7e/CaplVGR1MqoSGplVCS1mjwqSVamnmG7lnANsIzrWMI1wLyvY/KoALP95q2zhGuAZVzHEq4BZnwdJ0JUJC1I630q55z1unrrhZt79/+R517hnLNfN/jrv/Ovp292rM3b5LfkJY5yMqeOM8sOWsJ1LOEa4MS8jh/xn0eq6tyNvq718z/eeuEuHvjieZ1P+VN+56LfGPX5Aerll0c/hzQ399XffG/I1/njj6RWRkVSK6MiqZVRkdTKqEhqZVQktTIqklr9zKgsZX8eSTtno5XKUvbnkbRDNorKUvbnkbRDNorKIvbnkbRzNorKhvvzJFlJsppk9chzc932RlKXjaKy4f48VXV3Ve2tqr2bebexpGXaKCpL2Z9H0g75mR99UFVHObYlpiQN4s1vkloZFUmtjIqkVkZFUiujIqmVUZHUyqhIatW6Rcd3Hj2d/edf0fmUP+XA06ujPj/ADW+5bPRzSEvlSkVSK6MiqZVRkdTKqEhqZVQktTIqkloZFUmtjIqkVkZFUqtBUUlycpK/G3sYSfO34W36SU4DHgJ+afxxJM3dhiuVqvqfqvpV4PAOzCNp5rb9hsIkK8AKwG72bHsgSfO27Rdq1+/7czKndswkacb82x9JrYyKpFaDo1JVl4w5iKRlcKUiqZVRkdTKqEhqZVQktTIqkloZFUmtjIqkVq2bie2EJWz0deDpfxn9HEv4PmmeXKlIamVUJLUyKpJaGRVJrYyKpFZGRVIroyKplVGR1MqoSGq1YVRyzKeTPJjk80lmdxeupJ0zZKWyD9hVVe8AzgDeM+5IkuZsSFS+D9yx9vjFEWeRtAAb/ihTVd8GSHITcApwYP1xNxOTtN7QDdr3Ax8BbqyqV9YfczMxSesN2aD9zcDHgPdW1X+PP5KkORuyUrkFOA84kOT+JLeOPJOkGRvymsongE/swCySFsCb3yS1MiqSWhkVSa2MiqRWRkVSK6MiqZVRkdTKqEhqZVQktTIqkloZFUmtjIqkVkZFUiujIqmVUZHUyqhIajVk359dSf46yQNJ/mInhpI0X0NWKr8LPFJV+4Dzklw28kySZmzIboNfBP5hbWfCXwCeH3ckSXM25DNq/wsgyUPAM1X1xOhTSZqtIa+pnJ3kVOAq4Mwk1x53fCXJapLVlzg61pySZmLIayp/DPz+2iZiLwCnrT/oZmKS1hsSlbuAW5McBJ7juG1PJWm9Ia+p/Adw3Q7MImkBvPlNUiujIqmVUZHUyqhIamVUJLUyKpJaGRVJrYyKpFZD3qWsZjdccPno57jn8MHRz3HThb8++jmoGv8cauVKRVIroyKplVGR1MqoSGplVCS1MiqSWhkVSa2MiqRWg6OS5KNJ7htzGEnzNygqSS4CPjTuKJKWYOhK5Q7g9jEHkbQMQ/b9uRl4BPjG+ONImrshK5X3A9cDnwUuT/Lh9QfdTEzSekO26LgZIMkvAn9eVXced/xu4G6AM3KWbymVfs75V8qSWg3+PJWqehL4rfFGkbQErlQktTIqkloZFUmtjIqkVkZFUiujIqmVUZHUyn1/pvCTV0Y/xZ6TThn9HO7Jo9fiSkVSK6MiqZVRkdTKqEhqZVQktTIqkloZFUmtjIqkVkZFUqshn6b/3iSHk9y/9uuXd2IwSfM09Db9P6uqPx11EkmLMPTHn99LcijJ3ybJqBNJmrUhUXkc+JOquhI4D/jNcUeSNGdDfvz5AfDqxuxPAm9afzDJCrACsJs9nbNJmqEhK5U/Aj6Q5CTgV4B/W3+wqu6uqr1VtfdkTh1jRkkzMiQqdwJ/CDwE3FNV7qks6f81ZNvTZ4B3jT+KpCXw5jdJrYyKpFZGRVIroyKplVGR1MqoSGplVCS1MiqSWhkVSa2MiqRWRkVSK6MiqZVRkdTKqEhqZVQktTIqkloZFUmtBkUlyceTfC3JvUlOGXsoSfM1ZIfCi4FLq+oa4F7ggtGnkjRbQ1Yq1wNnJvkqcA3w3XFHkjRnQ6JyLvBsVb2TY6uUq9cfTLKSZDXJ6kscHWNGSTMyJCrPA4+tPX4COH/9Qff9kbTekKg8DFyx9vgSjoVFkl7ThlGpqoPAkSRfBx6rqkPjjyVprobspUxV3Tb2IJKWwZvfJLUyKpJaGRVJrYyKpFZGRVIroyKplVGR1GrQfSqanxvectno5zhpz57Rz3HlwR+Ofo7V/ReP+vwvP/X0qM8PsOtN54x+Dp4Z9mWuVCS1MiqSWhkVSa2MiqRWRkVSK6MiqZVRkdTKqEhqZVQktRqy78+7kty/9uupJLfsxGCS5mnD2/Sr6p9Y25Yjyd8D/zzyTJJmbPCPP0n2AJdU1aMjziNp5jbzhsJ3A186/jeTrAArALsZ/w1mkk5sm3mh9kbgC8f/ppuJSVpvUFSSBLgW+PK440iau6ErlSuAf6+qH485jKT5G7qZ2CFg/8izSFoAb36T1MqoSGplVCS1MiqSWhkVSa2MiqRWRkVSKzcT05b95IUXRj/HQ3tPH/0c9dJToz7/PYcPjfr8ADddcOXo5xjKlYqkVkZFUiujIqmVUZHUyqhIamVUJLUyKpJaGRVJrYbs+3N6ks8leSDJJ3diKEnzNWSl8kHgwaraB1ya5O0jzyRpxoZE5SiwZ+3Dr3cDL447kqQ5GxKVzwDvA74JfKuqHh93JElzNiQqtwOfqqq3AWcluWr9wSQrSVaTrL7E0VGGlDQfQ6LyBuDVrTmOAq9ff9DNxCStNyQqdwG3JTkInMZrbH0qSa/a8PNUqupJYN/4o0haAm9+k9TKqEhqZVQktTIqkloZFUmtjIqkVkZFUiujIqlVqqrvyZJnge9t8l87BzjSNsQ0lnANsIzrWMI1wIl5HRdV1bkbfVFrVLYiyWpV7Z10iG1awjXAMq5jCdcA874Of/yR1MqoSGp1IkTl7qkHaLCEa4BlXMcSrgFmfB2Tv6YiaVlOhJWKpAUxKpJaGRVJrYyKpFZGRVKr/wWrZEQD4j5m9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 320x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pair_visual = random.choice(pairs)\n",
    "output_words, attentions = evaluate(\n",
    "    encoder1, attn_decoder1, pair_visual[0])\n",
    "plt.matshow(attentions.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了更好的可视化体验，我们加上坐标轴标记。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:22: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "/explorer/pyenv/jupyter-py36/lib/python3.6/site-packages/ipykernel_launcher.py:33: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input = 我一點也不擔心。\n",
      "output = i am not the least bit worried . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAEMCAYAAABqRBWiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGipJREFUeJzt3X28XVV95/HPl8eEp4AEQSIPKspYasFpLMiDjRExTAGlKIqiRTpGbUWxomideYlTtdKilgE6NkrFUtA61LGKD0h5UkioSQQZQR0aC4IocsEhgiFg7rd/7H1nbq733nNu7tkPZ9/vm9d+ZZ991ll73ZvwO2v/9tpryTYREVGtrZpuQETEXJBgGxFRgwTbiIgaJNhGRNQgwTYiogYJthERNUiwjYioQYJtREQNEmwj+iRpvqTnN92OGE7bNN2AiGEgaT5wETAiab7ta5tuUwyXBNuIHiTtAFwIvM/2PZI+IOkJ299sum0xPJJGiJhGGWgvAt5v+x4A2/8FOF7SYY02LoaKMhFNxNQknQSstX3XJO+9w/ZH6m9VDKME2wGQtDVwku3PNd2WYSDpANv/2nQ7ZkLSb9teK0nAqcA84FLbjzXctBgSSSMMgO1NwOsAJJ0p6WJJF0haLelvGm5eq0j6Y+AKSfs23ZZ+Sfo74C3ly48AxwB7AJ9prFExdHKDbJYk7UPxpTWvDCB7A//V9n2SrrP9xmZb2B6SFgN7AYtt/0rSlcC9tt/UcNN6ebrtIyU9DVhi+z8CSPpGw+2KIZJgO3tHAIuArwKvAA4CxnIzydGMY3sNsGbc6+MkfbvBJvXrAUlnAS8F/kzSzsBJwKZmmxXDJDnbAZP0p8AzgV8AxwLXAWfa/mWjDWuQpBcBb7D9qkneW2n78Aaa1bdyRMJrgXtsf0XSgcDrgfNt/6TZ1sWwSM92liR9AniifLkN8EHgZcBPgXXAojkeaHcBXgOcWQapPce9vdWE1221keLL80WSTgQeBr4F3N9oq2KoJNjO3jMoejmvBEaAnYCjKS6XHwPmdM/H9nrgdIDyUddD2Ty98oEm2jVDfwtsB9wI/BLYkSKNsIzyZ4voJcF29p4FnA28gKI3+yKKvO1C4AvANpK2LkcszHXXAwJW2v5pw22ZiWdOkuq4UNKqRloTQynBdvaeAEaBHwGPAx8GzgVuAZ4DnF++H7A9cDjwFkl7AsvGnspqufslnQvcADxKcfWyBLivyUbFcJnTN8gkHQrcPZtelqRFFDk9U1xeHgr8b2BbYAfgacBXysvpgZK0PcVl7Bdt/3gL69gaeG45UmD88X1t/2gAzRx/ntOBf7F9m6SdgEc9BP8Ay7aeAfwOsDOwHrgZuMD2hibbFsNjTgRbSb8L/AvwT8AlwJdtr5d0OnCO7YEMsJc0D/iu7QMGUd+EuhcCJ1LkDB8B7qG4aXMmcLTtA7ew3m2BG20fOu7Y4cAVwPG21w6g7duU42rvpri59NfAycBHgVOAowb1d1AFSS+xfVW5v7ft+8r9t9k+v9nWxbDo/BNkkpYC/42il/k4RerkS5KOoLisvXJQ5yof3azq8c2HKXLDewHPpQhUe9r+Y+BnW1qp7Scofi9AEVgoAu0pgwi0pZMlfZbi7v0mYANF7vZxip9j3YDOU5X3jtv/+3H7J9XdkBhecyFneyPFTSsoLu2/SdEz/BLFiIFjB3y+SvKztp8ob8hcbvsnkk4A5o+9Pcvqt4P/d6n/Hopc6m2zrHOij1LksvemaO/4Nnf/8irmvLkQbK+l6EFtBH4TeCdwB3AU8D+AXYEHG2tdHyRdSDGsbFfgY5LuALamuDl3NUUvcTYOkvQd4OMUQXF/SfuPe38n25dvScWSnkzRM3w18GzgXorc9rYUVxZvA/bZ4pbXY9/yYRVR/G7G9tve7miRuRBszwDm214p6UsUOc4/tH27pL+imEDmY422sLerKALrauD9wKco/mcf60XPtmd4C7Ccoud5FHAe8KvyPVEExy1i+2eSTqa4e38/8Gng2xQ/yyMUPelHtrT+qkk6FriA4ssOit//mE/V36IYVnMh2D4DeLmkAyjmMDgFeLOkX1L0Dq9qsnH9sP2lcuTBNsBDFDf7fgkgaQ+KS/NZnsLfA04oe22LgVeW+dxZs307cLukXSnavwz4LsWd/QOBJ5U3m9o4XeFqil75W23/fOygpLOBQadaosM6f4OMYoKY36TooY1tD1Nckm8FXDyoE5VznVb1O30Z8HXgqRTB9msUXxSfpUiHDITtDwErGWCvTdK5ks6h+LI7gKL3fBCwH8UNxW8OOtBKWiZpyWzrsT0CvJ0ifbOgrPutwDrbX5ht/TF3dL5na/tRST8BLgVeZfvTknanmNrv85IG+STTDhQD3gfO9j8A/yDpucAbBzUtYdlj3nbCuc6T9GVJLxtQQDmfIm9+EEWP9lcUT5N9n+LL6SJJnxnUU3blELztKJ7e28b2r3p9Zjq2R8pZv86X9F3gPttXDKKtMXfMlXG2821vkPRM23eW85JuY/vOAZ9nG+B3bV8zyHonnEPAa2z/fc/C/dW3FfCMib8LSXvaHuhEK5J2tf1/Je1G8UDD4z0/tOXnegmwte2vDLDOhRRjgv/XoOqMuWNOBNuIiKbNhZxtRETjEmwjImowJ4OtpOWpv7v113GO1N9s/cNoTgZbigH8qb+79ddxjtTfbP1DZ64G24iIWnViNMLChQu973779V1+5IEHWLjHHn2Xv/WWW2bUHtsUI7T6Lx8x19me1Rwfy5Yt88jISO+CwNq1a6+yvWw255upTjzUsO9++3HjypWV1b/rzgsqqxvgiSc2Vlp/xFwwMjLC6tWr+yq71VZbLay4Ob+mE8E2IgJgtMVXiQm2EdEJpt0puQTbiOgI4xbPQ59gGxHdYNg0mmAbEVEpk5xtREQtkrONiKhBm4Ntq58gk3RB022IiOFgm9E+tya0umdr+4ym2xARwyM92y0k6fqm2xARw8HAJruvrQmt7tlOp5zCbTnAPvvs03BrIqIN0rOtgO0VthfbXjyTSWUioruSs42IqJrd6p5tgm1EdELmRpgF20uabkNEDI9No6NNN2FKrQ62ERH9y0Q0ERGVs6HF89Ak2EZEdyRnGxFRgwTbiIiKZYrFiIg62BmNULVbvv1tdpw3r7L6q740mcmy5xExtaQRIiIqZsjQr4iIOmToV0REDZJGiIioQYJtRETFnNEIERH1SM82IqJieaghIqImGfoVEVGDDP2KiKiYbUZzgywionrJ2U5D0o7A54CdgXXAc4AHgQ3AIuAy23/VXAsjYli0eTRCG5YyXwT8DXAM8HRgR+Ak4BDgFODgyT4kabmkNZLW1NXQiGg3lyvs9tqa0HjPFngMeG257QrMt/2IpLuAXwGTTollewWwAkBSe7/OIqIWtludRmhDz/YNwBeAVwOPNtyWiBhi7vO/JrShZ3s18HHgjRTjkvdutjkRMYwMbGrx2K/Gg63tbwC/McnxJeXuaXW2JyKG1yDysZLmAVcA+wC3Aa/zhIrLG/uXAwuBm2y/q1e9bUgjREQMxGiZt+219XAqcK/tg4HdgBdPUuY1wM22jwAOkvTsXpUm2EZEN/Q5EqGP3u9SivQmwLXACycpsxHYQcWaVvOAx3tVmmAbEZ1gZjT0a+HY0NFyWz6uqt2Bh8v99cCTJjnd5cCxwPeA79te16t9jedsIyIGZQZDv0ZsL57qPWBBub+gfD3Re4CP2/6kpM9IOtz2yulOmJ5tRHTGgHK211A8ZAVFSuG6ScrsTPGMABQphZ16VZpgGxGdMDaf7QCC7WXAIkm3AQ8B6ySdN6HMRcCbJa0C5lME6GkljdCHIgc+3Kp+RLELv6MYcgN6FNf2RuC4CYfPmlDmLuCImdSbYBsRndHmx3UTbCOiE8ZGI7RVgm1EdEZW142IqFxzk8z0I8E2IjrBLra2SrCNiM7IDbKIiBrkBllERMXGHmpoqwTbiOiGli9l3srHdSXtL2lJ0+2IiCEzdpes19aAVgZbYH9gScNtiIgh41H3tTWh1jSCpNOA5wCLgScDLwfeVL5+kGKF3dOA04EFko4GTrT9QJ3tjIjh1OKUbSM52+dT9FpPBE4CdrD9fEmvB95l+z2SbgGW2D5nqkrKyX6XT/V+RMwtRYagvdG2iTTC5bYfB+4H3g2sKo+vYpKFH6die4XtxdNMABwRc8yAlsWpRBPB9pFx+7cDh5X7h5WvATYAOwIoc/dFRF/M6KbRvrYmNH2D7MvAhnIC3pcDf1kevwU4UNJNwMlNNS4ihsdYGqGtPdtac7a2Lxm3fz1w/RTlHgdOqKVREdEZbc7Z5qGGiOiOBNuIiOq1ONYm2EZER9iN3fzqR4JtRHRClsWJiKhJgm1ERA0SbCMiqmZDQ5PM9CPBNiI6Iz3biIiKGRhNzzYiomItn/UrwTYiOqOpicH7kWAbER3R3CQz/UiwjYjOSLCNiKhY21dqSLCNiM7wpgTbiIjKpWc7jqRDAGzfWq62u9mk4hERW6TBVRj60UTP9pDyz1sbOHdEdFiCbUnSuRRLmFP2ai8BniPpBuDJFOuQ3QGsAJ4N/BR4pe1NdbYzIoZPplgcx/bZkr5X7l9SBtznA0sogvBLgWcC29o+UtJHgd8DvjixLknLgeU1NT0i2s7gAUweLmkecAWwD3Ab8DpPEsUlvQs4nmLF8JeWaydOqenVdQEuLxt5P7AdcCDwfEnXAy8A9pzsQ7ZX2F5se3FtLY2IFutvZd0+er+nAvfaPhjYDXjxxAKSng4cZPso4KvAU3tV2kSw3QDsWO6L4lthvB8An7W9BHgH8P36mhYRw6wYa9t762EpcHW5fy3wwknKvAjYTdI3gKOAf+tVaRPB9mrgJEmrmPzb4IvA3pJuBP6MPn6IiAhgJj3bhZLWjNvGpyR3Bx4u99cDT5rkVHsAD9h+AUUcO7JX22ofjWD7IYpvjonHrweuL1++ocYmRUQH2DOaiGZkmhTkCLCg3F9Qvp5oPcVVOMAPgUW9TtiGnG1ExEAMKGd7DXBMub8UuG6SMmuB55X7B1AE3Gkl2EZER5jR0dG+th4uAxZJug14CFgn6bzNzmSvAkYkrQZ+YPtbvSrN47oR0Q0DmojG9kbguAmHz5qk3JtnUm+CbUR0RyYPj4ioVvEEWdOtmFqCbUR0Rh7XjYioms3oAB7XrUqC7Rzx1nd/tNr631Nt/Ree+85K6x8dzVxHXZCebURExTLrV0REHVp+hyzBNiI6Iis1RETUwu29P5ZgGxEdYfp5FLcxCbYR0Qm5QRYRUZME24iIynkm89nWbqBTLJbrhg2inpdJ2nUQdUXEHOGBzWdbibbOZ/syIME2ImZmQIuQVaGSYCtpB0lXSLpJ0kXlsadIuk7SjZI+WB47sCyzWtJ7y2NfB44F/qekj1XRvojoHgOjo+5ra0JVPdvlwHdtHwE8RdJvUazB/j6KQHpCWe444PO2nwesA7B9DMXSwK+w/faK2hcRXVOuQdbP1oSqbpAdCBwuaQlFOmARcB/wXuBRYKey3KXAhyVdCXx5JicoV8Nc3rNgRMwRc/MJsh8A37L9KUkvBe4Bzgb+AvgOcFtZbinwYeBfgR9K+qTtJ4ANwI7TncD2CmAFgKT2/oYjojZzMdh+ArhE0huAnwOnAF8qj/8YeFTS3hRB9lJgW+BrZaAF+DvgYklbAy+wvaGidkZEh8yZYGt7Sfnno8ArJrz9uXIb7z7g0EnqWQUcNsi2RUS32eBMHh4RUb0Wd2wTbCOiK+bmDbKIiNol2EZEVM0JthERlTO0eiKaBNuI6AjjTB4eEVGxpBGiDS449x2V1l/1P/L//ud/Umn90Q0tjrUJthHRHcnZRkRULGuQRUTUITnbiIg6OEuZR0TUITnbiIiqFUnbplsxpbYu+BgRMSNjsXa26z1KmifpSknfkXSpJE1T9u2S/rmf9iXYRkRnDGgp81OBe20fDOwGvHiyQpL2A07rt221BltJ55Trkk08fsGE14dIOqS2hkXE8LMZ3TTa19bDUuDqcv9a4IVTlDsfeE+/zWtFztb2GRMOjQXaW+tuS0QMrxkM/Vooac241yvKdQ0BdgceLvfXUyxguxlJr6ZYT/GOfk/YRLA9U9IHgB8Bp9oelXT92JI6ks4FTiz3Txs7HhExnRk+1DBie/FU7wELyv0F5euJjgP2BV4CHCjpLbYvnO6ETeRsb7V9JMWS5sdPfNP22cCHgA8l0EbETAwoZ3sNcEy5vxS4bpLzvLqMY68C1vYKtNBMsF1V/rkGOGBLK5G0XNKaCZcCETFn9TkUoXewvQxYJOk24CFgnaTzZtu6JtIIzwOuosjLTjVkYgOwEECSPMlXUZlfWVGWae/guoioh8EDeIDM9kaKNMF4Z01R9i7g6H7qbaJne7ikm4A9gH+aoszVwEmSVgFH1tayiBhqo6OjfW1NqLVna/ucKY4vmfD6IYpcSUREXzLrV0REHTLrV0REHZyJaCIiapGebURE9UyCbUREpWwzOrqp6WZMKcE2IjojN8giImqQYBsRUYME24iIihWTzGTBx4iIyiXYRkTUIGmEiIgaJNhGRFQuOduIiMo5E9FERNQjwTYionLGDU0M3o8E24joDNPeYFvrsjiS9pL0pzMof32FzYmIjhnQ6rqVqHtZnJ9SLFMeETFQbb9BNm3PVtJqSU+S9HNJu0q6TtJnJN0k6XJJ25Xlrpf055K+Vr7eX9Jlkj4p6W/H1be/pEvGvX5W+dk1kl5XHnuJpFslfQ6YX8UPHRFd1F+vtqmA3CuNcAdwPLCy/PO3gDtsHwHcCby+LHcosNr2snGfPR74pO3Tp6n/L4BzgMOBsyUJOBd4CfAuYPepPihpeRmk1/T4GSJijhgd3dTX1oReaYS1wCuAK4GTgfXAqvK9VcCx5f7ttj8/4bNft31zj/qfBbyfYmHMrYFdgR1s3w8g6YGpPmh7BbCiLNfea4eIqM3QphGAbwNHA18BlgHnAYeV7x0G3F7uPzLJZyc7NtH/AU4rlzL/OPA4sFHSQkn7AHv0UUdExFjStr+tAb16trcC99q+W9KDwCeBSyTdBNzN7G92vRu4WNIuwA22H5X0XuAa4C7g57OsPyLmCNPuNcjU5m53v5JGaF7V/46KdH50me1Z/SXvtNNuPvjgJX2VXbnyC2ttL57N+WYqDzVEREc0N9KgHwm2EdEZo3lcNyKiWsW9rwTbiIiKJY0QEVGPBNuIiOq1eehXgm1EdEbSCNF5VY+D3WefZ1da/5+cW+1kdO//ozdVWj/AL37xYKX177XX0yqr+2c/+9Gs67Dd2LwH/UiwjYjOaHPPttbJwyMiqjSIKRYlzZN0paTvSLpUk1y2qfBpSTdL+qKknh3XBNuI6IwBzWd7KsWcMAcDuwEvnqTMEcA2tg8DdgGO6VVpgm1EdITBo/1t01sKXF3uXwu8cJIy9wPnl/uP99O65GwjohNsGO3/CbKFExYeWFHOkQ3FogUPl/vrgQN//Vy+E0DSicB2wFW9TphgGxGdMYMbZCPTzPo1Aiwo9xeUr3+NpBOAtwHH2+45DCJphIjoCGOP9rX1cA3/Pwe7FLhuYgFJewHvBH7P9i/6aV2CbUR0xoBukF0GLJJ0G/AQsE7SeRPK/AHwFOAqSTdKmm6tRSBphIjokEGMs7W9EThuwuGzJpQ5l2Jx2r4l2EZEJxRTLLb3oYYE24joCNPHfarGDG2wlbQcWN50OyKiPdKzrUA5Jm4FZMHHiCgk2EZEVK7dKzW0duiXpKdNMtwiImJSY2uQDWCcbSVa27O1/W9MGG4RETGdNvdsWxtsIyJmxjhLmUdEVC9rkEVE1KCpfGw/EmwjohPyBFlERC3aPfQrwTYiOmM0N8giIqrX5pyt2tzt7lce143Z2n67+ZXWv/HxDZXWD7B+Q7Xn2GV+tb8j27+2iu1MbLft9l648Kl9lf3JT3+4dpqVGiqRnm1EdILJ0K+IiFq0+Uo9wTYiOqPNOdsE24joCGc0QkRE1fJQQ0RETRJsIyIqZ2hxznbgk4dL2kPSf5rF518uacdBtiki5gb3+V8TBhpsJe0BfAz4DUl3S7qx3I6VtLOkL0i6WdKlkraVtGN5bGV5TMBa4K8TcCNipmz3tTVhYMG2DLQfAd4GjACfsH1kuX0VOAO40/ZhwLbAycBrgZW2DwdGgcXlCg3vAy5KwI2IftlmdHRTX1sTBhJsy0B7HnCm7QenKPY7wA3l/o3AYuBe4KWSnm77D2yvBrB9F0XAvTABNyL61eae7aBukB0F3G77oXHH/lDS0eX+a4CdgUfL178EdrF9paTtgH+UdAPwDtubAGzfLelB4BDgpoknlLQcWD6g9kdEB7R5NMJAera2Pw/8WNKZ4w5fbHtJuf0YWA/sVL63I/CwpP8AXAP8NrAQOHXsw5I+CHzN9q8F2vKcK2wvrnsyiYhorzb3bAeWs7V9GXCfpHdPUeRmYEm5fxTwLeD1wO+7eMbue8A8Ff4S+Lrtfx5U+yJiDiiebOi9NWDgUyxK+n3gOcDpwD3l4X8EVgCXAosoAut/Bp4MXE6RzngQOAX4I4qbZitncM72XjvEUMgUi721fYrFrbba2vPm7dBX2Q0bHql9isVK5rOVJG9hxVvy2QTbmK0E296GIdhuv31/bXzssUe7MZ/tlgba2X42Iua2NoePPK4bER2RBR8jImqR+WwjIirW9ikWBz4RTUREM4w92tc2HUnzJF0p6Tvj5myZcZmJEmwjojMGEWwpHq661/bBwG7Ai7ewzGYSbCOiMwb0BNlS4Opy/1rghVtYZjNdydmOAHfPoPzC8jNVSf3N1j/jc2zBONjW/Y5mOA62be3fbwDnvMr2wj7LzpO0ZtzrFbZXlPu7Aw+X++uBAyf5fD9lNtOJYGt7j5mUl7SmygHNqb/Z+us4R+pvtv7J2F42oKpGgAXl/gIm/9Lop8xmkkaIiNjcNcAx5f5S4LotLLOZBNuIiM1dBiySdBvwELBO0nk9ylzTq9JOpBG2wIreRVL/ENdfxzlSf7P1V8b2RuC4CYfP6qPMtCqZiCYiIjaXNEJERA0SbCMiapBgGxFRgwTbiIgaJNhGRNQgwTYiogb/Do/Bk1oq2mOIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------\n",
      "input = 我老打你\n",
      "output = i m not as you . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAD9CAYAAAB6DlaSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFbpJREFUeJzt3X+0ZWV93/H3B/kxA4wTZKbLSgVCUlFsYgJjRQEdRqGQYCxqTUQwiivTWmPUilZau9S0icsElzVqk47NinSKrsRfqCMUCT/lhwiDQoImi0WA5UhKGVgOiDqkc779Y+87HK537j13Zt9z9hneL9Zes885z93f596B73347mc/T6oKSVK/7TPpDkiSFmaylqQpYLKWpClgspakKWCylqQpYLKWpClgspakKWCylqQpYLLW2CVZnuSFk+6HNE32nXQH9OSSZDnwCWBrkuVVdeWk+yRNA5O1xibJgcDHgfdV1feS/Jck/1BVX59036S+swyisWgT9SeAD1TV9wCq6r3Ay5McP9HOSVMgLuSkcUjyKmBzVd0zx2fvrKoPj79X0vQwWWuskhxXVZuTBDgbWAZsrKqfTLhrUq9ZBtHYJPmfwG+3Lz8MnAqsBj4zsU5JU8IbjBqno6rqxCQ/C6ytqmMBklw74X5JvWey1jg9kOQ84BXAf06yAngVsGOy3ZL6zzKIxul1wCPAB6vqi8AzgGcDZ020V9IUcGStcdpOk6xfmuRMYBvwTeD+ifZKmgLOBtHYJLkQ2B+4DvgRcBBwEvBoVZ07yb5JfWey1tgkuaGqXjTH+zdWlWuFaK+SZD/gC1X18l18vgz4HPBM4Hbg9TVPQrYMonG6P8mHgGuAR4GDgbXAfZPslNS1dg2cm4BnzdPsbGBLVZ2RZBNwCvC1XTXea28wJnlBkqcv0bVfuhTXnRXjV5Y6xqx4R87x3jM6/hmeA/wA+C3gPwFvAh6g+ZdW2mtU1Y+r6heBLfM0Wwdc3p5fCZw83zWnfmSd5CU0v8G+BHwK+GpVPQw8F/gscHhHcQ4GnlpV9wHnA1cMfXYCcF9V3d1FrNZ64JIOr7eQTybZH/ggcDfN93g08C7g/3QU44Sq+iA0vwjanyVJ3gZ8tKMY0m477bTTauvWrSO13bx58x3A8JO3G6pqwyLCHUpzkx3gYZr/3nZpqpN1knXA+4Azgcdovp+vJPkPwAHApg7DHQN8LMm5bSyS/HFVvRlYCXwMOLbDeM9KcsOs9wLsB1xXVW/vMNaMk2lKE88C/qyqrun4+v8RuKw9/180Iwto5lqbrDVxW7du5eabbx6p7T777POTqlqzJ+Focgftn/P+lpjqZE0zq2CmJLEf8HXgh8BXgFuA07sKVFXfTHIp8Ec8Plr/XpJnV9Ul7cMeXbqjqv7VXB8kWYolRYtmnY6v8Pgvo/cCVNUpSxBP6qXB+CZdXEGz5MLnaQYuH5mv8bQn6ytpEst24J/R/C/7d2img/0x8DPAgx3GOxH4l8BNSV4N/Dnw68DvA/+vwzgwz/2EqjqpqyBJXgz8e+DwqvoR8JIk+1TVoP28y/sah7f/1xPgyKHzZ3YYQ9ptBSzFDLl2iYW3VNXwoO4i4JVJbgduY6i0Opdpv8H4VuC9VXU68C3g7cCOqroD+K/A67sKlOR64JdofgseApwALAeOa6foPKOrWK1r2l8IJDmm42sP+zpNKemKJN9I8gKa7+mGtpa8oosgSU6nKRV9n+amyweGzv+sixjSnquR/xnpalU/3/5596xETVVtr6ozquoXq+qc+abtwfQn658DfifJ64HDgNcCb25fr+Tx+ugeq6oTgFur6lTg21X1Dpqa+EqaG4Gdbk9VVX9E88sI4D1Jntrl9YesBu6sqrfQ/PwOqKqbgZfQzNx4TUdxbqb5Zfflqrpw5gCeTjPHVJq8gh2DGukYt2kvg1wKvL/9M+2xjab88Sjwp0CXD1vs286YmPmbOhs4tk2snUlyB03/Z24yrgBuT3IfzROAf1NVXU13WwP8XpK7aZIzSd4w0xWa7/WTexqkqrYmeQfwkSRvq6ptSX4HuKuqLt7T60tdKMZas16UqU7WVfVokr8HNgK/UVUXJjmUZqL5F5J0NeVsxltoatP7t/GvS/K7Sb481w4ou6uqngtN6aWqTmifdLq4qk5r37983gssLtYlwCVJng/8O2AVzayNmfU6ntJhrK3tjdiPJvlrmumOn+vq+lIX+vpU99Q/bt7ukP3jJP+0qu5sC/n7VtWdSxjzBVV1U3v+YuCmqtq+BHFeUVVfas9/var+vOsYc8R8PvBXS7lzS5JVwEntyntSbxx73HF17fXXj9R2xfLlm/dw6t6iTH2ylqSu/PKxx9Y1IybrlQceONZkPdVlEEnqWl8HsCZrSWoVsKOnyXrap+79lCTrjWWsScUx1vTFmq2qRjrGba9L1jQLIBnLWJOKY6zpi/UEg6qRjnGzDCJJMyY0ah5Fr5P1qlWr6ogjjljU1xx++OEcd9xxi/5p33rrrYv9EgCSjO1v1ljTEcdYE4u1tapW70nMpVobpAu9TtZHHHEEN95001hiLT9g2VjiAAwGO8YWS3oSubeLi+wYDLq4TOd6nawlabxGX6Rp3EzWktSqggms0TQSk7UkDbFmLUlTwGQtST3nEqmSNA2qnA0iSdPAMogk9VxBb6fuTWRtkCQfm0RcSVrIoEY7xm0iI+uqeuvCrSRp/PpaBpnUyPrqScSVpIX0dYnU3tWs23Vs10OzKJMkjUv1eDZI79azrqoNVbWmqtasWrVq0t2R9CTjyFqSes6HYiRpSvR16t6kZoOsnURcSVqIq+5JUs9VFYOe3mA0WUvSEGvWkjQF+vpQjMlakoaYrCWp56rKMogkTQOn7klSzxWwo6dz93r3uLkkTVIXj5snWZZkU5LbkmxMkjnaHJTkS0muT/IHC/Wr1yPrW2+9lQP2228sscZ5U2GOvzdJPdFRzfpsYEtVnZFkE3AK8LVZbV4HfKOqPpjkq0meU1Xf3dUFHVlL0owRR9UjDO7WAZe351cCJ8/RZjtwYDvqXgY8Nt8FTdaS1CoWVQZZleSWoWP90KUOBba15w8DT5sj3KeB04HvAn9TVXfN17del0EkadwWUQbZWlVrdvUZsLI9X9m+nu184E+q6n8k+UySF1XVDbsK5shakoYM2rnWCx0LuAI4tT1fB1w1R5sVwE/a8+3AwfNd0GQtSa2Z9aw7SNYXAYcluR14CLgryQWz2nwCeHOSG4HlNAl+lyyDSNKMjnaBqartwBmz3j5vVpt7gBNGvabJWpKG+Li5JPXczGyQPjJZS9KQvu5ubrKWpJ3KhZwkqe+qmqOPTNaSNMQbjECSq4G/B1YDTwG+XVXvGGcfJGk+3mB83Pk0q08dC3xl9oft8/XrZ78vSUtt5qGYPhp7sq6qe5LcV1U/nGuN16raAGwASNLPn5qkvVMVA2eDSNIUcGQtSf1XPd3Wa6zJuqrWzvWnJPVFTwfWjqwlaUYzz7qf2dpkLUlDTNaS1HvFYIezQSSp1yyDSNKUMFlL0jQwWUtS//U0V5usJWmn8gZj782xTMmSGVdNbJzfk7Q3cFsvSZoSJmtJmgIma0nquypwISdJ6j9H1pLUcwUMHFlLUs/5uLkkTQc3H5Ck3itH1pI0DUzWktRzLpE6JMmRwJFVdfW4Y0vSQmpHP5P1PhOIeSSwdgJxJWlBVTXSMW6djayTvAH4BWAN8I+AVwP/pn39IHAO8AbgXGBlkpcBZ1bVA131QZL2yIQS8Si6LoO8kGbUfCbwKuDAqnphkjcC766q85N8C1hbVe+f6wJJ1gPrO+6XJI2kr8m66zLIp6vqMeB+4D3Aje37NwLHjHKBqtpQVWuqak3HfZOkec0skdrHMkjXyfqHQ+d3AMe358e3rwF+DBwEEBdcltQnBbVjMNIxnyTLkmxKcluSjbvKdUneneTrSS5Nsv9811zKG4xfBX6c5Eaa+vUftu9/Czg6yfXAa5YwviQt0mij6hFG1mcDW6rqecAhwCmzGyQ5CnhuVZ0EXAr8k/ku2FnNuqo+NXR+NXD1Lto9BvxaV3ElqUsdVTjWAZ9vz68ETga+NqvNS4FDklxLUzr+2HwXnMTUPUnqrUWMrFcluWXoGJ4YcSiwrT1/GHjaHKFWAw9U1YtpRtUnztcvn2CUpFbVohZy2jrPRIitwMr2fGX7eraHgb9tz/8OOGy+YI6sJWlIRzXrK4BT2/N1wFVztNkMPL89/3mahL1LJmtJ2qkYDAYjHQu4CDgsye3AQ8BdSS54QqSqG4GtSW4G/raqvjnfBS2DSNKMjhZyqqrtwBmz3j5vjnZvHvWaJmtJGubmA5LUb80TjJPuxdxM1pI0pK9rg5isJ2BcT9mP8186Vw7QXqGKwQKPkk+KyVqShjiylqSem1l1r49M1pI0o8d3GE3WkrTTk2enGEmaatXP+4sma0naqRjlUfKJMFlLUssbjJI0JUzWktR7tZj1rMfKZC1JMzpadW8pmKwladiTMVknOQj4C2AFcBfw7vb1cuDWqvq3SxlfkhajgEFPyyBLvVPMYcB/p9ne5iia7W3+uqqOB65O4k41kvqj3YNxlGPcljpZ/gQ4B9gI/AzwVwBJNgHPqfrp6edJ1s/sFrzEfZOkWUbbf3ESde2lTta/BVwMnAU8CpwAfKaqzgBOTfJzs7+gqjZU1Zp5dg2WpCXT12S91DcYLwf+BPjXNOWgO4A/SHIAsAW4d4njS9KiPClng1TVtcAxs94+cSljStLuqoJy8wFJ6r+eDqxN1pL0OJdIlaSpYLKWpL7zcXNJ6r8CF3KSpP4rys0HJKnnLINI0nToaa42We/Nkowt1jhHI+P8vvTkY81aknrOPRglaRpYs5akaVAMnA0iSf1nzVqS+q4pWk+6F3NyWy1Jas3k6lGO+SRZlmRTktuSbMw8U5iSvCPJXy7UN5O1JA3paKeYs4EtVfU84BDglLkaJTkCeMMo/TJZS9KMKgY7BiMdC1hHs1MWwJXAybto91Hg/FG6Zs1akoYsYureqlkbe2+oqg3t+aHAtvb8YeDo2V+c5CzgNuA7owQzWUtSa5EPxWydZ2PvrcDK9nxl+3q2M4DDgX8BHJ3kt6vq47sKZhlEkoZ0VLO+Aji1PV8HXDVHnLOq6kTgN4DN8yVqMFlL0pARp4IsnKwvAg5LcjvwEHBXkgv2pGedlEGSvAv4QVV9Msl5wCPAWpoh/r00dzvPAqiqTyVZC6ytqvd3EV+SOlFQHTzAWFXbacocw87bRdt7gJctdM2uRtYXAa9pz18BLAO+U1UnAHcCbxz1QknWJ7llVuFeksZiMBiMdIxbJ8m6qu4D/iHJS4AtNHc+b2w/vhE4ZtaXLJ/nWhuqas08hXtJWhIzNxg7qFl3rsua9UXAhcBG4A7g+Pb949vXjwEHt++d3mFcSepG9TdZdzl172LgQ8Bl7XU/leR6mpr179M8xfPZJM/pMKYkdaj27oWckhwFfBa4oKp2ADuA185qdj/w4i7iSdKS6elCTp0k66r6O+C4Lq4lSZNU7MXJWpL2BlXFYLBj0t2Yk8lakoa4rZckTQGTtSRNAZO1JPVcM4faDXMlqfdM1pI0BSyDSNIUMFlLUu9Zs5ak3qtyZC1JU8FkLUm9V9QENhYYhclakoYUJmtJ6j3LIJLUc95glKSpMJktu0ZhspakIa5nLUlTwJG1JPVdU7SedC/mZLKWpFbhHoySNBVcG2RESdYD6yfdD0lPRs4GGVlVbQA2ACTp509N0l5r4OPmktRvzf3FfibrfcYdMMnPJrlg3HElaWHV7sO48DFuYx9ZV9XdwHnjjitJI7FmLUn959Q9SZoCzgaRpJ6rKtcGkaRp0NeR9dhng0hSn3UxGyTJsiSbktyWZGOSzNEmSS5M8o0kX04y7+DZZC1JQzqaunc2sKWqngccApwyR5sTgH2r6njgqcCp813QZC1JOxXUYLRjfuuAy9vzK4GT52hzP/DR9vyxhS5ozVqSWlUwGP0JxlVJbhl6vaFdLgPgUGBbe/4wcPRPx6o7AZKcCewPXDZfMJO1OrF69TPHFuuAAw4cW6z99182tliPPPLQWOJc8u1vjyUOwK/+8rFji9XVY+KLuMG4tarW7OozYGV7vrJ9/VOS/BrwNuDlVTXvNBTLIJK0U1E1GOlYwBU8XoNeB1w1u0GSpwPvAn61qh5Z6IIma0ka0tENxouAw5LcDjwE3DXHmki/Cfxj4LIk1yU5d74LWgaRpCFdzLOuqu3AGbPePm9Wmw8BHxr1miZrSWo1S6T286EYk7Uk7VQscJ9vYkzWkjTEkbUkTQGTtST1nhvmSlLv7TV7MCZZneRXdjdYklcnOWh3v16Sllpf92AcOVknWQ18BDgmyb3tJO7rkpyeZEWSi9ul/jYm2S/JQe17NwwtEbgZ+G8mbEn9VNRgMNIxbiMl6zZRf5jmGfatwCer6sT2uBR4K3Bnu9TffsBrgHOAG6rqRcAAWNNulvs+4BMmbEl9VCP+M24LJus2UV8AvL2qHtxFs38OXNOeXwesAbYAr0hyVFX9ZlXdDFBV99Ak7I+bsCX1TUdrg3RulBuMJwF3VNXwkmBvSvKy9vx1wArg0fb1j4CnVtWmJPsDn09yDfDOmVWlqureJA8CvwRcPxwsyXpg/W5/R5K0m/r8BOOCI+uq+gLw/SRvH3r7T6tqbXt8n2a91oPbzw4CtiV5Ns3KU8cBq2h2TgAgye8B/7uqnpCo23gbqmrNPEsPStISGe3mYm9vMFbVRcB9Sd6ziybfANa25ycB3wTeCLyymv9f+C6wrN1z7A+Br1XVX+5RzyVpCQwGg5GOcRt5nnVV/UWSVwK/AJyb5LT2o88DHwc2JrmJJjF/DrgW+HSSNwEPAq8F3gl8sapu6PB7kKTO9HWe9aIeiqmqLyT5YlV9YI6PXznr9X08PtoGIMmHq68FIUlqitaT7sWcFv0E454kWxO1pD4rmMi0vFH4uLkkDenrmNJkLUlD9oqatSTt3WoiMz1GYbKWpFafH4oxWUvSEJO1JPVegTVrSeq/vk7dS1+H/ABJHgDuXeSXraJZxnUcjDU9sfbG78lYT3REVa3ek6D77rtfrVjxtJHa/uAH/3fzONcw6vXIend+8EluGdcP0FjTE2tv/J6M1b2qYjDYMe6wI+l1spakcetrtcFkLUlDTNbjs8FYxppgHGNNX6wn6Guy7vUNRkkap6c8Zd9avuzghRsCj/5omzcYJWkSqopBeYNRknqvr9UGk7UkDTFZS1LvTWYz3FGYrCVpiOtZS1LP9XmJ1H0m3QFJ6o+iajDSMZ8ky5JsSnJbko1JsjtthpmsJWlIF8kaOBvYUlXPAw4BTtnNNjuZrCVpSFWNdCxgHXB5e34lcPJuttnJmrUkPe6yqlo1YttlSW4Zer2hqmYekz8U2NaePwwcPcfXj9JmJ5O1JLWq6rSOLrUVWNmer2TutblHabOTZRBJ6t4VwKnt+Trgqt1ss5PJWpK6dxFwWJLbgYeAu5JcsECbK+a7oKvuSdIUcGQtSVPAZC1JU8BkLUlTwGQtSVPAZC1JU8BkLUlTwGQtSVPg/wOUZN1jsH/DLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------\n",
      "input = 打那个狗\n",
      "output = i am afraid of dogs . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD9CAYAAAClQCyNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFwxJREFUeJzt3X+0XlV95/H3h/wgMWD4EdpqKrHUJSOO0kWig/zQkAqK4i9s7YBQrY7pVFFkyoy/6qodZ8bpNNZi1SnXTkcHkdEBB2eiDFJ+FgKahFmkRalMLCyCjnJhTCoKSp7P/HHOLTc3z69777nPOefJ55V1Vs55nv2cve9N1vfu+9377C3bRERE+xxUdwMiImJuEsAjIloqATwioqUSwCMiWioBPCKipRLAIyJaKgE8IqKlEsAjIloqATx6krRc0ovqbkdEdLe47gZEM0laDnwSmJS03PYNdbcpIvaVAB77kfQU4BPA79t+QNK/kfQz239Vd9si4klJocQ+yuD9SeAPbD8AYPv3gFdJOrHWxkXEPpTFrGI6Sa8Httu+r8t7v2v7o6NvVUR0kwBeEUnrgLtt/6TutlRB0lrb2yUJOA9YBlxm+7GamxYRpaRQqvNR4DV1N6IKkv4LcEF5+VHgDOAo4IraGhUR+8kgZgUkvQw4GPhC3W2pyDG2T5H0S8B62ycASLql5nZFxDQJ4PMk6VnAvwde6/HJRz0k6WKK3yg+LOlQ4PXA3nqbFRHTJQc+D5LeA7wSeBuwHpie/14ErLD9iRqaNi/lTJTzgQdsf1XSscBvAZfY/l69rYuIKemBz5GkZcBLgYfK41BAwNRPxEXA8npaN2+PA38P/Kqk1wG7gW8A36+1VRGxj/TA50nSm4CLgFfY/m7d7amCpM8CS4FbgR8DK4BTgUdtv6XOtkXEkxLAKyDp14D3ACfb/mnd7ZkvSVtsn9Tl9dttZ22UiDmStAT4ku1X9Xh/GXAl8AxgB/Cb/cbWMo2wAravpPhmX1R3WyryfUl/KOkVkl4i6ZWS/ggYi98wIupQri+0HTi9T7HzgF22jwcOH1B2PAO4pFeMqJ7nTLt8P/AfJf2ypGcvUH1PKX+CL7TzgR9SDM5+EHgrRZ7/vBHUHTGWbP/E9vOBXX2KbQCuK89vAE7rd89xHcTcCHx1BPV8RdJO4GaKb/Y/p8gZ/xNJr55aS6RCm4EflkHcQKc8lgBX2v5sRfWcbPsjAJKePpXbl3QhcElFdUQ03stf/nJPTk4OVXb79u13A9OfVJ6wPTHLKo+kmDQAsAc4tl/hcQ3gz5a0ZcZrogh0t9p+d0X1fN32OZLWA2cB77P9oKQ3AmuAqgO4bJ8t6QTgqRRBXMBfU/zUriqAfwC4tjz/HEWvAIq54AngccCYnJxk69atQ5U96KCDHrO9br5VAivL85XldU/jGsDvtv3r3d6QVOWSqM+X9HGKIArwnmLpEAScKOkNtt9VRUWSFvPkFMVNFL1xUfT6TwJOqaKeiNhXZ7QTPa6nWLriKoqO08f6FR7XAN4zt2/71ArreQfwKPs+oTgVzDsM+Ok5rPLBmpuBXyjTJwK2lX//CDjE9kNV1FU6WtL7y/s/c9r5MyqsI6LxDCzUTL1yqYp32L542suXA2dL2gHcRRHQe99jHKcRSnoX8F3bV0o6zvY3F6ie/wN8ETiXYqGnc4DPUwY+imVZN1VU1zLgXcDZFPn2hyn+fy0GtlW1Y46kM4Hj6P7DZ43tf11FPRFtsHbdWt/x9a8PVXbp4iXbK0ihzMpY9sBtf1zSzRTzKd8r6QLbexagql3ApRQjxX9BsfDT+wEkPQN4XoV1fZAibXIt8HTgZ8Dq8r3dvT40B1spfiC9y/b/m3qxXDZgR4X1RDSfYW+nuZ3csQvgku6mSGtMDWQeCuyQ9F2KpwvvsV3VdLijgd8Gnlb+vVrSFRSDpZ+yXeVMmEspHme/h+JXq9cAnwHuBy6UtMf2Bb0/Phzbk5IuAj4m6ULbu8vfaHbavnq+949oEzPyHPisjN08cNvPtf1CioHMk4AXUATtk8pfb36+inrKXPQ1ZY/7GxRPYv4N8CaK+dNvL3e3qcobgP9JsR7JJoqv64XAg7Z/E1hV1Rxx25PAxcAl5aqEk+XDShEHHNtDHXUYyxw4gKTX2P5yef4btke6VrekRcAy249WdL+DbT8u6Rhgse1vSzqMIm2zID1jSauAU23/94W4f0TTnbB2rW+57bahyh66fHly4FWZCt7l+cg3WrC9lyKVU9X9Hi///s60134ILFhao+yJJ3jHAct2o1MoYxvAIyKq0OQsRQJ4REQPBvY2OICP3SBmN5I2pq7m15O62lPPONc1U5MHMQ+IAE6xuFXqan49qas99YxzXfvolHnwQUcdkkKJiOilxt71MFoXwFetWuU1a9bM6jPPOPpo1q5dO+t/hTvv/N+z/QgA0kFz+Bef238SSSP53zWqelJXe+ppQ122NbhUn8+TQcxKrVmzhi133DGSug55yiEjqQfgiSdavxNbxFja2+nU3YSeWhfAIyJGx3iOvx2PQgJ4REQPNjR4LasE8IiIfpIDj4hoqQTwiIgWavpysgngERG92JmFEhHRVkmhRES0kKHR0wgbsxaKpD+tuw0RETN1PNxRh8b0wG2/s+42RETM1OQUSpN64DfV3YaIiJmavJxsY3rg/ZRrAW+EYmGqiIhRcMNnoTSmB96P7Qnb62yvO2rVqrqbExEHkPTAIyJaKA/yRES0WJOnETYmgNteX3cbIiJmymqEEREtZJtOgwcxE8AjIvpIDjwioqWa/CBPAnhERB8J4BERLWQ7KZSIiLbKNMKIiBYysLfB8whb8Sh9RERdqniUXtIySZsl3SXpMknqUmaFpC9Luk3Sfximba3rgd95550sW7p0JHWNcvCiy79nRDRARTnw84Bdts+StBk4HfjajDJvBO6w/RFJX5H0HNvf6nfT9MAjInoZsvc9RGdvA3BdeX4DcFqXMo8DTyl758uAnw66aQJ4REQPZlYplFWStk07Nk671ZHA7vJ8D3BEl+o+D5wJfAu4x/bOQe1rXQolImKUZpFCmbS9rtd7wMryfGV5PdP7gD+z/eeSrpB0ku0t/SpMDzwioo9OORd80DHA9cAZ5fkG4MYuZQ4FHivPHwcOGXTTBPCIiB6m1gOvIIBfDqyWtAN4BNgpadOMMp8EfkfS7cByiqDfV1IoERG9VLTbju3HgbNmvHzxjDL3ASfP5r4J4BERfeRR+oiIFpqahdJUCeAREX00eVf6BPCIiJ6cxawiItrILo6mSgCPiOgjg5glSSuAL1JMWN8JPA94GPgJsBq43PafjLJNERH9NHkQc9QP8qwGLqV4IukYYAXweuBXgHOA47t9SNLGqfUFRtXQiIgKH+RZEKNOoTwGnF8ehwHLbf9I0n3AE0DXNVVtTwATAJKa++MwIsaLTafBs1BG3QN/G3A1cC7w6IjrjoiYvamRzEFHDUbdA78O+DPgtyl+O3n6iOuPiJgVN3hLtZEGcNu3AMd1eX19efrmUbYnImKQBo9hZhphREQvRXakuRE8ATwioo8E8IiIVjKdvc2dhZIAHhHRQ1IoEREtlgAeEdFWCeAREe3U4PidAB4R0ZMziNlaUtelWVptlPm8cfz+xYElW6pFRLRYAnhEREslgEdEtJENWcwqIqKd0gOPiGghA530wCMiWiiP0kdEtFc2dIiIaCWnBx4R0VYJ4BERLdT05WTntCu9pGdK2iLpVkkvmuVnz5R0Zo/3bppLeyIiFor3eqijDnPtgb8Y+M+2Pz3bD9q+Zo51RkSMXJN74AMDuKQVwBeBQ4GdwA+A15XvvdH2eknrgVcDRwP32P698rU/BAT8O9tXl595M4Dtz0y7vhC4s7ovKyKiAm7/IOZq4FLga8C1wHuBb8GTQbj0G8B62/eW1z8H/DqwAvhj4OqZN5a0CPgA8HxgHfDhbg2QtBHYOERbIyIq1fYA/hhwfnkcBizvUe6KacEbYBHwKeDBPp85Cvie7Z9I+kavBtieACYAJDX3uxkRY6Xpy8kOM4j5Nore87nAo33K/WjG9R8ArwU+3uczk8DTJB0MnDBEWyIiRsfgvZ2hjn4kLZO0WdJdki5Tj8XyJf0rSX8l6RpJSwc1b5gAfh1FmuP64svh6UN8BmAzsBV4P3B4twK2nwA2AXcAFwx534iIESly4MMcA5wH7LJ9PEU8PH1mAUnHAM+1fSpwDfCLg246MIVi+xbguBkvb5lR5ibgphmv/Yse9/vMjOtLKXLsERGNU1EGZQNwVXl+A3AaxbjidL8KHC7pFuD7wJ8Ouumc5oFHRBwoZtEDXyVp27Rj+sSLI4Hd5fke4IguVR0FPGT7xRS971MGtS1PYkZE9GDPajGrSdvrer0HrCzPV5bXM+0B/rY8/w7FDMC+0gOPiOijohz49cAZ5fkG4MYuZbYDLyjPn0URxPtKAI+I6Ml0Op2hjgEuB1ZL2gE8AuyUtGmfmuzbgUlJW4G/td1zavWUpFAiInqpaDEr248DZ814+eIu5X5nNvdNAI+I6CcbOkREtE/xJGbdregtATwioo8mP0qfAB4R0YtNZ8Bj8nVKAI+I6CM98IiIFmr6aoQJ4BERvTR8FDMBPCKip/bvyBMRccByc8cwE8AjInoywzwmX5sE8IiIHjKIGRHRYgngERGt5NmsBz5ytS8nK2mlpBsl3Srp7LrbExHxD1zZeuALogk98OOBLbY/UHdDIiL2kxRKQdLBwGeAo4H7gbsodmteKeklwOtsPzTKNkVE9GKgkxTKP3gb8E3bJwP3Aj8E3gn8he1TErwjolHKPTGHOeow6gB+HHB7eX57eT2QpI1TOz0vWMsiIvYzXP67rhz4qAP43cCJ5fmJ5fVAtidsr+uz43NExIJIAH/SnwPPlXQb8GyKfHhERGM1OYCPdBCz3NjznBkv31QeERGNYoOzoUNERDs1eBZhAnhERG9ZTjYiorUSwCMi2sgJ4BERrWRo9GJWCeARET0ZZ0OHiIgWSgolIqK9Ghy/E8AjIvpJDjwiooWyJ2ZERFslBx4R0Vamk1koERHtlBx4REQbFUnwulvRU+270kdENNVU/B7m6EfSMkmbJd0l6TJJ6lP2Ikl/OUz7EsAjIvqoaEOH84Bdto8HDgdO71ZI0hrgzcO2LQE8IqIXm87ezlDHABuA68rzG4DTepS7BHjfsM1LDjwioo9ZTCNcNWPj9QnbE+X5kcDu8nwPcOzMD0s6F7gL+OawFSaAR0T0MMsHeSb7bLw+Cawsz1eW1zOdBRwNvAw4VtIFtj/Rr8LKUyiSPiRpfdX3jYioQ0U58OuBM8rzDcCNXeo51/YpwD8Ftg8K3pAceEREH0NOQRkcwC8HVkvaATwC7JS0ab6tqySFIukw4L8BS4G9wF9L+gpwBPB12++W9PPAVcDBwN3AbcDVwBeB5cCdtt9eRXsiIiphcAUPYtp+nCJFMt3FPcreB7x0mPtW1QPfCHzV9kuADsUo6n+1/SLgcEkvA04CrgFeCxxp+9PAi4G/sX0icJOkru2RtFHSthkDBBERC67T6Qx11KGqAH4MxegpwFbgOOD28vr28non8HrgCxRTZaAI6EjaDDzH7v6zzvaE7XV9BggiIio3NYhZQQ58QVQVwO8H/nF5fgJFiuTE8vrE8vq1wFttn2J76imjk4ErbJ8FnCHplytqT0TE/LnZAbyqaYSfBq6U9Iby+iPAWyS9gyIH/jVJS4AvS/oOsAt4D3Av8DlJB5ev3V9ReyIiKuDxX8zK9iSwfsbLX5px/QLg74CfUcyDPMz23cApVbQhImJBNHgxq5E9yGP7Q8CHRlVfREQVTAJ4RETr2KbT2Vt3M3pKAI+I6CNbqkVEtFQCeERESyWAR0S0UDHHO5saR0S0UgJ4RERLJYVSocWLl3LEEU8bSV0PPfTASOoBeNazThhJPYsWje6f/KIPXjK4UEW+8oXPjayub39768jqgp5731ZsdEFqVP8H9+59opL7JIBHRLRScuAREa1kpwceEdFaCeAREa1kXNNmDcNIAI+I6MMkgEdEtFJSKBERLZRBzIiI1qpvu7RhJIBHRPSR9cAjIloqPfCIiDYqkuB1t6KnBPCIiB5M9sSMiGitrIUyT5I2AhsBDjpoUc2tiYgDR2ahzJvtCWACYMmSg5v73YyIsdPJo/QREe1TjGE2N4AfVHcDJP2SpE11tyMiYn8u98UcfNSh9h647b8DLq67HRERXSUHHhHRTplGGBHRUpmFEhHRQrazFkpERFs1uQde+yyUiIgmq2IWiqRlkjZLukvSZZLUpYwkfVbSHZL+h6SBHewE8IiIPiqaRngesMv28cDhwOldypwMLLZ9IvBU4IxBN00Aj4joyeDOcEd/G4DryvMbgNO6lPk+cEl5/tNhWpcceEREDzZ0hn8Sc5WkbdOuJ8plQACOBHaX53uAY/evy/cCSHodsBS4dlCFrQvgTzzxU37wg/vrbkbl7r132+BCLfOxD19YdxPGwGgG0PaOcL2PRQe16xf/WQxiTtpe1+s9YGV5vrK83o+kVwMXAq+yPXD6S7u+kxERI2XszlDHANfzZE57A3DjzAKSfgH4l8Arbf/9MK1LAI+I6KOiQczLgdWSdgCPADu7rAH1JuBpwLWSbpX0lkE3VZPnOHYjqV0NjmiBcU2h2N5vut5sLF9+iJ/5zOcNVfaee+7Y3ieFsiBalwOPiBiVYjnZ5vYZE8AjInoyQ4wl1iYBPCKij/TAIyJaKgE8IqKVsqlxREQrjfWemJKOkvSKeXz+1yStmE8bIiIWUpP3xJxzAJd0FPAx4DhJ95cTz2+VdKakQyVdXS6LeJmkJZJWlK9tmbac4nbgUwniEdFMxp3OUEcd5hTAy+D9UYpn9ieBT9s+pTyuAd4J3Fsui7gEeANwPrDF9klAB1hXbmj8+8AnE8Qjook85J86zDqAl8F7E/Bu2w/3KPZC4Oby/FZgHbALeI2kY2y/yfZWANv3UQTxTySIR0TTVLQWyoKYyyDmqcDdth+Z9tpbJb20PH8jcCjwaHn9Y+CptjdLWgpcJelm4HenVtuyfb+kh4FfAW6bWaGkjcDGObQ1ImLOmv4k5qx74La/BDwo6d3TXv5PtteXx4MU690eUr63Atgt6R9RrMi1FlhFsUMFAJL+LfC/bO8XvMs6J2yvG/U6AxFxoBtuALNVg5i2Lwe+K+m9PYrcAawvz08FvgH8FnC2i981vgUsK/eA+yPga7b/ci5tiYhYSJ1OZ6ijDvNajVDS2cDzgLcAD5QvXwVMAJcBqymC9T8Dfg74PEXa5mHgHODtFAObW2ZRZ3N/n4loqaxG2N3Spcu8atUvDlX2e9/bOfLVCOe9nKwkeY43mctnE8AjqpcA3t3SJQcPH8D/73fat5zsXIP3fD8bEbHQDLVNERxGHqWPiOijyf3MBPCIiD6avBZKAnhERE+ubYbJMBLAIyJ6aPqDPAngERF9JIBHRLSSITnwiIh2yjTCak0C98/yM6vKz43CONY1jl/TuNY1p3rm+HBN079/a6qoOCmUCtk+arafkbRtVE9IjWNd4/g1jWtd4/g1jbqu6WzT6ewddbVDa10Aj4gYpfTAIyJaKgG8fhOpqxX1pK721DPOde2jyQF83qsRRkSMq0WLFnv5skMGFwQe/fHu9q1GGBExrmzTcQYxIyJaqclZigTwiIg+EsAjIlqpvg2Lh5EAHhHRR9YDj4hooaYvJzu63UUjIlrH2J2hjn4kLZO0WdJdki6TtN9my8OUmSkBPCKijyoCOHAesMv28cDhwOlzLLOPBPCIiD5sD3UMsAG4rjy/AThtjmX2kRx4RERv19peNWTZZZK2TbuesD21BMCRwO7yfA9wbJfPD1NmHwngERE92H55RbeaBFaW5yvpvrb5MGX2kRRKRMTCux44ozzfANw4xzL7SACPiFh4lwOrJe0AHgF2Sto0oMz1g26a1QgjIloqPfCIiJZKAI+IaKkE8IiIlkoAj4hoqQTwiIiWSgCPiGipBPCIiJb6/wDdddymj+l1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------\n",
      "input = 男人，就要勇敢一点\n",
      "output = she s not the same hat . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAEKCAYAAADU7nSHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGZNJREFUeJzt3X+UXXV97vH3kxBICBiBhApRjGgLyKJETZWfGqKkgCDFUEUERaxZWqXWigotaym1XZaKbanQ681FwYug97a2CrQUEAiIwoWEAipIvRFQQLkZUsIPI4Gc5/6x94TjOJk5M7PPOXufPC/WXmefPfvsz3eS8JnvfH/KNhERUW/T+l2AiIgYX5J1REQDJFlHRDRAknVERAMkWUdENECSdUREAyRZR0Q0QJJ1REQDJFnHQJM0S9KB/S5HxFRt0+8CRHSLpFnABcCQpFm2r+93mSImK8k6BpKk7YHzgU/a/qmkv5D0rO1v97tsEZORZpAYOGWivgA42/ZPAWyfBRwj6YC+Fi5ikpSFnGLQSFoGrLb9wChf+6jtz/W+VBFTM9A1a0kvlDS33+WYKEkvH+Xa+9rOZ/e2RL1R1d+V7a/bfkDSa8rnStLJ5Z/hBVXEiOi1gU7WwGeAj/S7EBMhaRrwxfL845JOl7QP8PvlNQGXdyn2iVu4/jsVxriofJ0laTtJsyUdIOlW4NQK4/xP4EPl288BS4F5wFerihHRSwPbwVgmnoOBg/pdlomw3ZK0sXz7NuAi4LeBZ8prBwM3dSn8pyTNBH4M3GH7CUlHAScD76goxovL17uAh4H9gZcDR9r+r4piAOxp+xBJLwMW2341gKRu/dlFdNXAJWtJM4BPAQcAS2w/1d8STcqLJL0N2BbYGRgC5kn6EkXSPreqQGVNfSfb64CngQeABcCRkvYCdqOolVZluJPkYduHSboBEPBeSU/a/kJFcdZKOh04Fvi0pB2BZcCmip4f0VONbwaRtEzSUklHSPoT4H8B99h+o+2hfpdvkgTMKF+nl6//D/g8cKztNRXGehlwlaTLgJnArsArKGrAdwA/A/arIpCk7YoXTeP5pA3FD4RHgMuqiFN6J/Ak8Bnb/wLsDuwNjNrUE1F3ja5Zl7XoV1HUNl8ALASeAB7vZ7kqsIki0WwCNgB/AOwBrKWY4HG87X+qIpDtHwOvk/QS4GLgbOCvbJ9VtpX/PXAVUMUswJXAc8CNwAvbrt8HbAe8BPhBBXGg+DfxJPBGSccB64HbgEcren5ETzU6Wdt+Fjir/ZqkecCZkt4PnGR7fV8KNzUzKDrDZgBzKJLox4FvAGdQ1BorSdYAkt5Ckcj2BM4B9pP0LuAY4PYy7pTZ3pzwJQ3PJjQwF/hD4JtUl6y/RNGMdDPwC2A2RTPIEVTYkRnRK41O1qOxvRb4k7LN9wZJh9p+ut/lmqCHbX9R0gcpatPPAg8CF1J0Lp5Wcbxzge9R1OT3p+j8+xjwFWA+cF3F8QBeXLZXLwS2Bz5h+7EKn/+btkd2Lp8v6ZYKY0T0TOPbrLfE9v8GrgXO7HdZJqJsz92ufPsVimaDO4Hptr9ZdphukLRThWHvsb2MooPxJxSJ+yngGmCT7W4MFXwVcCRF08dewE2Sqhxm+aikcyQdJekNkt4s6bMUbeMRjTPQMxgl7UDxPT7Z77JMhKSFtu+UtDOA7XWSPgT8N4ofPufbrqRdvmz3P9v2n0p6CPiXEbfsa3tJFbFGiXvscNu7pG2Al1bVeVr+3Z8GvBbYkaIv41bg87Y3VBEjopcan6wlHc7zY5CnAz8ETqDowHoYWGn7P/tUvEmRdCRwCkVn31m2TyyvzwDeB2xvu7Lhe+WzpwMftP33I66favtLVcYqnzsT+L7tV1T97PL5v2v76vJ8d9uPlOcftn1eN2JGdNMgtFn/A/AF4P0UnW57UHSM/VF5fhHFRJImWUuRlHcDTiiGQm82E/ibKoNJOsH21yTtJunPy8vPUvz5zZc0zfaFVca0/UtJv6zymSP8GXB1ef4VYPi3g2VAknU0ziC0Wf+0XJjnIYrEDIDt79v+N6odu9t1kvamSMaXUjR7DFH8Gv8s8G/ACbZvrjjsxZJ2pWhD3p7ih93/oZhZ+HvASRXHG9bq0nMjBs4g1KznSXorxVC3pRQdcpvZbtTCPbZ/CLxe0qvKSwttX1QucnQ8cH35q/zqCsPeRzEkcB1wL/Bq29dIOsP2RkmXSJpuu0mz//aQ9KcUE4oWtJ2/pL/FipicQahZb0NR85wO7AC8HVgk6aYyiTfVecB/As9KuoBiCvibgHdRfcL5CUUH3IuBN1IMqzsSeLmkeba/2KREXZb98xR9Fg9RtP0Pn180xkcjamsQkvXPbH8Z+DnwzxQTSFYBRwHvklT5SIZuk3Q08Fw5PvwfKaZ77w9cAbwO+H4Xwv4WxWzJteXrEEUC/7qk+V2I1023U4zfvtz2l4cP4EXA3f0tWsTkDEKy3q2cbfci4Ljhi+V45A9SzPxrDBX7BraPkJhJMVnlZxSjXjYAKyoOa+A9FM0gdwGP2b6dImG/l6KzrlLlAlJd+fdXrgnzEeBvJc0p4/0RsMZ2JbMxI3ptEJL12RSTOc6iWNrzJuD/liMYHgaelLRtNwJLeomkb0l6d1XPtL3B9t8B25QLH72bYtr0tymmmc8Cjq4qXpk0Z9h+hmK6+TKK5o+zKdp411CMCqna9hTNVl1RJuzTgfPK1feGqlpPJaIfGj/OejySZpRriHTj2TtRLHB/ETDL9o8qfPbbKX6N3zDi+msq7lxE0m/bvlvSgRSjToZrvXsDlwB/TrG4U2XLzZaTYN5guxtT2dvjzAUOLVfei2isgU/WERGDYBCaQSIiBt7AJWtJyxMrsfoVJ7GaF6spBi5ZA738S06s5sQaxO8psbYigzCDMSKiEkcccYSHhjrbDXD16tVX2z6iy0XarNbJeu7cuV6wYMGEPrPHHnuwaNGiCfearl49uQEWknrWQ5tYzYiTWH2LNWR73lRiDg0Ncfvtt3d077Rp0+ZOJdZE1TpZL1iwgFWrVvUk1oiV7SKieR6s4iGtmo6Qq3WyjojoJQN1Hc6cZB0RsZkxSdYREfVm2NRKso6IqDWTNuuIiEZIm3VERAMkWUdE1JztNINERDRBatYRETVnYNPWlqwlnQJg++JuxYiIqFpq1hERDTDwbdaSZlLsxL0L8Ajw78A+km4EdgWOB+6h2Ox1H4rdyN9ue1NVZYiImBK7tjXrKtez3hew7YOACyk2Qz0QOBz4FHBsecywfQjwE+DNIx8iabmkVZJWrV27tsLiRUSMbXhtkE6OXqsyWd8BfE/SFcCRFDuOX2Z7I/AoxQ7dewEHSloJvB74jZEPsb3C9iLbi+bNm9JqhxERE7ap1ero6LUqk/VC4FbbxwBzgRYwcjfs+4Cv2V4MfBT4YYXxIyKmyB3/12tVJuv7gdMk3QbsTtEMMtLlwO6SbgY+XX4mIqIWbGh1ePRaZR2Mth8Hlm7hayuBleXb91UVMyKianXtYMzQvYiINknWERE1lyVSIyKawO7LSI9OJFlHRLRJM0hERM0ZsgdjREQT1HQLxiTriIh2aQaJiGiAJOtJWL16NZL6XYzoQC//geffRHSLMxokIqIZUrOOiKi5TIqJiGiIDN2LiGiADN2LiKg527TSwRgRUX9ps46IaICMBomIaIAk64iImrOdZpCIiCbI0L2IiJozsKmmY/eq3N08IqLxbHd0jEXSTElXSrpL0iUaZUEbSbMlfVPSdyT99Xjl6nmyLr+JKyR9V9I/SUrtPiJqo1W2W493jOMk4CHb+wM7AYePcs87gVttHwzsK2mfsR7Yj5r1voBtHwRcCOzQ/kVJyyWtkrSqD2WLiK1Zh7XqDkaMLAGuLc+vBw4b5Z5ngO3LWvdMYONYD+xHsr4D+J6kK4Ajgafbv2h7he1Fthf1oWwRsRUzE2oGmTtcsSyP5W2P2gVYX54/Aew8SrjLKHLgvcAPba8Zq2z9SNYLKar+xwBzgdf3oQwREaOaQDPI0HDFsjxWtD1mCJhTns8p3490JvAF23sDO0s6aKxy9SNZ3w+cJuk2YHcgzR0RURsVtVlfBywtz5cAN4xyz47AL8vzZxjRJDxSz5O17cdtL7X9WtuH2V4//qciIrpveD3rCpL1pcB8SXcD64A1ks4dcc8FwAck3QLMokjwW5SRGBERwzrrPOzgMX4GOHrE5dNH3PMAcHCnz0yyjohok+nmERE1NzwapI6SrCMi2mR384iI2nMWcoqIqDu7OOooyToiok06GCMiGiAdjLHZNtts25M4T/3iqZ7EARhlBciIxhmeFFNHSdYREcNsWhkNEhHRAKlZR0TUn2u6rVeSdUREm5pWrJOsIyKGFeOs65mtk6wjItokWUdE1J5pbcpokIiIWkszSEREQyRZR0Q0QU2Tdc/3YJS0QNLiXseNiOjE8Mp74x291o/dzRcAi/sQNyJibC46GDs5eq2yZhBJpwD7AYuAXYHjgfeX7x8DTgZOAU4F5kh6E3Cc7bUjnrMcWF5VuSIiOrU1bet1IEWt+ThgGbC97QMlvQf4uO0zJf0HsNj2p0Z7gO0VwAoASfX8U4uIgVXXZF11M8hltjcCjwJnALeU128BXllxrIiIytnu6Oi1qpN1+wLKPwAOKM8PKN8DbABmAyiLIEdEndjQ6vDosW52MP4rsEHSLRTt158tr/8HsJek7wBv62L8iIgJq2vNurI2a9sXt52vBFZu4b6NwFuqihsRURUDrSyRGhFRc5luHhHRDNl8ICKi9vrTHt2JJOuIiDZJ1hERNZclUiMiGsKbkqwjImovNeuIiLrr04SXTiRZ98Fzz23sSZztZszoSZyIQZJkHRFRc3VeIrUfmw9ERNSTwZtaHR1jkTRT0pWS7pJ0yZYWrZP0cUnflnSVpG3HemaSdUTEZp0t4tRB7fsk4CHb+wM7AYePvEHSnsC+tg8FrgJePNYDk6wjItpUtAfjEuDa8vx64LBR7nkjsJOkm4BDgfvHemCSdUREmwnUrOdKWtV2tG9HuAuwvjx/Ath5lFDzgLW2X09Rqz5krHKlgzEiomRPaCGnIduLtvQ1YE55Pqd8P9ITwH3l+Y+B+WMFS806IqJNRW3W1wFLy/MlwA2j3LMa+J3y/BUUCXuLkqwjIjYzrVaro2MclwLzJd0NrAPWSDr3VyLZtwBDkm4H7rN921gPTDNIRMSwihZysv0McPSIy6ePct8HOn1mZcla0sIy+J2STinPL67q+RERPbEVbD6wsHy9s8JnRkT0TDGDsd+lGF0lyVrSOcBx5fkpwMXAfpJuBHal2N38HmAFsA/wc+DttjdVET8ioip1nW5eSbK2/QlJ95bnF5cJ+0BgMUUSPxb4TWCG7UMk/Q3wZuDykc8qxyouH3k9IqLrbFrjTCXvl252MF5me6OkRylq03sBB0paCewA3Dvah2yvoKiBI6meP+IiYmANdM26tAGYW54LeGrE1+8Dvmb7k5LeANTzx1dEbLW2llX3rgWWSbqF0RckuRzYXdLNwKcZZx58RETPDfcwVrA4SNUqq1nbXkcxU2fk9ZXAyvLt+6qKFxFRvewUExHRCK5pA22SdUTEMNPJVPK+SLKOiCjVuYMxyToiok2SdURE7Xki61n3VJJ1RMSwilbd64Yk64iIdknWERH1ZqCVZpDoNUk9i/Xcpt4toDhjmxk9i+W6DrqN7pjYHow9lWQdEbFZZjBGRDRCknVERAMkWUdE1JwN3go3H4iIaJyaVqyTrCMinpcOxoiIRkiyjoiou0w3j4ioP5NJMRERDWCczQciImouzSAREc1Q01w9uWQtaSbwj8AuwCPAe4HLgB2BNbbfI2kV8BiwAZgPXAp8FfgysBPwDdufmfJ3EBFRobq2WU+b5Of2BWz7IOBCYC/gvwNLgT0l/QYwG1gGLATeAewPnAl8zfbrgGMl7TLywZKWS1pVJvuIiJ4Z3oOxk6PXJpus7wC+J+kK4EjgUeBk4BLghcAs4FHbTwEPAM8BokjqH5C0EtgB2H3kg22vsL3I9qJJli0iYnI8eMl6IXCr7WOAucBXgG8AJwJPj/G5+4AzbC8GzgX+a5LxIyK6wLRarY6OXptssr4fOE3SbRS147OAPwOuo/hN4tdqzKW/Aj4m6VbgTcDPJxk/IqIr3HJHR69NqoPR9uMU7dPtXjni/eLy3sXl+1PK16MmEzMiouuKRut+l2JUk61ZR0QMnOFc3ckxFkkzJV0p6S5Jl2iMPfYkfUTSt8YrW5J1RESbijoYTwIesr0/xVDlw0e7SdJLeb7VYUxJ1hERw2xam1odHeNYAlxbnl8PHLaF+86jGNI8rsxgjIhoM4FheXNHzAdZYXtFeb4LsL48f4Ji2PKvkHQicBdwTyfBkqwjIkrDk2I6NDTGfJAhYE55Pqd8P9LRwB7A7wJ7SfqQ7fO3FCzNIBERbSpqs76O50fMLQFuGCXOibYPAU4AVo+VqCHJOiKiTYdDQcZP1pcC8yXdDawD1kg6dyolSzNIRMQwgyuYnGj7GYpmjnanb+HeBygmCY4pyToqMX1a735JcxX/N0VsQT+mknciyToiojTBDsaeSrKOiBiWnWIiIpqgP4s0dSLJOiKiXWrWERH1Z5KsIyJqzTat1qZ+F2NUSdYREW3SwRgR0QBJ1hERDZBkHRFRc8UiTQM+g1HSKQC2Lx7nvoXlfXdWFTsioioDn6wnYGH5mmQdEbWztTSD7CfpRmBX4HiK7WoWUCwRuAz4C+A4KGribTufR0TUwtaSrA8EFlMk5GXANcBXgYuAV9v+hKR7YcvNJZKWA8srLldERAe2gjbr0mW2N0p6lCJx/xbwe8A8YFYnDyj3MFsBIKmeP+IiYiC5xgs5Vb0I8VNt58uAe8vXn7Vd3wDMBpCkiuNHRExJRdt6Va6bK8bfDLyjfH0hsHt5/VpgmaRbgEO6GD8iYoKMW62Ojl6rrBmkvQ3a9kpg5RbuW0exgWRERO2YraPNOiKi0eraZp1kHRFRqnMHY5J1RMRm/ek87ESSdUREm6xnHRHRAKlZR0TUXdFo3e9SjCrJOiKiZLIHY0REI2wta4NERDRYRoNERDRCqw9TyTuRZB0RUSr6F5OsIyJqLs0gERHNkGQdEVF/GboXEdEAaQaJiKg521kbJCKiCepas+7mtl4REY1TxR6MkmZKulLSXZIuGW2/WRW+LOlWSZdLGrPynGQdEdGmog1zTwIesr0/sBNw+Cj3HAxsY/sA4AXA0rEemGQdEbGZwa3OjrEtodgcHOB64LBR7nkUOK883zjeA9NmHRFRsqHV+QzGuZJWtb1fYXtFeb4LsL48fwLY69dj+UcAko4DtgWuHitYknVERJsJdDAO2V60pa8Bc8rzOeX7XyPpLcCHgWNsjzkMpXbNIJKWS1o14idWREQPGLvV0TGO63i+DXoJcMPIGyS9CPgY8GbbT473wNola9srbC8a4ydWRETXVNTBeCkwX9LdwDpgjaRzR9zzbmA34GpJN0s6dawHphkkIqJNFeOsbT8DHD3i8ukj7jkHOKfTZ/a8Zi3pZaP8hImI6LtiidRKataV63nN2vb9jPgJExFRD2acfr6+STNIRESbuk43T7KOiGiTZB0RUXvZKSYiovayB2NEREOkZh0RUXvGrdSsIyJqL3swRkQ0QNqsY6CNshHGgOjl99WbGt2mHv6aP31a7ZYfGtPwDMY6SrKOiNgsQ/ciIhqhlQ7GiIj6S5t1RETdFY3W/S7FqJKsIyJKJkP3IiIaIR2MERENkDbriIjac0aDRETUXZ0nxUxoepGkeZKOmmwwScdLmj3Zz0dEdFtd92DsOFlLmgf8LfBKSQ+WW6ffLOlISTtK+oakWyVdImmGpNnlte+W1wSsBv4hCTsi6sngVmdHj3WUrMtE/Tngw8AQ8D9sH1IeVwGnAT+yfQAwA3gbcDLwXdsHAS1gUblZ7ieBC5KwI6KO3OF/vTZusi4T9bnAH9t+bAu3vRa4sTy/GVgEPAQcK2lP2++2fTuA7QcoEvb5oyVsScslrZK0asLfTUTEFNW1GaSTDsZDgR/YXtd27b2S3lSevxPYEXi6fP8L4AW2r5S0LfB1STcCH3W5x7vtByU9BiwEvtMezPYKYAWApHq29EfEQLJNq7Wp38UY1bg1a9v/DDws6Y/bLn/R9uLyeBh4Atih/NpsYL2kvYHrgNcAc4GThj8s6S+Bf7f9K4k6IqLf6lqz7qjN2valwCOSztjCLbcCi8vzQ4HbgPcAb3UxwvxeYKYKnwWusf2tKZU8IqIL6pqsNZGgkt4K7AecCvy0vPx1imaLS4D5FIn5D4BdgcsomloeA94B/CFFp+N3O4yXZpDos2w+MBU93nxgte1FU3nA9OnbeNasHTu69+mnH59yvImYULIGkCRP8sfKRD+bZB39l2Q9FY1M1jN3GP9G4OlfrO9psp7wDMbJJuqpfjYiotts03I9Oxgz3Twiok1d65RJ1hERbZKsIyJqLxvmRkQ0QtazjoiouYFZIjUiYrAZu9XRMRZJMyVdKemutlVHJ3xPuyTriIg2VSRriuU1HrK9P7ATcPgk79ms7s0gQ8CDE/zM3PJzvZBYzYk1yTiT+pW41n9+U5ioUuvvC3hpFYEragZZQjG7G+B64DDgmkncs1mtk7XteRP9jKRVvZpVlFjNiTWI31NidcXVtud2eO/MEUs5ryhXDQXYBVhfnj8B7DXK5zu5Z7NaJ+uIiF6yfURFjxoC5pTncxj9t4RO7tksbdYREdW7Dlhani8BbpjkPZsNYrJeMf4tibUVxhrE7ymx6utSYL6ku4F1wBpJ545zz3VjPXDCq+5FRETvDWLNOiJi4CRZR0Q0QJJ1REQDJFlHRDRAknVERAMkWUdENECSdUREA/x/OFIgJ3G/zYIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# 绘制图形\n",
    "def showAttention(input_sentence, output_words, attentions):\n",
    "    # 设置带 colorbar 的图形\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    cax = ax.matshow(attentions.numpy(), cmap='bone')\n",
    "    fig.colorbar(cax)\n",
    "\n",
    "    # 设置坐标轴标记\n",
    "    ax.set_xticklabels([''] + list(jieba.cut(input_sentence)) + ['<EOS>'], rotation=90)\n",
    "    ax.set_yticklabels([''] + output_words)\n",
    "\n",
    "    # 设置坐标轴标记位置\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "# Attention可视化\n",
    "def evaluateAndShowAttention(input_sentence):\n",
    "    output_words, attentions = evaluate(\n",
    "        encoder1, attn_decoder1, input_sentence)\n",
    "    print('input =', input_sentence)\n",
    "    print('output =', ' '.join(output_words))\n",
    "    showAttention(input_sentence, output_words, attentions)\n",
    "    print('----------------------------------------------')\n",
    "\n",
    "# 输出四个句子进行观察分析Attention的可视化过程\n",
    "evaluateAndShowAttention(pair_visual[0]) \n",
    "\n",
    "evaluateAndShowAttention(u'我老打你')\n",
    "\n",
    "evaluateAndShowAttention(u'打那个狗')\n",
    "\n",
    "evaluateAndShowAttention(u'男人，就要勇敢一点')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "观察结果可以发现，由于我们的语料有限，得到的翻译器对非语料中的句子的翻译效果并不是很好。后期我们可以尝试丰富我们的语料库，使得翻译更为准确。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"6\"></div>\n",
    "\n",
    "# 6 更多的尝试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于为了快速展示模型效果，上面的训练过程一切从简，比如限制了输入和输出句子的单词数，其实我们还可以有更多的尝试：\n",
    "1. 不对翻译对过滤，使用完整的数据集（可以通过增大`MAX_LENGTH`实现）\n",
    "2. 使用不同的数据集\n",
    "    + 另一种语言对\n",
    "    + 人→机器（例如IOT命令）\n",
    "    + 聊天→响应\n",
    "    + 问题→回答\n",
    "3. 用预先训练的词嵌入替换独热的方式，如word2vec或GloVe\n",
    "4. 尝试更多的隐层，更多的隐藏单位和更多的句子。比较训练时间和结果。\n",
    "5. 如果你使用的翻译文件对有两个相同的短语（I am test \\ I am test），那么你可以使用这个网络作为一个自动编码器。 尝试一下过程：\n",
    " + 训练一个自动编码器\n",
    " + 只保存编码器网络\n",
    " + 训练一个新的解码器对编码器输出进行翻译"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}