16_nlp_with_rnns_and_attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uuxf62kbtv8P"
   },
   "source": [
    "**Chapter 16 – Natural Language Processing with RNNs and Attention**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dtjXopCdtv8R"
   },
   "source": [
    "_This notebook contains all the sample code in chapter 16._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "As36LGNPtv8R"
   },
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/handson-ml2/blob/master/16_nlp_with_rnns_and_attention.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sy44ghc0tv8S"
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a0lV3k33tv8S"
   },
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "caLeuj3ntv8S",
    "outputId": "3eb920aa-e154-48ee-b4b9-a44e0a8dfb6a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 1.1 MB 3.2 MB/s \n",
      "\u001b[K     |████████████████████████████████| 2.6 MB 4.0 MB/s \n",
      "\u001b[K     |████████████████████████████████| 895 kB 71.1 MB/s \n",
      "\u001b[K     |████████████████████████████████| 3.3 MB 20.5 MB/s \n",
      "\u001b[K     |████████████████████████████████| 636 kB 67.3 MB/s \n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    %pip install -q -U tensorflow-addons\n",
    "    %pip install -q -U transformers\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"nlp\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BgY8rP0htv8U"
   },
   "source": [
    "# Char-RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qnqmHS-dtv8U"
   },
   "source": [
    "## Splitting a sequence into batches of shuffled windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z-ONrcMgtv8V"
   },
   "source": [
    "For example, let's split the sequence 0 to 14 into windows of length 5, each shifted by 2 (e.g.,`[0, 1, 2, 3, 4]`, `[2, 3, 4, 5, 6]`, etc.), then shuffle them, and split them into inputs (the first 4 steps) and targets (the last 4 steps) (e.g., `[2, 3, 4, 5, 6]` would be split into `[[2, 3, 4, 5], [3, 4, 5, 6]]`), then create batches of 3 such input/target pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y6xeL2EAtv8V",
    "outputId": "cfab52da-9116-4775-e603-8b5fd0a6979a",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________ Batch 0 \n",
      "X_batch\n",
      "[[6 7 8 9]\n",
      " [2 3 4 5]\n",
      " [4 5 6 7]]\n",
      "===== \n",
      "Y_batch\n",
      "[[ 7  8  9 10]\n",
      " [ 3  4  5  6]\n",
      " [ 5  6  7  8]]\n",
      "____________________ Batch 1 \n",
      "X_batch\n",
      "[[ 0  1  2  3]\n",
      " [ 8  9 10 11]\n",
      " [10 11 12 13]]\n",
      "===== \n",
      "Y_batch\n",
      "[[ 1  2  3  4]\n",
      " [ 9 10 11 12]\n",
      " [11 12 13 14]]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_steps = 5\n",
    "dataset = tf.data.Dataset.from_tensor_slices(tf.range(15))\n",
    "dataset = dataset.window(n_steps, shift=2, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(n_steps))\n",
    "dataset = dataset.shuffle(10).map(lambda window: (window[:-1], window[1:]))\n",
    "dataset = dataset.batch(3).prefetch(1)\n",
    "for index, (X_batch, Y_batch) in enumerate(dataset):\n",
    "    print(\"_\" * 20, \"Batch\", index, \"\\nX_batch\")\n",
    "    print(X_batch.numpy())\n",
    "    print(\"=\" * 5, \"\\nY_batch\")\n",
    "    print(Y_batch.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EbFRieqqtv8W"
   },
   "source": [
    "## Loading the Data and Preparing the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KM9qs1gltv8W",
    "outputId": "cbb2edcb-2488-41d4-fbcd-4c6cbd173151"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
      "1122304/1115394 [==============================] - 0s 0us/step\n",
      "1130496/1115394 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "shakespeare_url = \"https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\"\n",
    "filepath = keras.utils.get_file(\"shakespeare.txt\", shakespeare_url)\n",
    "with open(filepath) as f:\n",
    "    shakespeare_text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TFN1oyNItv8W",
    "outputId": "cf2c3c6a-a9cf-422b-e370-72de5ad40c2a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(shakespeare_text[:148])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "aXtmJYzutv8X",
    "outputId": "252ea764-3b90-4606-8bb2-d96f1c5635b7"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\"\\n !$&',-.3:;?abcdefghijklmnopqrstuvwxyz\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\".join(sorted(set(shakespeare_text.lower())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "vR-d82zktv8X"
   },
   "outputs": [],
   "source": [
    "tokenizer = keras.preprocessing.text.Tokenizer(char_level=True)\n",
    "tokenizer.fit_on_texts(shakespeare_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z7Ew2ZqItv8X",
    "outputId": "b67db908-5b64-4752-f15b-8be09ac112ba"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[20, 6, 9, 8, 3]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.texts_to_sequences([\"First\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i2OqPgaetv8Y",
    "outputId": "3eb24a6e-72f4-4b5d-9195-8b5f260dfd23"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['f i r s t']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.sequences_to_texts([[20, 6, 9, 8, 3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "aZbH0t5Utv8Y"
   },
   "outputs": [],
   "source": [
    "max_id = len(tokenizer.word_index) # number of distinct characters\n",
    "dataset_size = tokenizer.document_count # total number of characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "CHKUXWkZtv8Y"
   },
   "outputs": [],
   "source": [
    "[encoded] = np.array(tokenizer.texts_to_sequences([shakespeare_text])) - 1\n",
    "train_size = dataset_size * 90 // 100\n",
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qpcJ1RMRtv8Y"
   },
   "source": [
    "**Note**: in previous versions of this code, we used `dataset.repeat()` now to make the dataset \"infinite\", and later in the notebook we set the `steps_per_epoch` argument when calling the `model.fit()` method. This was needed to work around some TensorFlow bugs. However, since these bugs have now been fixed, we can simplify the code: no need for `dataset.repeat()` or `steps_per_epoch` anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "KbUR-r0etv8Y"
   },
   "outputs": [],
   "source": [
    "n_steps = 100\n",
    "window_length = n_steps + 1 # target = input shifted 1 character ahead\n",
    "dataset = dataset.window(window_length, shift=1, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "Ye9EnvsNtv8Z"
   },
   "outputs": [],
   "source": [
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "CgbRf_3Ttv8Z"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "-1EhiiPTtv8Z"
   },
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "dataset = dataset.shuffle(10000).batch(batch_size)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "rHvQY_f7tv8Z"
   },
   "outputs": [],
   "source": [
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "_aN59dDXtv8Z"
   },
   "outputs": [],
   "source": [
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i8-oYF1Ytv8Z",
    "outputId": "90b6f4a1-c89a-4562-9fe8-191152ea1795"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 100, 39) (32, 100)\n"
     ]
    }
   ],
   "source": [
    "for X_batch, Y_batch in dataset.take(1):\n",
    "    print(X_batch.shape, Y_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FHHzJwNvtv8Z"
   },
   "source": [
    "## Creating and Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ELzs38IHtv8a"
   },
   "source": [
    "**Warning**: the following code may take up to 24 hours to run, depending on your hardware. If you use a GPU, it may take just 1 or 2 hours, or less."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NrcuFoErtv8a"
   },
   "source": [
    "**Note**: the `GRU` class will only use the GPU (if you have one) when using the default values for the following arguments: `activation`, `recurrent_activation`, `recurrent_dropout`, `unroll`, `use_bias` and `reset_after`. This is why I commented out `recurrent_dropout=0.2` (compared to the book)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a9kyiPvKtv8a",
    "outputId": "70576ffc-d431-40e6-9af4-75dba89811c8"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "31368/31368 [==============================] - 376s 12ms/step - loss: 1.6206\n",
      "Epoch 2/10\n",
      "31368/31368 [==============================] - 351s 11ms/step - loss: 1.5369\n",
      "Epoch 3/10\n",
      "31368/31368 [==============================] - 347s 11ms/step - loss: 1.5171\n",
      "Epoch 4/10\n",
      "31368/31368 [==============================] - 344s 11ms/step - loss: 1.5053\n",
      "Epoch 5/10\n",
      "31368/31368 [==============================] - 346s 11ms/step - loss: 1.4980\n",
      "Epoch 6/10\n",
      "31368/31368 [==============================] - 344s 11ms/step - loss: 1.4927\n",
      "Epoch 7/10\n",
      "31368/31368 [==============================] - 344s 11ms/step - loss: 1.4891\n",
      "Epoch 8/10\n",
      "31368/31368 [==============================] - 347s 11ms/step - loss: 1.4864\n",
      "Epoch 9/10\n",
      "31368/31368 [==============================] - 345s 11ms/step - loss: 1.4842\n",
      "Epoch 10/10\n",
      "31368/31368 [==============================] - 346s 11ms/step - loss: 1.4821\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id],\n",
    "                     #dropout=0.2, recurrent_dropout=0.2),\n",
    "                     dropout=0.2),\n",
    "    keras.layers.GRU(128, return_sequences=True,\n",
    "                     #dropout=0.2, recurrent_dropout=0.2),\n",
    "                     dropout=0.2),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "history = model.fit(dataset, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uRLSVGwNtv8a"
   },
   "source": [
    "## Using the Model to Generate Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "fIykIportv8a"
   },
   "outputs": [],
   "source": [
    "def preprocess(texts):\n",
    "    X = np.array(tokenizer.texts_to_sequences(texts)) - 1\n",
    "    return tf.one_hot(X, max_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VFa0syKNtv8a"
   },
   "source": [
    "**Warning**: the `predict_classes()` method is deprecated. Instead, we must use `np.argmax(model(X_new), axis=-1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "TLQSphICtv8a",
    "outputId": "1a596e89-182b-456f-aa6a-cb1900ad9731"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = preprocess([\"How are yo\"])\n",
    "#Y_pred = model.predict_classes(X_new)\n",
    "Y_pred = np.argmax(model(X_new), axis=-1)\n",
    "tokenizer.sequences_to_texts(Y_pred + 1)[0][-1] # 1st sentence, last char"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "70qpvyJotv8b",
    "outputId": "62a6d5ba-e080-4a6b-84de-893c880ae621"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0,\n",
       "        2, 0, 0, 1, 1, 1, 0, 0, 1, 2, 0, 0, 1, 1, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "tf.random.categorical([[np.log(0.5), np.log(0.4), np.log(0.1)]], num_samples=40).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "9thNHN1rtv8b"
   },
   "outputs": [],
   "source": [
    "def next_char(text, temperature=1):\n",
    "    X_new = preprocess([text])\n",
    "    y_proba = model(X_new)[0, -1:, :]\n",
    "    rescaled_logits = tf.math.log(y_proba) / temperature\n",
    "    char_id = tf.random.categorical(rescaled_logits, num_samples=1) + 1\n",
    "    return tokenizer.sequences_to_texts(char_id.numpy())[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "cn1DKcmXtv8b",
    "outputId": "39a9ca0a-f93f-40ba-cf23-b3e92f8091ec"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "next_char(\"How are yo\", temperature=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "wkCwbGritv8b"
   },
   "outputs": [],
   "source": [
    "def complete_text(text, n_chars=50, temperature=1):\n",
    "    for _ in range(n_chars):\n",
    "        text += next_char(text, temperature)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cbulJBostv8b",
    "outputId": "b6bd632c-a6ac-47e3-c9c6-94af9ae0e012"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the maid in padua for my father is a stood\n",
      "and so m\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\", temperature=0.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f7mHvXartv8b",
    "outputId": "92f99ad1-32e8-4bee-99cb-fec63349b8ab"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toke on advised in sobel countryman,\n",
      "and signior gr\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eQ4bne4Ttv8b",
    "outputId": "7c4321dc-77e3-4cb2-de54-ed4ad1ece239"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tpeniomently!\n",
      "well maze: yet 'pale deficuruli-faeem\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GJZt_KEttv8b"
   },
   "source": [
    "## Stateful RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "AmtuPKFutv8c"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "01BB7utQtv8c"
   },
   "outputs": [],
   "source": [
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])\n",
    "dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "dataset = dataset.batch(1)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "DtmRl3Lktv8c"
   },
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "encoded_parts = np.array_split(encoded[:train_size], batch_size)\n",
    "datasets = []\n",
    "for encoded_part in encoded_parts:\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(encoded_part)\n",
    "    dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "    dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "    datasets.append(dataset)\n",
    "dataset = tf.data.Dataset.zip(tuple(datasets)).map(lambda *windows: tf.stack(windows))\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nIeQlIwDtv8c"
   },
   "source": [
    "**Note**: once again, I commented out `recurrent_dropout=0.2` (compared to the book) so you can get GPU acceleration (if you have one)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "Jtqxh8catv8c"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     #dropout=0.2, recurrent_dropout=0.2,\n",
    "                     dropout=0.2,\n",
    "                     batch_input_shape=[batch_size, None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     #dropout=0.2, recurrent_dropout=0.2),\n",
    "                     dropout=0.2),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "W5R2Uowztv8c"
   },
   "outputs": [],
   "source": [
    "class ResetStatesCallback(keras.callbacks.Callback):\n",
    "    def on_epoch_begin(self, epoch, logs):\n",
    "        self.model.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2U1RYYINtv8c",
    "outputId": "d8e6b02c-2b2c-47f0-d0d1-8e14a184d851"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "313/313 [==============================] - 6s 12ms/step - loss: 2.6200\n",
      "Epoch 2/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 2.2410\n",
      "Epoch 3/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 2.1105\n",
      "Epoch 4/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 2.0368\n",
      "Epoch 5/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.9860\n",
      "Epoch 6/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.9488\n",
      "Epoch 7/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.9205\n",
      "Epoch 8/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8985\n",
      "Epoch 9/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8797\n",
      "Epoch 10/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8655\n",
      "Epoch 11/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8533\n",
      "Epoch 12/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8412\n",
      "Epoch 13/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.8328\n",
      "Epoch 14/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.8233\n",
      "Epoch 15/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.8160\n",
      "Epoch 16/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.8072\n",
      "Epoch 17/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.8008\n",
      "Epoch 18/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7936\n",
      "Epoch 19/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7885\n",
      "Epoch 20/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7851\n",
      "Epoch 21/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7814\n",
      "Epoch 22/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7760\n",
      "Epoch 23/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7729\n",
      "Epoch 24/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7697\n",
      "Epoch 25/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7645\n",
      "Epoch 26/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7606\n",
      "Epoch 27/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7584\n",
      "Epoch 28/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7564\n",
      "Epoch 29/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7538\n",
      "Epoch 30/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7496\n",
      "Epoch 31/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7470\n",
      "Epoch 32/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7455\n",
      "Epoch 33/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7432\n",
      "Epoch 34/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7408\n",
      "Epoch 35/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7376\n",
      "Epoch 36/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7363\n",
      "Epoch 37/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7343\n",
      "Epoch 38/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7308\n",
      "Epoch 39/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7286\n",
      "Epoch 40/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7284\n",
      "Epoch 41/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7269\n",
      "Epoch 42/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7252\n",
      "Epoch 43/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7233\n",
      "Epoch 44/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7233\n",
      "Epoch 45/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7222\n",
      "Epoch 46/50\n",
      "313/313 [==============================] - 4s 11ms/step - loss: 1.7193\n",
      "Epoch 47/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7181\n",
      "Epoch 48/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7175\n",
      "Epoch 49/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7146\n",
      "Epoch 50/50\n",
      "313/313 [==============================] - 4s 12ms/step - loss: 1.7138\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "history = model.fit(dataset, epochs=50,\n",
    "                    callbacks=[ResetStatesCallback()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2RGPpbcGtv8c"
   },
   "source": [
    "To use the model with different batch sizes, we need to create a stateless copy. We can get rid of dropout since it is only used during training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "IIC90vH0tv8c"
   },
   "outputs": [],
   "source": [
    "stateless_model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-OI31vj-tv8d"
   },
   "source": [
    "To set the weights, we first need to build the model (so the weights get created):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "egRIiYQ5tv8d"
   },
   "outputs": [],
   "source": [
    "stateless_model.build(tf.TensorShape([None, None, max_id]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "lrb3Fe3ntv8d"
   },
   "outputs": [],
   "source": [
    "stateless_model.set_weights(model.get_weights())\n",
    "model = stateless_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ak3Kbakbtv8d",
    "outputId": "755df3e9-8c69-4f68-937c-b001fb9618a5"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thing idsumper your shint.\n",
      "why, he has go too stone\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7MwWEzGatv8d"
   },
   "source": [
    "# Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "xq2e8DSctv8d"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZNmhfUTmtv8d"
   },
   "source": [
    "You can load the IMDB dataset easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qE1O3ctLtv8d",
    "outputId": "f1443ccb-7adf-45aa-d0fb-697b2c5c950d"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 0s 0us/step\n",
      "17473536/17464789 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2po-SjZLtv8d",
    "outputId": "6af47e1b-703b-4cfe-9ff2-400f006ca477"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "id": "gJhVZdrDtv8d",
    "outputId": "10f0ba39-10af-4f75-ce28-5996234bb35a"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n",
      "1654784/1641221 [==============================] - 0s 0us/step\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'<sos> this film was just brilliant casting location scenery story'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index = keras.datasets.imdb.get_word_index()\n",
    "id_to_word = {id_ + 3: word for word, id_ in word_index.items()}\n",
    "for id_, token in enumerate((\"<pad>\", \"<sos>\", \"<unk>\")):\n",
    "    id_to_word[id_] = token\n",
    "\" \".join([id_to_word[id_] for id_ in X_train[0][:10]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 314,
     "referenced_widgets": [
      "b72961c5f20144f1b6a5fb4ee8f229c0",
      "d25b97937c34400dbec01634f50903ce",
      "a9e3740391834a0ab6323769f87fabde",
      "ed245df63f5e45be93ba69845d938f1f",
      "8c5416cc97414dd683f02251c8e9e84e",
      "12d54c3d7f6b4b649fb91903ea237b46",
      "c4e1ce8cec7c40d1ad6f64ca065f061a",
      "aac6f05b44f34c32937bda7bcdc230f9"
     ]
    },
    "id": "pRN6EmIetv8e",
    "outputId": "b61195cf-5b6f-4441-f982-38ce0a9f410e"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0...\u001b[0m\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b72961c5f20144f1b6a5fb4ee8f229c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Completed...: 0 url [00:00, ? url/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d25b97937c34400dbec01634f50903ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Size...: 0 MiB [00:00, ? MiB/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9e3740391834a0ab6323769f87fabde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteAII20M/imdb_reviews-train.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed245df63f5e45be93ba69845d938f1f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/25000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c5416cc97414dd683f02251c8e9e84e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteAII20M/imdb_reviews-test.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12d54c3d7f6b4b649fb91903ea237b46",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/25000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4e1ce8cec7c40d1ad6f64ca065f061a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteAII20M/imdb_reviews-unsupervised.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aac6f05b44f34c32937bda7bcdc230f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/50000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:Dataset is using deprecated text encoder API which will be removed soon. Please use the plain_text version of the dataset and migrate to `tensorflow_text`.\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset imdb_reviews downloaded and prepared to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rhPcmxlBtv8e",
    "outputId": "09f261fa-014e-4f42-df61-408ff371a52d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['test', 'train', 'unsupervised'])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "BcYohezutv8e"
   },
   "outputs": [],
   "source": [
    "train_size = info.splits[\"train\"].num_examples\n",
    "test_size = info.splits[\"test\"].num_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cH23o0Hutv8e",
    "outputId": "7c8e2839-8191-4f1c-84b6-b234484af154"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 25000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_size, test_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Sjaiqp9Ytv8e",
    "outputId": "d148512c-d75a-4d4f-c71a-d0d7148a3953"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting  ...\n",
      "Label: 0 = Negative\n",
      "\n",
      "Review: I have been known to fall asleep during films, but this is usually due to a combination of things including, really tired, being warm and comfortable on the sette and having just eaten a lot. However  ...\n",
      "Label: 0 = Negative\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in datasets[\"train\"].batch(2).take(1):\n",
    "    for review, label in zip(X_batch.numpy(), y_batch.numpy()):\n",
    "        print(\"Review:\", review.decode(\"utf-8\")[:200], \"...\")\n",
    "        print(\"Label:\", label, \"= Positive\" if label else \"= Negative\")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "C1Hgdwebtv8e"
   },
   "outputs": [],
   "source": [
    "def preprocess(X_batch, y_batch):\n",
    "    X_batch = tf.strings.substr(X_batch, 0, 300)\n",
    "    X_batch = tf.strings.regex_replace(X_batch, rb\"<br\\s*/?>\", b\" \")\n",
    "    X_batch = tf.strings.regex_replace(X_batch, b\"[^a-zA-Z']\", b\" \")\n",
    "    X_batch = tf.strings.split(X_batch)\n",
    "    return X_batch.to_tensor(default_value=b\"<pad>\"), y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IeH4S4n_tv8e",
    "outputId": "c8b1931e-b4ed-49fb-f2ce-b857f833b9f6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2, 53), dtype=string, numpy=\n",
       " array([[b'This', b'was', b'an', b'absolutely', b'terrible', b'movie',\n",
       "         b\"Don't\", b'be', b'lured', b'in', b'by', b'Christopher',\n",
       "         b'Walken', b'or', b'Michael', b'Ironside', b'Both', b'are',\n",
       "         b'great', b'actors', b'but', b'this', b'must', b'simply', b'be',\n",
       "         b'their', b'worst', b'role', b'in', b'history', b'Even',\n",
       "         b'their', b'great', b'acting', b'could', b'not', b'redeem',\n",
       "         b'this', b\"movie's\", b'ridiculous', b'storyline', b'This',\n",
       "         b'movie', b'is', b'an', b'early', b'nineties', b'US',\n",
       "         b'propaganda', b'pi', b'<pad>', b'<pad>', b'<pad>'],\n",
       "        [b'I', b'have', b'been', b'known', b'to', b'fall', b'asleep',\n",
       "         b'during', b'films', b'but', b'this', b'is', b'usually', b'due',\n",
       "         b'to', b'a', b'combination', b'of', b'things', b'including',\n",
       "         b'really', b'tired', b'being', b'warm', b'and', b'comfortable',\n",
       "         b'on', b'the', b'sette', b'and', b'having', b'just', b'eaten',\n",
       "         b'a', b'lot', b'However', b'on', b'this', b'occasion', b'I',\n",
       "         b'fell', b'asleep', b'because', b'the', b'film', b'was',\n",
       "         b'rubbish', b'The', b'plot', b'development', b'was', b'constant',\n",
       "         b'Cons']], dtype=object)>,\n",
       " <tf.Tensor: shape=(2,), dtype=int64, numpy=array([0, 0])>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess(X_batch, y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "cw1ZWHX8tv8e"
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "vocabulary = Counter()\n",
    "for X_batch, y_batch in datasets[\"train\"].batch(32).map(preprocess):\n",
    "    for review in X_batch:\n",
    "        vocabulary.update(list(review.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JoYwyBIMtv8e",
    "outputId": "7b10f55f-3f8d-4656-a9e1-00bf7fce01dc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(b'<pad>', 214309), (b'the', 61137), (b'a', 38564)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary.most_common()[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rpGTjDvatv8e",
    "outputId": "452db943-1c98-4e00-8fb8-811279288388"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53893"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "MIfThoO3tv8f"
   },
   "outputs": [],
   "source": [
    "vocab_size = 10000\n",
    "truncated_vocabulary = [\n",
    "    word for word, count in vocabulary.most_common()[:vocab_size]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UabK1uOdtv8f",
    "outputId": "5ee62e80-8798-4d76-b9dd-0570f0c24cf1"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22\n",
      "12\n",
      "11\n",
      "10000\n"
     ]
    }
   ],
   "source": [
    "word_to_id = {word: index for index, word in enumerate(truncated_vocabulary)}\n",
    "for word in b\"This movie was faaaaaantastic\".split():\n",
    "    print(word_to_id.get(word) or vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "Yh3rF3SZtv8f"
   },
   "outputs": [],
   "source": [
    "words = tf.constant(truncated_vocabulary)\n",
    "word_ids = tf.range(len(truncated_vocabulary), dtype=tf.int64)\n",
    "vocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids)\n",
    "num_oov_buckets = 1000\n",
    "table = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fFlaVVEstv8f",
    "outputId": "90bdde22-d10f-4922-f05b-8c3f503570c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 4), dtype=int64, numpy=array([[   22,    12,    11, 10053]])>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.lookup(tf.constant([b\"This movie was faaaaaantastic\".split()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "BPwxwEELtv8f"
   },
   "outputs": [],
   "source": [
    "def encode_words(X_batch, y_batch):\n",
    "    return table.lookup(X_batch), y_batch\n",
    "\n",
    "train_set = datasets[\"train\"].batch(32).map(preprocess)\n",
    "train_set = train_set.map(encode_words).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wICjZfKItv8f",
    "outputId": "a812a24d-7bba-438d-9dc0-f9f247c66209"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[  22   11   28 ...    0    0    0]\n",
      " [   6   21   70 ...    0    0    0]\n",
      " [4099 6881    1 ...    0    0    0]\n",
      " ...\n",
      " [  22   12  118 ...  331 1047    0]\n",
      " [1757 4101  451 ...    0    0    0]\n",
      " [3365 4392    6 ...    0    0    0]], shape=(32, 60), dtype=int64)\n",
      "tf.Tensor([0 0 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 1 0 0 0 1 0 0 0], shape=(32,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in train_set.take(1):\n",
    "    print(X_batch)\n",
    "    print(y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0KFA5SeTtv8f",
    "outputId": "b0eecf4d-3a55-480b-da8b-91b53a6f9ad0"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 18s 16ms/step - loss: 0.5305 - accuracy: 0.7281\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.3459 - accuracy: 0.8549\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1934 - accuracy: 0.9313\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1361 - accuracy: 0.9503\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1032 - accuracy: 0.9634\n"
     ]
    }
   ],
   "source": [
    "embed_size = 128\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size,\n",
    "                           mask_zero=True, # not shown in the book\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.GRU(128),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3-uiKuVCtv8f"
   },
   "source": [
    "Or using manual masking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IKeLuilwtv8f",
    "outputId": "ecd605d0-2c7c-4b98-fe0f-dda222955e22"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 18s 15ms/step - loss: 0.5426 - accuracy: 0.7156\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.3469 - accuracy: 0.8572\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1753 - accuracy: 0.9384\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1274 - accuracy: 0.9542\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 12s 16ms/step - loss: 0.1131 - accuracy: 0.9577\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "embed_size = 128\n",
    "inputs = keras.layers.Input(shape=[None])\n",
    "mask = keras.layers.Lambda(lambda inputs: K.not_equal(inputs, 0))(inputs)\n",
    "z = keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size)(inputs)\n",
    "z = keras.layers.GRU(128, return_sequences=True)(z, mask=mask)\n",
    "z = keras.layers.GRU(128)(z, mask=mask)\n",
    "outputs = keras.layers.Dense(1, activation=\"sigmoid\")(z)\n",
    "model = keras.models.Model(inputs=[inputs], outputs=[outputs])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GeY3V219tv8f"
   },
   "source": [
    "## Reusing Pretrained Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "id": "lxi2b-FItv8f"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "id": "w9fLTsxbtv8g"
   },
   "outputs": [],
   "source": [
    "TFHUB_CACHE_DIR = os.path.join(os.curdir, \"my_tfhub_cache\")\n",
    "os.environ[\"TFHUB_CACHE_DIR\"] = TFHUB_CACHE_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "id": "jexBicYjtv8g"
   },
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "model = keras.Sequential([\n",
    "    hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\",\n",
    "                   dtype=tf.string, input_shape=[], output_shape=[50]),\n",
    "    keras.layers.Dense(128, activation=\"relu\"),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SpaomKpntv8g",
    "outputId": "3dcf8083-ad29-41c0-bb41-e932f7fa4ec3"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe.descriptor.txt\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/saved_model.pb\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/assets/tokens.txt\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.index\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.data-00000-of-00001\n"
     ]
    }
   ],
   "source": [
    "for dirpath, dirnames, filenames in os.walk(TFHUB_CACHE_DIR):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirpath, filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ppj3seNEtv8g",
    "outputId": "ff9d3978-83c4-4fad-be8b-3ee8ea063a75"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 5s 5ms/step - loss: 0.5461 - accuracy: 0.7267\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 4s 5ms/step - loss: 0.5130 - accuracy: 0.7495\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 4s 5ms/step - loss: 0.5081 - accuracy: 0.7532\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 4s 5ms/step - loss: 0.5047 - accuracy: 0.7540\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 4s 5ms/step - loss: 0.5018 - accuracy: 0.7566\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)\n",
    "train_size = info.splits[\"train\"].num_examples\n",
    "batch_size = 32\n",
    "train_set = datasets[\"train\"].batch(batch_size).prefetch(1)\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mtmrsDqktv8g"
   },
   "source": [
    "## Automatic Translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "id": "gftP5fRYtv8g"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "id": "4ZvWFTXwtv8g"
   },
   "outputs": [],
   "source": [
    "vocab_size = 100\n",
    "embed_size = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "id": "B5rC-V6htv8g"
   },
   "outputs": [],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(512, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.TrainingSampler()\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(512)\n",
    "output_layer = keras.layers.Dense(vocab_size)\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell, sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings, initial_state=encoder_state,\n",
    "    sequence_length=sequence_lengths)\n",
    "Y_proba = tf.nn.softmax(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(\n",
    "    inputs=[encoder_inputs, decoder_inputs, sequence_lengths],\n",
    "    outputs=[Y_proba])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "id": "CLuhwZXDtv8g"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GpqD3sHUtv8g",
    "outputId": "903a8ab1-ba5d-4e53-fb71-d25b805b0d3b"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "32/32 [==============================] - 4s 36ms/step - loss: 4.6054\n",
      "Epoch 2/2\n",
      "32/32 [==============================] - 1s 35ms/step - loss: 4.6031\n"
     ]
    }
   ],
   "source": [
    "X = np.random.randint(100, size=10*1000).reshape(1000, 10)\n",
    "Y = np.random.randint(100, size=15*1000).reshape(1000, 15)\n",
    "X_decoder = np.c_[np.zeros((1000, 1)), Y[:, :-1]]\n",
    "seq_lengths = np.full([1000], 15)\n",
    "\n",
    "history = model.fit([X, X_decoder, seq_lengths], Y, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gv-dSC0Rtv8g"
   },
   "source": [
    "### Bidirectional Recurrent Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xVx_rQsPtv8g",
    "outputId": "6daed4f9-8fe4-4058-84cf-a2513b875183"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "gru_10 (GRU)                 (None, None, 10)          660       \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, None, 20)          1320      \n",
      "=================================================================\n",
      "Total params: 1,980\n",
      "Trainable params: 1,980\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(10, return_sequences=True, input_shape=[None, 10]),\n",
    "    keras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QdIRt3CNtv8h"
   },
   "source": [
    "### Positional Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "id": "XiCUANCAtv8h"
   },
   "outputs": [],
   "source": [
    "class PositionalEncoding(keras.layers.Layer):\n",
    "    def __init__(self, max_steps, max_dims, dtype=tf.float32, **kwargs):\n",
    "        super().__init__(dtype=dtype, **kwargs)\n",
    "        if max_dims % 2 == 1: max_dims += 1 # max_dims must be even\n",
    "        p, i = np.meshgrid(np.arange(max_steps), np.arange(max_dims // 2))\n",
    "        pos_emb = np.empty((1, max_steps, max_dims))\n",
    "        pos_emb[0, :, ::2] = np.sin(p / 10000**(2 * i / max_dims)).T\n",
    "        pos_emb[0, :, 1::2] = np.cos(p / 10000**(2 * i / max_dims)).T\n",
    "        self.positional_embedding = tf.constant(pos_emb.astype(self.dtype))\n",
    "    def call(self, inputs):\n",
    "        shape = tf.shape(inputs)\n",
    "        return inputs + self.positional_embedding[:, :shape[-2], :shape[-1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "id": "TSqp0L4Xtv8h"
   },
   "outputs": [],
   "source": [
    "max_steps = 201\n",
    "max_dims = 512\n",
    "pos_emb = PositionalEncoding(max_steps, max_dims)\n",
    "PE = pos_emb(np.zeros((1, max_steps, max_dims), np.float32))[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "id": "JNQMosdrtv8h",
    "outputId": "b99d8a09-a555-493d-9193-732bb3a7301b"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure positional_embedding_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAFgCAYAAAArYcg8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOx9eXhURdb+W1kgkIQ17Iuyg2xhlT2rQGQRRhQUUOZzGdQZEXHmp45+6sjg6AyuIzojnyCKCiqoLIGEpJFAQNZm32SRAAESlkASsnWf3x8nN+kk3enb3beqO9jv8/QD6a5bp+49datOnVUQEfzwww8//PDDDz/8+O0gwNsD8MMPP/zwww8//PBDLfwCoB9++OGHH3744cdvDH4B0A8//PDDDz/88OM3Br8A6Icffvjhhx9++PEbg18A9MMPP/zwww8//PiNwS8A+uGHH3744YcffvzG4BcA/fDDDz/88MMPP35j8HkBUAjxRyHETiFEoRBisZO2s4UQF4QQ14UQnwohaisaph9++OGHH3744UeNgc8LgADOA5gL4NPqGgkhRgF4HkAcgNsAtAfwmvTR+eGHH3744YcfftQw+LwASEQriOh7AJedNH0YwP8R0UEiugrgdQAzZI/PDz/88MMPP/zwo6YhyNsDMBDdAfxg8/deAM2EEI2JqIrwKIR4HMDjANAY6HdbSAioQQNYmzQBAoyTi69cCcD58wGwWMq/Cw8ntGljQXCwYWQMh8ViQWBgoFfHIK5fR0B2NkRuLlBaspDq1gU1bAhr48aAEIbQIQIuXQrAxYsBsK2M2LAhoVUrC2wfgy88F19DwOXLEFeuQOTnl31H4eGwNm4Mql/fMDoWC3DhQgCysyu+n02bWtG8udWo6WAofGa+ECHg0iWIa9cgCgr4OyF4zWvcGBQaahip4mLg/PlAXLtWzhAhgJYtrYiIsPrOM/ExWCwWBAqBgMxMBFy/DhQV8Q8BAbA2agSKiADVNs6rqaBA4Ny5AOTmlvMpMBBo3dqCBg18p0SsT86XkhIEnjvHe1NJCX8XFARr48awRkQAQcaJVrm5AufPB+DmTdsFblc2ETXxuHMiqhEfsBl4cTW/nwAw2ubvYAAE4HZnffdp1YpoxAgiIYgGDybKySFPceYM0ciRRADR8OFE//d//Hn1VaI6dYjq1yf69FMiq9VjUlKQlZXl3QH8+9/88Nq2JZo1i2jtWqJ584i6d+fv77uPqLjYYzK7dhH16MFd3nsv0ZIlRAsXEj3zDFFAAJNPSSlv7/Xn4kuwWolmzyYCqLh7d6KXXyZav57ohReIWrfmh/qXvxgyyVetImrRgl/RP/yB6IsviD7+mGj6dCbTrx/R4cMG3JPB8In5cvMm0cSJ/KCioojeeINo3Tqip57ihQgg+u9/DSH1738ThYYS1a5N9OKLzKcPPiBKSGAy48YRHTmSbQitWw3ZR44QDRzIC8+ECUTvvku0ejXRgw8S1apFFBzMfPMQFgvRs88ymYYNif75T+bT/PlEAwYwn556iqeNL8An3iFbHD1K1K4db+QPP0z0n/8QrVhRPskbNiTat89jMvn55a9tmza83n32GW+DAHaSEXKVEZ2o+OgQAPcCuN/m78alAmBjZ3337t2bn/h33xEFBfFLePWqGyxjXLtG1LEjUVgY0Ycf8gtni+PHWd4EmKmqMWvWLJo1a1a1bbz60v3rX/xwxo8nKihw/LuHQuDJk/yutmpF9MMPVX/fupWoUycWOpKS+DuVzyUxMZESExOV0XMJJSVEjz7KfHj6acq6eLHq7088wb+/9JJHpLZs4b2vd2+in3+u+vuKFUSNG/P7duiQR6QMh9c3r5wcopgY5sO771b9PS+P6O67eZJ//rlHpL74gsmMHk104kTF36xWovfeYzmmdesS8vZj8TmcOkXFHTsShYQQrVxZ9ffMTH4B6tQhSkvziNQrrzCfHnuMKLuSLF5YSDRnDv8+eLD95Vc1vP4O2SI9nRebJk3sL0b79xO1bEnUvHnVl8AFWK1EDzzAfJg7l4VBW/gFwKq/fwng7zZ/xwK4oKfvMgGQiOj773m36d+/6lPXAauVpfbAQKJNmxy3s1h4oaxVi2jnTpfJeISoqCiKioqqto3XXrr33isX7oqKHLfThMDJk93SMOXnE0VGEjVoQPTLL47b3bhBdMcd/L6fO6f2uSxatIgWLVqkjJ5LsBXurFb7z8ViKRcSX3/dLTLnzvFa2qED0ZUrjtv9+ivzqEcPlml8BV7dvCwWFv4CA6sX7vLziWJjWSX07bdukdq5k2WXESOqf223biWqVctKY8b4rvVDOXJziTp0IEv9+tULdxcvEnXpQlSvHpsu3MA33/DrOGNG9c//yy+53bPPukXGUPiMAHjmDD/7jh2r3zQOHCBq1IiofXui8+fdIvX66/z8582z//tvRgAE+ymGAHgDwOel/w+y0240gAsA7gDQAEAqgH/ooVFBACTiExjANgwX8c9/8qVvv+28bVYWq3Zvv736zc1o+KwAePw4247GjtWn2WNdOOvFXYDVypp7gC0sznDwIFHdumzKz8z0C4CUksIPb/bssq8czheLhWjaNG6/ZYtLZAoLWQsRGsoHa2dYt47JPPqoS2Skwqub10cfkW7z7o0bREOG8EQ/c8YlMhcv8jrWpg3/3xnmzbtBANE777hE5tbFM88QAXT1+++dtz1zhui22/hh5+a6RGbPHmavXs3eU0/x9PG2EcInBECrlTU2devq0+xt28YL19ChLp90NCF92jTHl/6WBMBXS025tp9XAbQFkAugrU3bZwFcBHAdwCIAtfXQqCIAEhE99BBrAg8edMypStiyhQ/bkybp5/m2bUxm/Hh1J2KfFACtVqK4OD5hnTun7xqLhVezJk2ILl/WTerTT3nmv/KK/uF9/jlfM2uWOvWSTwqA+fmsjuvQoYKGvNr5cuMG29l79XLJZP/nP/MzX75c//Cef56vWbpU/zUy4bXNKyODKDyc3ym9C8upU2xinDTJJVJjx/JlepVSly5l0fjxvO6ptn74HLZsYfP7k0/qnyubN/Mk/+tfdZMpLibq1o1fw8xMfdfk57NGvWlTogsXdJMyHD4hAC5axM/8/ff1X6NtNEuW6L7k8mW2St15Z/U+mL8ZAVDFx64AeOkSq3GHDavqxGcHViu7DrZp43oMiWbNNMC/Vxd8UgBcsoQfwocfunad2cxS9+OP62qem8smxSFDdLG1An7/e6LAQCsdO+bade7CJwVATcKyjYwhHfPl22/JFbXP6dPsHvH737s2vOJi5m2jRobEcnkMr2xeVitLZXq1FbaYO5f5tH69ruYmEzd/8039JLKysig7m+OEevRw/T28ZXDzJlHXrhxpdv26a3Nl+nR+QY4f19X8P/9hPtlzL6wOBw6waf/ee127zkh4XQA8d46lsuHDXZusFgsLBc2b616MZs9mTwxnFg+/AChbACTisF2Aw0KdYMUKbvrpp06bVkFhIZuB+/ZVsxg+9thj9Nhjj1XbRulLl5XFjrWDB7v3AJ59lh9+errTpprVePNm18lcuEBUt66VHnjA9WvdwY8//kg//vijGmJ6oAnbdqQyp/NFM6GEh+vS8D70EG88GRmuD3PnTubxa6+5fq3R8MrmtWwZP4D5812/tqCAfZw6dXJqJ9QOva1bu+YurT0Tzc/s669dH+YtAS0ao/Tk79JcOX+e36UxY5w2vXGDZRA3rJEVhrl7t+vXGgGvC4D3388qbndO/j//zA/vz3922vTECdaKP/KI826NEgAF9/XbRmRkJJnN5qo/EAFRUcDhw8CvvwJ169q93mIBevUCrFZg/373UgAtWQI8/DDwzTfApEmuX280srOzERERoYbYX/4CzJ8PmM1Az56uX3/jBnDHHUDz5sD27Q7zA165ArRvD4wYAfz4o3tDnTUrHx98UBd797o31BqNe+4BNm8Gjh8HGjWq8JOu+fLLL0CPHsB99wGff+6w2b59QGQk8NxzwFtvuTfUiROB1FTg1KkqQ1UKpe8RwItQ9+5AcDCwZw/gTv609euB0aOBefOAF15w2Oybb4D77wc+/RT4/e/1d689E4sF6N2b188DB9wbao1FTg7Qti0QHw989x0AN+bK228Dc+bwYjZunMNmf/sb8MorQHo6MHiw826vXr2KzMxMFBUVQZMPCgs5Pa4v566VAiLOxxgY6H5uv5ISnuS1alWbu7a4GLBYBEJDa6FNmxZo2LChw7ZCiF1E1N+9AdnACCmypn8cagCJiH76iSX4BQscNvnsM27yzTeOu3GGkhL20eja1ZD0dh5D2anr2jU+yU6Z4lk/n3zCTEhNddjkL39hdxtPUjQdO5ZN9eoR3XOP+33USBw8SNU5TuqeL3PmsBbx9GmHTcaMYYuLC26dVbBvH/P6hRfc78MIKNdeaAFsX33lWT9jxhBFRDhU7RUVsaKwRw9eu1yB7TPRHN49zEBT8/DGG3zjNo6TLs+VoiKizp3ZdORAtZeZybEIet06r1y5Qvv376fc3Fyy2FhjCgrYiukqr42AxZs+Avn5fOOejMFi4T6qSU9QXKw1sVBubi7t37+frlQTGQq/CViRAKjZOTp0sDv7jTTffvcdc0S225dPmYDffLPKQugWbt5kb+WEBLs/nzvHJsXp0z0jk5WVRX/7Gw/ZXhooI+FTJuAZM9gM4mBe6J4vZ85wrk2bCGJbpKXxs/3HP9wdaDmmTOHNT09kqiwoFQCtVvYeb9fO81Pkxo3MiP/8x+7Pmk+Znij6yrB9JhYLxwZ17OgbB18lyM/ntWrkyApfuzVXNEb89JPdn2fP5tdNr/Xy4MGDlGsnuthqJbp+3TsplrwmAGqCmxEZsW/erFaQzMvj56vJ8bm5uXSwmgBUowRAn68F7HUIAfz5z8CJE8D331f5edky4PRpYO5czyvITZwI9OsH/P3vbMmRhWPHjuHYsWPyCOhFYSHw3ntsBunb17O+QkKAP/0JSExke1IlfPgha/JffdUzMgDwzDNARATw+uue91UdLl++jMuXnZXAVoCzZ4GlS4FHHuEb9wRt2rDd8JNPgGvXqvz87rtssv3TnzwjAzCvb94E/vUvz/uqEdi0Cfj5Z16vPC1FNWIEv5PvvFNlMSICPvgA6NMHuPtuz8gEBLCJ8pdfqvUKuLWwaBFw6VK15nXdmD4daNyY+VQJeXlsnr/vPqBTJ33dFRUVoU6dOlW+F4ItmJo18zeBwkL+t1Ytz/vS+tDK+9nAYuHnamshrlOnDorstDUafgFQDyZOBDp0YIckqugz+dFHQOfO7DLjKYRg4eKXXwCTyfP+fB5ffgmcP88+gEbgiSeAOnXYN8YGRUXAwoXAmDHsA+gpwsOBmTOBNWvYNfSWhyYEzJljTH9z5gC5uSwE2uDcOT5jPfKIQ3dbl9ClC3DvvcD//R+glb+9pfGPfwBNmwIzZnjelxDMpyNHgHXrKvyUlsZnrKeeMqYc9/jx7E/773973pfPo6QE+Oc/gUGD2L/cU9Spw+veDz/wxmGDpUvZ1fCpp/R3R0QIcKDJqEaGufVgtbJTXnCw55odgPsICuI+K8kQxcX8r61/ZUBAAJtoJcMvAOpBYCAvhtu38+pXir17ga1bWRgwqhD9pEmsAfn4Y2P681lYrbwQRkayBtAING4M/M//AF98AWRmln29ciUfuJ94whgyAPDoo/zvwoXG9emTuHoV+O9/gcmTgdtvN6bPvn2BmBjg/ffLVz+wPGi1An/4gzFkAO7rypUyP/tbF3v3sqD2zDMsFBiB++4DWrXiAC0bLFgANGwIPPCAMWSEYD7t3g3s2mVMnz6Lb79lk9ELLxi3aTz5JAsX779f9hURC9SRkcCQIcaQEYKFlOJiuRYqn4Am5Rqh/dNQqxYzxmbNo9IYk6AgY+RMV+EXAPVixgw2f737btlXH3/MlseHHzaOTEgIR9R9/30FGebWQ0oKR1c/95xxCyEAzJ7Np2wbdcJHHwHt2gGjRhlH5rbbgIQE1i7ZvM+3Hr74grV1zz1nbL9z5rBpeflyAPwMP/mENekdOhhHJiaG+/vvf43r0yfxySe8eMycaVyfwcHA009zOHVploTMTBamf/97Y7S0GqZOZbm1klL41sPChbwYjR1rXJ8tWgAPPsj23lK3is2bOSOFUVpaDZo8dEuveZqQFhhobGh6YCBLeUVFZVpA7TkaKWe6Ar8AqBd16rCkt3o1kJ2NGzd4b5w82fg0E48/zjLMp58a26+GyMhIREZGyulcLxYvBho0YBudkejQgR2TPvsMsFhw6BDw00+sYTD6hDVzJm+Iq1YZ26+G5s2bo3nz5nI614vFi9nZq08fY/tNSAA6dmQJGpzJ4vx5VmYYiYAAfp82beLzxi2JwkJ2p5g4kVVzRuLxx3nt+89/ALCAVlJirJwJ8FIweTKbLXNzje3bZ/DrryxMP/yw8YvR7Nns9PfZZwDY57lBA5YLjYQmE9mxZN46sFj45oyWyoQAatdm9anFUqb9CwjwYgokIyJJavqn2ihgW+zdyxFXH3xQVmZz61Z9l7qKuDhOEO+NsHsiydGLOTkcUTpzppz+tUS4ycn0xz9ywvxLl4zp2va5FBdzEtxKwXy3Dvbt4+f47rtOm7o1X157jfv/9VeKjeUSpzLm+8WLnGDVQeCxVCiJAtZyqeis3uEyHnyQqGFDKrpRQC1bEo0a5Vl3jp5JejrfxiefeNa/z0Kb76dO2f3Z47nSty9Rv350/ny1gfbVYreObM+FhbyEq4raVh4FnJfHN+ggtc68efOof//+FB4eThERETR27FjaX6l0h8M2Wjh1fn5Z6pfCQvvDqI4X8EcBewG9egGRkaAlS/Dxx+xfceedckjNnAmcOVPF//rWwDffcHimkbZzW4wbB9Svj5JPl2DJEvarbNLEeDJBQcBjjwFJScDJk8b373V89hnfpNFqBA3TpgEAst9bitRU1tLKOAk3bcrKsc8+u0WDQRYtYl+9uDg5/U+fDly9CvO8tTh/3lhfWlsMGsR5wm9Jc73Vytr02FjjfGkrY/p0YNcurJ1/WIqWVoMWrHBLmoGJWMUdHOzQdr5x40Y8+eSTSE9PR2pqKoKCghAfH48rV644b3P1alkwSHExq1C9mlzbCCmypn90awCJiN5+mwigrjjkctlaV1BUxKmiXKzLrgtTp06lqVOnVttGquZi+HBOYOpOXSK9eOwxKq5dl0Jxo7rc0C6j8nM5e5ZrN770knE0NHz33Xf03XffGd+xHhQXEzVrRjRhgq7mbs+XYcPoUkRXErDS2bPudaEHKSmsfFm6VB4Ne5CuATx/niegzIzXpXPh59YTqXFjXps8QXXP5P33mU979nhGw+eg5VVcssRhE4/nSmYmUUAALWz2Ig0Y4F4XejSAROX5kWUu4Rr0agCzsrIIAL399tvUv39/ql27NnXq1InWu6IZ19SbLpgibty4QQEBAdXmbK3QplT1l59TWG0JRb8G0Bfx4IOwiEDMEEtw333yyAQHcxDemjXG+8ScPXsWZ8+eNbZTvTh5kiOpZ8ww1ju5Mh56CEGF+fif+iswYoQ8Mq1acaDBsmXG+8Rcv34d169fN7ZTvVi/Hrh4UZ6WthQ0/SE0yT6Cx/ruQqtW8uhER3MKwq+/lkfDK/jiC9YuyeRTUBCK73sQkWdX46GxV6RqLKZN47Xvyy/l0fAKFi3i/FFG+zzbonlz5A4dibiLS/HAZGPDdKOjoyt8Ro+OxiefLEBxMZCfn1/l9+joaCxevBgAl7iz9/uyZcsAABkZGVV+cwdaOdeFCxfizTffxL59+9CrVy88+OCDuHnzZoW28+bNQ1hYWNVPo0YIa9kSYfXrI80m40d1uHHjBqxWa7Wl2yq0CQwECYFgFHu9tJ5fAHQR1LQZfqozGv9T63M0aSQ3I+bkyWwplRVk4BUsWcKC3/TpUslc7zkUp9AOT9VbIt3BdvJkLo9rr5x0jcVnn3HUu6eZfp3g4B33oQC18cd6S6TSCQjgA9X69XbzT9dMELFZcfBgTnooEamtpqMWijGz0XKpdBo2BO66i4PDjT5QeQ25uZz+ZfJkY0On7WB9xDTcjl8xvd1mqXSE4I8vmYHNZjMCAwOxcuVKxMbGonPnznjzzTdx+fJlHDlypELbmTNnwmw2V/zs2gVzWhrM27fDbDajf399pXZnzZqFyMhIDK6m0HKFNkKgRAQjEBYEBng5n44RasSa/nHFBLxjB9F9KA0ySErSfZ07sFiIWrUyvu5sVFQURUVFVdtGiunKauUyVfHxxvddCUuWEL2CV8gqBJcfMwj2nktWFpe3ff55w8gQEdGiRYtokey6gPZw9SpHzjz9tO5L3J0vL7xAtFzcR5bGEZ7bFp1g2zZ+bT/7TCqZCpBqAt69m2/o44/l0SjFhHusdDioO1kHD/G4L2fPZPFiUlJqURm+/JJvaNOmapt5OlesVqLeHXMpPyCUyEmpT0fQawImclrdzDDoNQFPnTqVJlRyWbl06RIB0HdfbtzQ7NmzqUWLFnTixAndbaxWohs5JUyroMDhdX4TsA9i2TIgMWg8KCy8LIeZLGhai8REzuhe47F7N3DqlLygAhssWwaktJgOQcR/SEREBOeylmEG9grWrOH8BFOmSCVDxCbZ/b2nI+ByNqvnJGLgQM7fKPm1VYcVK3iR+N3vpJLJyQHWJgocv3M6xNZ0LospEffcw2bgb76RSkYdVqwAmjcHhg6VSsZsBvb+EoqMgb/jSS454kkzX5aUSCWjG2azGX0qpatKT09HSEgIulTSkNs1AUdEsPm3Xj2EhYU5NQHPnj0bX331FVJTU9HeQYkpe21KSgArAkEBAV5XofoFQBdgtfJ7FTUqBGLsGC6/I7kw4uTJvBf/8INxfQ4ePLhadbU0rFzJYZ7jx0slc+UKyxKDpnbgUO2VK6XSA7i87alTxlYyaN26NVq3bm1ch3qxYgUnl5UV4l6KHTv4mXV4chRQr550PgnBfEpK4gInNR4rVnDNXhkh7jZYuZLXoNbPTSmnKxENGgAjR94iZuD8fGDtWmDCBOmlHr76igNMmz87laX25GSp9AICfMcMXFBQgKNHj8JaqUTJ/PnzMWXKFNStZHqvYgKuZP51ZgKeNWtWmWDXtWtXl9oUF5e6vwcHl+UE9BqMUCPW9I9eE/CWLVQeyKXlmvvpJ13XugurlfMB3n23VDJVIMV01a0bUUyM8f1WwiefMGt27iTOvSUER8gZAEfP5coVzjX33HOGkPEe8vKI6tYlevJJly5zZ77Mns3P7OpVInrgAaKICOmJL3fs4Lnx6adSyZRBmgn48GG+kfffl9O/DUaNIrr99tKIzz59iIZ4ZgbW80w++4xvb9s2j0h5HytXkpaT1Bk8mSsWC1GbNqX7RGEhUb16RI884nI/rpiAidSYgfWYgHfs2EGBgYHUuXNn2rRpEx05coSmTZtGLVq0oEw9a39Bge4befLJJyk8PJxSUlIoMzOz7HPjxg2nbXJyblBODj83sliqNQP7TcA+hmXLOJH3PfeAKxnUrq1Ua2GTZqjm4cgRLsUg2VwFMJ86dOCSs5g4kdUIRqpQ7eCWcV5PSmKthWQ+adr0hATW+GDCBCA7G9iyRSrdfv24EleNNwNr687EiVLJXL4MbNjA3gBCgPm0dStw4YJUuuPHcyGGGs+n777jxSEqSiqZHTuAjAy2GKFWLWDMGC6vI1m75CtmYLPZjE6dOuG1117DAw88gD59+uDq1atIS0vTV01JK/2mQ0u7YMEC3LhxA3FxcWjRokXZ51//+pfTNv/8J7cJDgbTCgjw6sPzC4A6QcSWj9Gj2VqF8HDe8VeulL7jT57Mc+THH43p795778W9MtMR2IO2YU2YIJXM1auAycS+k0KAM8t26MDFlSVj8mRO3r19uzH9LVu2rCxVgjKsWMEblszcOWB30HPnbLJiJCTwxiWZT9qBasMGFm5qLFasYBO9ZBeBxESWIcrkTO1AJTk1QYMGXLv7m2/4sFAjUVTEz0lzapSIVatYfikrMTxhApCVxcK6RPiKGdhsNqNnz56YMmUKzp49i/z8fKxevRod9BQWt1r5ExSki5Yjbdqrr77qtM0LL7wKIWzkzOBgfsG8NMn9AqBO7N3Ltevvucfmy4kTub6j5Pwf/fpxvjmj1tzLly/jsurdb+VK9sKXvGGtW8fvU5mboRC8GKakSI+kGTeOF2GjBPWbN29WyV8lFdqGNX68kg0rIMAmy0x4OEfSfP+99APVvffygSoxUSoZeThzBti5U4k2fdUqjl8oc4fq0QNo317JgWrSJNZqGelXqxQmE685Cvj044/AsGE2delHj1Z2oNJkGG9aPsxmM3r16uXexZr0KnnNo9IiI0FBNilwNaHTS1pAvwCoE6tWMdMqpEUbN453MQVm4LFj2TpXWCiVlBxkZLCNQrK5CmA+NWnCsmYZJk7kl3ztWqm0GzbkRXj1aqlk5GHjRk6Sp0iwGDyYI6jLMHEiR4Xs2yeVdr9+LNTUWD4pMv8WFbGQPHasjcZCO1Bt2ADcuCGV/t13M901a6SSkYfvvgPCwthSJBGnTwP79/N2VIZ69bjsnIIDlZdlGBBRWdJnt1BSUm6OlQjNGl9B0aipUP0CoG9j9WoWKpo1s/mySRNg+HAlUaZjx3I+0U2bpJMyHtopVPKGpWl17r67Uk3ZwYOZcQq0FmPHsvxy5ox0UsZj5UogNFT6hnX2LLBnT6UNC+AvhJDOp4AAdpFat877piu38N13QM+eQKdOUsls2sQyXhU+TZjA0qHkQuUREVwfuEYK6hYLz+MxY4CQEKmkNMtQleQKEyZwyp5Dh6TSDwz0qgwDIQSuX7+O8e5kl9CicBWU5NDWmgoCoKZCLSnxigrVLwDqwIUL7NdVZSEEWFty4ACXgpCIuDigTp0aWhVk5UqgWzfp1Qq2bGEFVhU+BQSw7X7tWum5sTQfnBqntdACZUaP5okmEdqGXoVPzZpxrjRFB6qcHOkxJ8bjyhUedAVfFDlYtYpll/j4Sj8MGcLSmaID1a5dQGamdFLGYudO9sGTnPIKYD516WLnPKDRVmAGDgrymgzjGYT5EaAAACAASURBVDSpVaf/n7uwa/7V4EUVao0QAIUQjYQQK4UQeUKIX4UQdjMJCyFeFUIUCyFybT72MzS6AM1yWOZgawvtJZNsXqxTh4XA1as9f8ni4uIQFxdnzMCc4fp1rv1rV3o2FqtWsdvLyJF2frznHlahbtwodQxdunDMiRFai3bt2qFdu3aed6QHe/fyLjtmjHRSq1axG1m3bnZ+nDCBx/Lrr1LHEB/Pc6XGaZeSklhrIZlPROxXFh9vp3qZlstz9WrpKlRtzZW8vBqPtWt5px81SiqZnBxe0uzKmS1asApVwYEqKIjnjDdT2rmFkhJUjMqQA6uVn49dOdOLKtQaIQAC+BBAEYBmAKYC+EgI0d1B22VEFGbzOekp8VWruJC8XReD228HunZV4lE+bhy7SHmq0X/55Zfx8ssvGzMoZ0hJ4YmdkCCd1KpVQHQ0xxNUQUwMqzMkm600f82UFCAvz7O+oqKiECU5fUQZtPk7erRUMnl5/Gw0a28VaIKNZD6FhfFcqXEC4Nq1QOPGwIABUskcPMi+ZQ7PbePG8eEuPV3qOHr04LW3xvEpMZGFr8aNpZJZv55lcIeKxvHjWYUqOW2Pt/0A3UK1ajljUa2i0YsqVJ8XAIUQoQDuBfAyEeUS0WYAPwKYroJ+QQEnVB87tpo5kpDAx7D8fKlj0fbGGrUYJiayRCa5DNKxY/xxuGHVqcM7vmTBAuAxFBYCqanSSRmHxESgTx/WGkjEhg38bBzyqUsXrtem4EA1dixw9Cjwyy/SSRkDq5Xn78iRlZxcjYfmamLX6gFwgEFQkLIDVXJyDQqAu3SJg94qRAzKwapVLGM6LOykHbyTkqSOQwiekjVKALQblSEHTtMMaipUxelg5N+55+gMoISIjtl8txeAI9XIOCHEFQCZAP5NRB/ZaySEeBzA4wDQsmVLZGdn2+0sJSUYeXn1MWJEDrKz7Zs7gocMQf133kHODz+gWKIDfe3aQM+eDbByJeGRR9xPaTJ58mQAqDbHXI4RKVOI0HDNGpSMGIEbklOwLFsWAiAMQ4ZcQXa2/ZcoZNgwhK1bhyu7dsF6221u0dHzXLp1A8LCGuHbbwsxeLD7asBvSouh3nfffW73oQciJweN0tNx809/Qr6D98AZ9M6Xb74JQ3h4LXTrdgWOSIXGxKD2N9/gyvnzbKeVhMGDAwA0wrJlufjDH+T4hhryHpUiyGxGg6ws3Bg+HIVu8kkvVq6sj969gVq1chzyqf7AgRCrV+PanDku9e3qMxk+PBgffVQfP/6Yg5gY34/aqf3ttwgHcHXwYFhc4JOrz8ViAdasaYS77irC1au59hu1aoVGTZui6IcfkKtTIK1cTk0vgoIECgsFLBaCEMZqstwdU3UQpdIqBQZKFbxYrgtArVoEq9XBcwkIgACA4mKQjabJkVxiFGqCABgG4Hql73IA2DP0LQfwXwAXAdwJ4DshxDUi+qpyQyL6b2lbREZGUkSFfBTlSEtjH5gJE+o7DuYaOxaoWxf109OBBx7QdVPuYuJEYO5cQIgIt60LltKTj6N71uDsd6c4cAA4fx6Br76K2p725QQbNwLduwN9+zZy3Oi++4CXXkKjn3/mXCBuQs9zGT0a2LChDho3ruO2daFOaTCGx3xwBpMJsFhQ9957UdcDWs7GScSkRo8GWraspu2ECcDixYg4doy1tpIQEQHccQewcWMY/vrXMIl0DOJfejogBMInTUK4xDlx9SpbDf/6VydjHz8eeP55RBQXu6w5duWZTJjACvy0tPqQfBYyBmlpQLNmaBgT47JvmSvPZft25tWECSGIiKgm0jghASGrVyOkYUOnmuOMjAwEuOkPFxTEWlqLRaBWLeNMqlar1e0xVYuSEiAwEEKy/19REf8bHCwQEFDNcwkMBCyWCuORvfb7vAkYQC6AepW+qwegShIqIjpEROeJyEJE6QDeAzDJE+JJSeXuYw4REsKNFJgXExL4sJKSIp2U59DMeJL9//LygM2bdbivderE0QcK+DRmDHD+POfn8nkkJnLphUGDpJI5dIifiVO/+NhYTo2gwAw8Zgzw008cH+TzSEzkjMxNmkglk5LCa4xTPmkv3Pr1UsejBcDViMj6khJ+HgkJ0gMLkpLY9FolSrsyRo/msjeSM2p7OaWda3Cx+ocnsFh0xpmUCoAq/QBrggB4DECQEMI2yL03gIM6riUAbh9FTp3i7C52o0orIyGBnYkkOxT17897tWSXDmOQmMhe3JKrf2zaxKcsp3wSghfD1FTpDkWaJ4DP84mo3K9M8mKoyQlOvSTCwzmjtgJBfdQo9s/56SfppDxDdjbw889K/MqSkjiP8J13OmnYqxdn1FbAp9GjgZMnOa2dT0NTyykIelu/nuudOz0P3HUXr30K/DVrTDoYX0j/UhleiKTxeQGQiPIArADwNyFEqBBiKIB7AHxeua0Q4h4hREPBGAjgaQA/uEs7OZn/1S0AAtK1FkFBfBpOSvLxl+zGDVbLKVgIk5JYCTt8uI7GCQnlKkOJaNWKzYs+LwBq6V8U8alrV6BtWx2NExI4o/a5c1LHNHQoa5h8nk/aCy+ZT0QsWMTF6dgbtQNVUpL0/B815kC1di1rciQnU8/J4TK/urLMNG7MVQwUaNS1OePz6WAUp3/RFbOlNfILgFXwJIA6AC4B+ArAE0R0UAgxXAhha7yZAuAXsHl4CYA3iegzd4kmJXEKAl35i9u3ZxOjgpds5Eiurnb0qHvXjx07FmMdhvcZhJQUVq0oEixGjNCZvzgmhgMLFPEpLQ1wt5xv586d0blzZ2MHVRmK0r8UFLCWTddhynY8ks2LISE8d3xesEhMZKfFsqK8cnDsGFex0Z2+LiGBNV47dkgdV6dOHByuHcp9FomJHJLbsKFUMqVuu669T9u3sylYImpEOhhfSf9SGV5QodYIAZCIrhDRBCIKJaK2RPRl6fdpRBRm0+4BImpcmv+vKxG97y7NkhKWYUaOdGGOJCTwm+nujq8Tnp6Gn3vuOTz33HPGDcgekpI42Zrk9C9nz7Jvme6FMDSUd3xFAmBBgfvKxiFDhmDIkCHGDqoykpOB3r3ZlCcRmzfzs9AtWPTowWpUBebFkSOBI0f4UOWTIGI+xcdLT/+iydu636f4eNaiSH6fhOB1T0sr6pPIzuYah7ofnvtYv56XV4fpXypj9GhWR23YIHVcNSIdjAHpXzIzM/Hwww+jSZMmCAkJwR133IGfKvmRLFiwAF26tEPTpiEYMKAf0tLSnHesOB1MjRAAvYGdO7msmEvv8qhRvMtJTo7arh2fiH1aa5GayoKWxDQegItmeg2jR7PUKNm8qN2+z/JJm6sKqsKsX89xHbrzWmvmxeRkZeZFn9UuHTwIXLwo3awI8Fzt1InXGF1o1IidBRUI6nfdxbmnJSsb3YfJxJu306gMz6CZ6WNjXVheBwxgraQCPgUFlcdY+CS09cTNw9S1a9cwdOhQEBHWrFmDw4cP44MPPkDTpk3L2ixbtgyzZs3Cs8++iG3b9mDIkCFISEjAGWdF4hWrUP0CoANoEVYu7Y0jRjADFYTo3nUXrzfuxDJER0cjWmJ6DZw/z/bpmBh5NEqxfj0rr3r0cOEibYGWzKfQUFaAuisALl68GIsXLzZ0TBWwdStPIAV8SkriuI7QUBcuiovjU5jZLG1cAM+d5s19WFDXtDaSBYvCQl5TXFZg3XUXn5gl5/qMi+M12WcF9Q0bOIBJcpWWEyc4QNElPgUG8vzZsEG6eVGTq1T7AWZnZ0MIgXfeeQcDBgxASEgIOnfujKTKL3ZJCWut3fT/e+utt9CiRQssWbIEAwcORLt27RAXF4duNrUt3377bTz00AzMmPEYevTohg8++AAtWrTARx/ZTUtcDm1cigTAmpAH0CtISuJUcS7l2gsL41QaGzYA8+ZJGxvAL/+CBbyHy5Tl3ILJxP/GxkolY7HoqNJiDz17sj9VSgrw0EPSxgcwn154gSsxSbayuo7UVF6tR4yQSiYzk+M53njDxQs1wTQ11aO8jc6gmRfXrmWthWS/cNexYQOr5XRFz7iP9HQuZuRy+drYWOBvf+NwfIk1vxs35mmQlAT87/9KI+M+NmzgOasomt4tPn3zDWeq6NTJeXsb2DuIdu/eHQMGDEBxcTGWLl1a9j0RGxf69InEoEGRyM/Px/Lly6tc379/f/To0QM5OTlYWale8YwZM1waHwCYSw+KCxcuxAcffIDWrVvjxRdfxIMPPoiMjAzOq1pasHjeu+9i3ltvVdtfYmIihtuJLPz+++8xevRoTJ48GSaTCS1btsSjjz6Kp556CkIIFBUVYdeuXXj6aXaz0gTikSNHIl2PdTAoqDx5oGT42lLnE8jJAbZtc9OVIz6eT8NXrxo+LlvExPDE8kmthcnEuWp695ZKZs8e4MoVN/gUEMDqBAWnYW1skl1v3IPJxDtqvcppNo2FprFxecNq3pxDqRXU1Bs5kv3j9+yRTso1FBdzlnPJ2j+A15KgIDcOlIMGcTSNIj5t28amYJ/CyZP8UWCmT05mE33Hji5eqB3IJfNJC65VnNIOZrMZgYGBWLlyJWJjY9G5c2e8+eabuHz5Mo4cOcKNSjVrM2fOhNlsrvbT30HA1cmTJ7FgwQK0b98e69evx6xZs/D888/jww8/BMCaSIvFgsaNm1WIM2nWrBku6KnJrEmMCmzofg2gHbgcYWWLuDjg1Vd50Z440eCRlaNePXYATkqSrmx0HampvItIdljXBAu39sa4OGDZMjZVd+1q6LhsERnJysakJGDaNGlkXEduLueVc7GMlzvYsIGfgVvngdhYYNEiPhFL9CfV5lByslRlo+v4+WdOW6RAAExJYVku3F6NpepQuzbb9xUIgHfdxeudyQTcc490cvqhuZNI5pPFwlvL/fe7cXGnThxYlZoK/OEPLl1anUYuODi4yu9FRawFtFqBunXrVnt9/fr13dL4VYbZbMa4ceMqZE6oV/lwW2qXbtSkCRrZ+Oy5AqvViv79++ONUpNGnz59cPz4cXz44Yf44x//WNZOd/qXytA0yAqkZ78G0A5SUjiliO4IK1vceSebghWofOLjgd27WQvmMzh9mh1UFPiVpaay/1azZm5crC3UkvmkKRuTk30sb+OWLXwalmym18q/uVEVixEbywLQ9u2Gj80WzZtzXmOf06hv2MAqBMnvU04OF4pwezrExrKdPyvL0HFVxuDBXJrT5/iUnMzCla6cYe5jzx7mlVvTQQjmk8kkXbvkjXyAZrMZffr0qfBdeno6QkJC0EXjS2n5t3lvvIGwsLBqP46idlu0aIE77rijwnfdunUrC/CIiIhAYGAgLl26WMEb4OLFi2iuxw9IU6Eq0AD6BUA7MJn4QOuWwkELdVQgAMbE8Aa7aZNr191///24360jpA5o/n+SN6zCQpZh3CbTrh1/FATsxMayD6CreRu7d++O7t27yxmUycRzVXKanhMnOFWP23yKjuYFUYF2KS6O/eAKCqST0o8NGzj3n+S8cps28X7jNp80yXHjRqOGZBe1a7PLqoLpoB9abc74eOl55bTl1W2/79hYFtIP6imk5T6EUFsWrqCgAEePHoW1ktA0f/58TJkyBXXr1q1Q/s0TE/DQoUNxtNJifuzYMdx2220AgFq1aqFPn37YuDG5wqE3OTlZf1ovLZQ6L0//Q3AHRPSb//Tu3Zs0XLxIBBDNm0fu4+23uZMzZzzoxDkKCohCQoieftr4vrOysty7cPp0oiZNiCwWYwdUCZs28SNescKDTh57jKh+faLiYt2XuPNcjh3jsS5Y4PKl8jBgANGwYYZ15+i5/Oc/fO+HD3vQeb9+RFFRHnSgDz/+yGM1mYzr0+33iIgoJ4coMJDohReMG5ADzJ5NVLs20c2bbnZQXEwUHk40c6bTph49EyJ6803m0/nzHnVjHHbt4gF9/rlH3eh5LgkJRN26eUDk9Gke67vv2v159+7dHnReEfn5RNevE1mtnvVj0bGX7NixgwIDA6lz5860adMmOnLkCE2bNo1atGhBmZmZ3KioiN+pkhKPxrN9+3YKCgqiuXPn0vHjx2n58uVUr149+ve//13WZtGiryk4OJg++eQTOnToED399NMUGhpKp0+f1kekuJh2b91KtH693Z8B7CQDZB+/BrAStAOsRwosRWlGNNcb7VSoF/n5+cjPzzd+QJq9LzpaeihlaiqfMHXnlbOH+Hi2p+zebdi47KFjR7YOucqn4uJiFBcXGz8gzd6nwExvMrF51SPLWGwsh7vLmLM2GDGCp62rfJKGTZvYhqYgsMBkYmVwSIibHQQF8cuoQDWnTVvJykb90Kw9kvNpFhfzlPDotb3tNqBDByV8CgxUl9PYbDajU6dOeO211/DAAw+gT58+uHr1KtLS0srNrgaVfxswYAC+//57LF++HD169MBf//pXvP7663jyyScB8P3+7neT8a9/vYu5c+ciMjISmzdvxtq1a8u0hE6hOQ9K5pPuJyGEeFQIQTafAiHEASHEwzIHqBomE7vweeQI3qMH0LSpMjPw/v2uud7cfffduFtGUflffmF7n2S/MoD5FBnJeWjdhraSKsiOHxPDG5YrfoBLly6tkF7BMGj2PkX+f7GxHlrGYmPZq1xygvX69YG+fX1IAExJYYnMLWdk/bh8mVMtenweiI3lWnJnzxoyLkfo04eD4HyGT6mpHK3eooVUMjt3skXQED799JN0Bz2VOY3NZjN69uyJKVOm4OzZs8jPz8fq1avRoUMHbqCVfwsMNMRMP2bMGOzduxcFBQU4duwYnn76aYjSfrX7feqpJ3H69GkUFhZi165dGOFKui1NUPUVARBAHwAFAAaXfiYCuA5gsRBCvipBEUwm1gQEB3vQiZZBOiVFuue/thhUqkLjHSjy/7t5kxVCHpNp0oSlSEWCugLXG30wmVh9PGiQVDJHjnABC4/5NGwY7yaKtEvbtklXNuqDyQQMGeKBWk4ftLXDEMECkC6ZBQXxGu0TAmBxMdc5VBT0BhiQ9zU2lq0AknMeaTmNVQSCmM1m9OrVy3EDNnxLz9EI8P0aoGjkDnbt4mT4kuDKECMBHCKibaWfRACPlP4mQZ2kHoYWsHDX899F9O/P1RV8wik6NZVPwTZh+DKwdSsrhAzjkwLPf22sPrFppaYqESy0Oekxn8LCOLpekQBYXCxd2egcV65wVK0iM31oqAEFLHr25GzNivikGRy8Ck0tpyAbv8nEkeoRER52ZJtgXTK0usAy9SBEhH379lUvAGpqOcmpyQxVNArBlhpXozxdgC4BULBusxeA/ZV+0tJx1jVyUN6CoQosRTt+cDAwfLgPCBZEbOP02N7nHCaTgQUsYmI4pHjbNgM6c4x27YDbb/cBPl2+DOzdq8xM37Yt0L69AZ3FxnIRWMnlxoYN47nldT799BO/UwoEi9RUD7Ie2CIggN+n1FRllg+v80kbgEfOyM6hZT0w5LVt1gzo3l2JAKgp3GT6AQohcP36dYwfP95xI8PUctXDajVQ0RgQwId0iZNc79PoBCAMwL5K32uzfqdhI/IitAIWkZEGdNa+Pe9+ik7DR45wyS2v4fBhg+x9zmFoAYsRI3jHV8Snn37ycpF0w+x91cNq5fNATIxB54HYWOmnYaC8lKtPCBZ16gADB0olc/EicOiQgdMhNhY4c4arYkhE796cGcfrfNq4kX2+mzSRSmbbNjZSGMqntDTpJcc0hZuqdDB2YbD/X3XQzN2GKRolJ1jXKwBqItEhIUSQEKKhEOJ3AN4BcATAV1JGpxia/58hzLP1/Je847saFTdjxgxDMq9XgDZJJWuWtAIWhi2E9eqxNKlgJ4mJKbfs6UFkZCQiDTmN2CA11SB7X/U4cICVjYbxSWG5sZgYVjbm5kon5RgbN3JYrsTqJxoZwGDBApDvvB7ASjevCoBFRR4mI9UPk4nv2bCy3bGx7OgqOcG6Sj9Ah1Do/2dQoHE5JCdYd1UATARQDOAKgK8BbAQQQ0QFACCE+EoI4bYuXAhRTwixSQgRKITYIoS43d2+XIV2aDX0XY6JAbKzeTeUiD59OIJR72IoRQA0mTjFQLt2xvZbCVoBC0P5FBtbXnJLIlw1W0kRAD3Kcq4fhvn/aQgJYYFIkQBYUsK+/V5BVhaH9isKLKhXj6OfDUHnzkDLlsr4dPo0f7yCHTtYiFIkAPbtyxYqQxAVpSzBugo/wGqh0P/PYkGF+r8eQ3KCdb0CYB8AZwEMANAfQHcA9YloMhFdAAAhRF8ALYnI7XhUIrpORCOIyAJgPoDX3O3LVWibsqEKLEWOKq5GxWVnZyM7O9u4Adja+yTDZOL7HTbMwE41z/8tWwzstCpat+acgHr5ZHi+xgsX2N6nyP+vQwf2gjAMcXFKyo0NHcq+tV7TLmlmbkWBBSNGGKgc0cqN/Rb8AE0mvl/D1HL2kZ9vUNYDWzRsyBKlQj9Ar2kBFfv/GSpn9uvHfimS+OSKBnAnEe0kol1EdIiIblZq8wcAX9p+IYR4UQiRLIRIE0IcEkKkCyEcFsMTQvxNCPG30j9XAbhbCFFf7814ApOJA9h69DCw07ZteRdUZF7UGxU3adIkTJo0yTji+/axbVORYHHnnWzFNAzajq9Ia6Hl93WG5cuXY/ny5cYRN9zeZx8WC7saGk5GUbmxunWVBR3bh2FhudXj7Fng+HFJfLp0SXrOo+7dOSLWawmhtbDcxo2lkklP5/OpFD4pSLCuCUReEQC94P9nqKVZ0+54SwAUQjQD0ByAs6RBcQAqJ08YACAUwHgiugPAGQBPVtNHP5QGlBBRMTjqeLizMRoBaQUsYmN5hZI8+716GlaU/y8nh7MuGE4mNJR3fEWCuoIUXPaRmsq+ApUKphsNs9mDgvXVQfJp2BYxMVwgRnLQsX1s3Mgqbo+SkTqHFKuHbYcK/ACjo/k+lJsXCwtZMlMUpW241QNQlmBd8wP0SiCIoWG51cNw/z8NEhOs6xmqtls427LaALhQ6bsBAGYR0dXSv80Aqsti1A/ALpu/LwBorWOMHqGwkH0Apcgv2o5vNkvovBy9enFVDK8IgKmpQKdObOOUiLQ0DwvWV4eYGJYuJe/42n7hNUHdUHuffRju/6dBcbkxBUHHVaFpzhSZfxs14rXDUNx2G2dBUMSnjAzpQcdV8fPPBoflOobJxMrg8HCDO1aYYD0wkHUgygV1w8Ny7UNTNEpZWiUmWNcjAGpe6M4EwHwAdbQ/hBAtwJpD20KrA+EgZYwQojUAIqJzNl+HAKhsajYcubn8GKS8y9pCfqtGxZWUGFCgUh+0AhZSKmNpaUbS0iR0Xo4WLYCuXb3Ap4wM9hFQxKeuXSVVxlJUbmzwYJ5ryvmkKE0PwPcWFSXJNepWt3xs3KjE/+/GDY41kTIdFCZY95ofoDS1XEVoiT6kCICadkcCn5w+FSL6BxEJInK24u4D0NXm7wEAAgF0AQAhxFgA3VDqJyiEeEMI8Ueb9pW1fyhtv9fZGD1Fbq5As2ZAt24SOm/RgjtWsELFxnJE3KlT0kmVY/du4Pp1ZRvW4MGcHs1wDBqkbMePiWE5s7hYOqlySLP3VURxMd+btOmgyLyoleBVLlhoxcgNC8u1j1OneK2QNh0UlRvr2hVo3txLfIqM5GAKiUhLY6FJKp8UJFj3ih+glLBc+5AaaCwxwbqRYvE3ABJs/h4AYCGAhUKIg2Dfv1FayhgAvVHRZFxBABRCtCsdnxIB0LCEtfagaMfXexp+4okn8MQTTxhDVJH/35UrBhWsdwTFaUZyc9niXB369++P/v37G0NUi3Lq2dOY/hxg1y6+N2l80sqNKRLU9+7luacMJhOX9lHk/yeNT4rKjQnhBT/AggIJYbn2YTJxxqYhQyQR0HwdJFs+vOIHKCUs1z4slvJ7lAJJCdaNHO5iAHcJITRPhQEAVhLRECLqTkR3E9EZABBCBAJoAmCFdjER/S8RvWLT35MA3iKS/1obnleuMmJj9e34HuKOO4CmTZ3vjZMnT8bkyZONIZqayoSbNTOmPwfYtInfZal80nb8y5clEtHvB9ijRw/0MCIsnYj5JCXKqSIMK1jvCJqvg4LQz5gYfnSaVVY6Llzgkj6KBIumTfnVlYLmzdnyoYhPmZnsGaAE27ax47giP01pVg9Aqa9DUJBiP0ApYbnAhx9+iF69eqFevXqoV68eBg8ejDVr1lSQM1999VUIISp8mjd3mADFOVyt9qAThu0GRJQL4E8AOpR+1R8O/P2IyEJEA4jIbomM0trDGQA+Lf27kRBipRAiTwjxqxDiQUfXCSHeFEJcLv28WdqXU0hdc7U6kZJfMr2n4YyMDGRkZHhOsKiIs+UqSP+SmqqgMpaiHb9JE1ZkOZsOOTk5yDHCNHPqlMQop4owmfjepFbGio5WkgF44ECec8rMi9riLlmwIOJ7kmr1APg+fMjyYRgML8thH1evsoeN1NdWgq/DW29V7c5kAt55h/+vzAys+f8ZPMlbt26NN998E7t378bOnTsRHR2LBx+cgEOHKpZ46tKlCzIzM8s++/fvd59o1658YvNVARAAiMhERObS/0cQ0SU3+yEiet9G+/chgCIAzQBMBfCREKK7nUsfBzABbF7uBWAcOD9htQgO5gS90hARwY6cisyL586xv78jTJ8+HdOnT/ecmOJM+MOG8WFVGgYM4JQwisyLW7awIsERVq5ciZUrV3pOTFGZPmWVsRTt+LVrs1eAUgGwXj3paXqOH+c1QgmfcnNZipGIjh2BVq0U86lvX06pJBFKrB4AEzCbWeI0AAMGAPffX84Pk4n/vvNO/luWAJidnQ0hBN555x0MGDAAIY0aoXPfvkhKTjaUzj333IOEhAR07NgRnTt3xquv/h1hYeHYvn1rhXZBQUFo3rx52aeJJ6diTbuzcaOhKlS59iADIIQIBXAvgJeJKJeINgP4DD4ehQAAIABJREFUEYA9CeZhAPOJ6GxpNPF8ADOc0QgLI9k+orz5OtvxDYDS03BqKk9MTcMpCVlZXE1P+kJYqxZLmYoEwJs3pZfiZJhMbJLr2tV5Ww/w8898T9L5pDADcEwMzz3JxUcYmv+f5DQ9itx2lVo+tLLr0s2LN2+yCViR+TckpFxwkoboaH5wBuU8iokBli9noe9//5f/Xb6ct0CZfoDm0lRrCxcuxJvz5mFfejp69eyJBx98EDdvVkwmMm/ePISFhVX7SdPhF2mxWPDVV18jLy8Xw4ZVdNQ8efIkWrZsiXbt2mHKlCk46an/XnQ0Zz8w0A9QfnZEz9EZQAkR2Xp47AVgT+rojopBI3tLv6sCIcTjYI0h6tdvi6VLlxozWgdoJQSiCwqQPHcuLknciImAhg0nYtGiSwgNtV/a7OLFiwBQ7T3n5eUh1Em5jbivv0attm2RuG6d+wPWgZ9/bgtgOIqK1mHpUrn+eXc0bIg+69fjuwULUGDnhK/nuehBXl4tCDEJ7723D2fO2K8VvXUrnyiDPQkIIMLv1qzBxW7dsOXLL523dxN5eXlISroTQvTCpUvfYunSImm0AGBY+/aIWLMG33/xhVQ7ZlFRBIBReP31NNx55xmXr9c7X+pcvYrfHTuGXf3744jktWjx4qFo2LAptm9fiR07pJLCmNatkf/llzC1aVP2nVHvkC3q1GmPS5cG4623VqN1a3kRrc0OHEB8URFMAM4bzKfKz2XlyrvRoUMBvv1Wcgqx4mLcFxyM4x9/DOvcuSg2wGQ/bBjw+OMBeP31QLz4ogXDhllRXAwEBASgpCQARUUlul9bIoIeT65du3YhMDAQy5cvR9f27RFYXIy5c+eiW/fu2L9/P/rYaNYfeeQRTJw4sdr+WrVq5fBZ7N+/HyNGjEBBQQHCwsLw1VffomvXrmXt+/Xrh4ULF6JLly7IysrCG2+8gSFDhsBsNqOxzsoxVqu1wh5dLzcX4wBs+8c/dF2vC0Tk0x9wJZALlb57DMBGO20tALra/N0JAAEQ1dHo3bs3ScfVq0QBAUSvvCKd1NSpRM2aEVmt9n+PioqiqKioavvIysqqnsjNm0S1axPNnu3eIF3AE08QhYURFRVJJ0W0fTsRQPT113Z/dvpcXECfPkTR0Y5/X7RoES1atMgzIocP8/3897+e9eMEWVlZFB1N1LevVDLl+PBDvq8TJ6SSKSriuffEE+5dr3u+LF3K97Nzp3uEdMJq5bVh6lSpZMrx1FNEdetWeHmNfIc0nDzJj++DDwzvuiJeeokoMJAoJ8fwrm2fS1YW38/cuYaTsY/YWKLevWn37t2GdJeaShQRQfTyy/xvaip/X1TEj664WH9fFotFV7upU6fShAkT+I+8PKLr1+nSpUsEwLD70lBYWEjHjx+nbdt20uzZz1Pjxo1p//79DtvfuHGDmjRpQvPnz9dNo8qYtZd32jQCl+b1WL7yeRMwgFwA9Sp9Vw/ADR1t6wHIJVKef7wqGjRgvxFF5sWLF4HDhyUS2bqVzdmK6v8qyIzB6NOHfXsU+Wtu3cpWJWlQZO+7eZMrSilwB2UoKqkSHMxzT/prazLxGhEZ6bytBzh8mNcGZXyKiWE/YcmqxnbtgNtvV8Snfv3YV1MiFOYDLye015iMa5rP3/LlwN/+Vm4ONpnKvRtkmIHNZjNr+Wzy/6WnpyMkJARdunSp0NZTE3CtWrXQsWNHREb2w6uvvoHIyEi8o0W52EFYWBi6d++O48ePu3+DtlGeBqEmCIDHAAQJITrZfNcbgL1K4wdLf3PWzjvQdnzJxbed+QHOmTMHc+bM8YyIFgk3XG6p5sxMZZkxGFrxbUWJuwsLeUrYw+DBgzHY07InqalAmzZAhw7O23qAnTuDUVSkkE/dukmJirOHmBieg5mZEols3MjzTnLOMmX+fxo0P0BFfNq4sbwqg+HIy2OnXUVBb6GhHFChBNo9GfDwduxgoU/rUvMJ3LGDZRitLJyRKCgowNGjR2G1Wivk/5s/fz6mTJmCunXrVmg/c+ZMmM3maj96crCWlPA2aLVaUViNf39BQQGOHDmCFp6WR4qO5gguo2CEGlH2B8DXAL4CEApgKIAcAN3ttJsJ4DCAVgBagoW/mc76V2ICJiJKTGS9fnKyVDJWK1HbtkT33ut+H07NNEOHEg0c6D4BnVBkGauIt99mohkZVX4y0nyVk8PWpJdeMqzLirBY2P7y0EOSCJTj2WfzZFnGHOP++4latXLs62AQdu7k6bB0qevX6povGRlM4O23XSfgIu69l9cGyY+sInr2JIqPL/tThgmYiOjzz/kx7tkjpXuipCQmkJgopXvb53LHHUSjRkkhYx+FhUR169Lu7dulk7p5k9cJvXNQjwl4x44dFBgYSJ07d6ZNGzbQkZ07adrUqdSiRQvKzMz0cMQV8f/+3/+jTZs20cmTpyg9fR8999zzJISgtWvXlrWZM2cObdy4kU6ePEnbtm2jMWPGUHh4OJ0+fVo3Hbtm61KXHvyGTMAAJ4WuA+ASWBB8gogOCiGGCyFybdr9B8AqAPsBHACwpvQ734BWfFtBVJxWitPege7o0aM4evSo+wTy8jjkU9FJWIFlrCIkFt+2Rb16bE1yRCY7OxvZ2dnuEzhwAMjOVmKm37w5WIVlrCK0nEcnTkglExnJc1DadFCUpsdq5TVBev6/ytByHhXJDQySXnwkNZXXb8lWj4sXgUOHFGppAc6AMHSoRPVpOWTUBTabzejUqRNee+01PPDQQ+gzfDiuXruGtLQ0zxIw28GFCxcwbdo0dO3aBePHx2H37h1ITExEQkJ5IbSzZ8/igQceQJcuXfC73/0OtWvXxrZt23Dbbbd5RrxLF0OLLtSEKGAQ0RVwfr/K36cBCLP5mwD8pfTjewgL4+yyivzLFi8G9u8Heveu+Nsf/sCpETe6a5bZvFlB+RSGyaTEMlYRtuXGjMiXWA1iYoD581mmrhwYuXr1agDAjBkz3Otckb0vLw/YsycIzz4rlUxV2PoBSkzkGRgo2SsgNVVJmb4DB7jIjVLBAmA+vf8+m0+HDZNGplUroFMn5pOUuZiayjXDDY5grgxtWfYKn4hYCJRYMUhby0tKjMt4ZDab0bNnT0yZPBlT7r6bnXcllU9ZvHgxAHbfKSwEwsOrHqi+/vprKbTL/ACXLTOku5qiAbx1EBPDzhA37MWwGEsGkLRpad68EhdzAMjIYOWO8oUwIIBfMgWCemwsL4SbN0voPDWVff/atpXQeTm2bAGKi4V6PnXpwvkNFflrnjjBBVUMBZWW6YuJkV6mT7n/n4aoKN64FPFp0yYJQQbXrnEpT0VBb+HhHDOoFNrEkFyqQ/MDNJJHZrMZvXr1KtdgSs6lCZT7/ynVpgOG5qD0C4CqERPDL5jk4tua37+UNTc1lbOTSj4Je23D0oj++iuXUZOIoUP5sGo4nywWDiVUpKUNCiIMHSqdVEVIyo5vD9IOVCdO8EknLs7gjqvCZALat5d+HqiKRo24EpKiQJDr1yUUH9m0iYULRQLgiBFKZJiK0IIeZGVqtkFQUHmshqcgIuzbt48FQG3skk1GNoHG6vHoo4Z15RcAVWPIEPa3UJQO5qefDD7Q5eQAu3YpWwgVWMbsQ7s/yVrA0FD2CjB8OuzZw7xSJAD26VOCsDDnbQ1HTAyH5x475rytB+jRo9wrwFAo8v9TeB6wj5gYzhMkuRKStOxAqalclmPQIIM7rojz53kqe4VPwcGs0lJQrNfWDOwphBC4fv06xo8fX66Wk6xN1x6RUtckDQZKnX4BUDXq1DG8+LYjxMayDFBaIccYpKXx0U2RYBEVJf1dto+uXZWaF3fuZF4ZBkXq0xs3eOzDhnlePcAtaPcnWbsUEMCkUlMNVjamppY7r0nE3r1sxfSqAFhQwMFjEtGsGVcKlCIASi9G7mWrB6DlNJEeDKIJTobKmgrVcprg6hUNoIHwC4DeQGws2ygMKr7tCNppuLIS66WXXsJLL73kXqepqbwIepqfzglOnWILrNcWQs28aPiOXxUxMbzeVvYKGDFiBEaMGOFepyYTC7Ge5p1ygrQ0XnO9JgB27Ai0bKlMo56RYWApTquV51dsrHRHIq8LFsOHK/MDjInheWlY0PGlSxxNp8jq0aBB1cA9ZdBO2zXQD1ClWs5iYTLK/f8Mhl8A9AZiYgwtvu0ILVqwDFB5zY2Pj0d8fLx7nZpMbMYOCfF8gE7IAErWXMeIjVViXhw8mGXqynxq37492rdv73qHxcU8txRtWLVqAQMGeEkAFKI8A3BN8wM8eBDIylLGp86dWVb2Cho25Co7ijTqhhYf0bTLivgUFeUl0yJQLtEo9AM0TNmoCYCS1XKaotFrPDIQfgHQGxg4kE3BCk/DtjWttUznLuPyZbYnK1AjpKayOadbN+mkHENqKHU5QkLsewVcuHABFy5ccL3DnTs5N4siM/2gQdIyLuhDdDQnTztyRCoZw70CNNW8ZD6VlPB5wGvaPw3R0cC2bWwKlggt6Ngw993UVA7L7dfPoA7t4+zZAJw86QN8CgpS6gdoGClFYbmK5Ewl8AuA3kDt2uxPoigfYG4ux21oeOaZZ/DMM8+43plWoFLySZiIN9noaC+r2Dt04HBqRelgzGbgypXy79atW4d169a53pk2XgPTBdjDtWsca+JVLS2gzA9QUzYa5hWgpenxNDmsE+zezb6aXhcsYmKAwkIE79wplUyjRmxCNVRQj4qSvuNv3szFzr3Op8BApX6AhigbveD/59cA+uE+YmLYryQrSyoZQ6PiTCagbl3pBSqPH+doOK8vhLbmRcmLoeYVoMnYHsFk4rQbEREGdOYYWmYMr/OpfXugdWtlGvULFwBPCukA4F1k40ZlZkVA+nnAOYYPBwICELxli3RSsbEcdOyxsjEjgxckBXzasiUYjRtzxLlXIaNUhx0IYaCyUaFa7lbx/wP8AqD3oC0okrUWTZpwGhVDlFgpKbyI16plQGeO4XWHdVvExrKQfvCgVDIDB7Js7bEMc/MmZ5VWkFdOy4xx553SSVUPhX6AhlUJ3LOHE9YpEgDvuMPQClLuoX59oG9fBEvJel4RpcpGbN3qYUcKnZG3bAlGdLSXsh7YQhuAAj9Aw5SNivP/3QraP8AvAHoP/fqxX4kircWWLR6m4Dp3Djh8GHA3eMQFmEzsrC45M4Y+KPID1EpxekxGy7WmKLHw0KHSM2PoQ0wMC+oHDkgl0769QV4Bivz/iov5POAThykAiIlB0K5d7KMqEVr5SI/fJ0Vl+k6dAjIyAn2DT4aq5qqHYcpGRWq5WyX9iwa/AOgtaEXFFQmAN29yKU63oW1YkgULIi8VrHeEtm1511fkB3jgAGedcBsbNvDccjd9jE5kZwP79vmQYKHNyw0bpJIxzCsgNZUT1klWy+3YoSweSB/i4yE0qVQi6tXjM7ZHr+1voUyfIyjyAzRE2ahQLefVBNAS4BcAvYnYWI5cPH9eKpnKpTjnzZuHefPmudbJhg18EpacoOrwYQ7o9JmFEGA+GV5SpSoqxzLExcUhzlWBOyWFbbLh4YaOrTI0X0Wf4VPbtpznRLIACPB0yM72wCugqIhD8xX6/0VFSSelD8OGgWrVApKTpZOKieFDr9vKRq1MnyI+NWli9W7WA1toKi7JZmBN2egRGRu13FNPPYWJEycaMjZHpG4V/z/ALwB6F4qiFyun4BoyZAiGDBmivwMi3ljj4pSdhL0eWWqLmBgOed27VyqZyl4Bbdq0QZs2bfR3cPUqp4BRZKYPDZUeD+Qa4uNZMjUsA7B9aK+t29qln39mlbwiM72CeCD9qFsXxQMHKhPUi4vZ/cUtKCrTpykahw4t9h3BQkunosgPkMgDZaONWm7u3Ln4/PPPDRvbpk2bMH78eLRq1QpCCHz++eIq5t8FCxagXbt2CAkJQb9+/ZBWOaO/zjbegF8A9CZ692bpTFE6mPR03nfS09ORnp6u/+KjR1lLqWDDSknhrBjt2kknpR8e7/j6oFluNQEwIyMDGRkZ+jvQgiAU8GnDBvZgCA6WTko/4uNZ3SO53JjmFeC290ZqKm+wktVy+flsaVVwHnAJxSNG8GHKI18H5xg6lOen269taio7I3fubOi4KuPQIV5eo6LkHlxcgq0foOTAKo+VjTZquYYNGyLMwKLkubm56NGjB9577z3UKU12amv+XbZsGWbNmoUXX3wRe/bswZAhQ5CQkIAzZ8641MZb8AuA3kRgIG8CCvwA4+PLLU8vvvgiXnzxRf0Xa6d1yTtJSQkLgHfdJZWM63BUUkUCYmNZ3s7IAFJSUpCSkqL/4g0bWC0nOSz3zBke48iRUsm4Ds1XS5F2aeNGNzetlBSgb1+u+SURaWkcD+RrfCrWBF/JB6rQUI6ud4uMppZTUKZPs4ZHR3upmo4jeKya0wdN2eiWh43mpxgUhLNnz0IIgSMGJoS/++67MW/ePEyaNAkBpdYvWwHw7bffxowZM/DYY4+hW7du+OCDD9CiRQt89NFHLrXxFm6RWJYajJgY4PvvufCtxISwI0ZwpGlSkhsXp6QAt9/Oag+J2LGDM2P42oYFgDeCJUsqllSRAO3ek5PdsLanpJQzWiK0Dcvn+NSgAdukN2wAXntNKqm77gIWLuQ561JZ7Lw8roYxe7a0sWlITuapMHy4dFIuoaR3b+bVhg3AlClSacXHA6+/zgnWGzVy4UKFZfqSkljJ2Lq1XEHLZZSq5p55hmDeL5eU1cqypiZcRUYC776r40Ib8+/evXtRt25ddK6ksdXj856YmIjh1bwomhLUttBIUVERdu3aheeee65C25EjR5ZZ2PS08Sb8GkBvw7DEYtWjbl3eCFwWAEtKeGwK7EhJSfxyKbBguo7SkipBkv0Au3dnhaPLfDp7ltVyivjUsiXnlvM5xMezCfj6dalk4uJ4rrrMpy1b+BChSLAYNozffZ9CYCDff3KydPPiqFEsXLiiSAegzP+vsJDdVn3uMAWwtBMQAFjl8ggoF6pcng42+f/MZjN69uxZpqnTMHPmzLLyp44+/fv3r5aMNi5b7V92djYsFguaVYrkb9asWVkJTz1tvAm/BtDb6N6dszWbTMCMGVJJ3XUX8PzzwKBBjVC79hXnFwBcQy4nR5lgMWCAiyd1VSgtoxCclgaMHi2NjBC8GaxaBYwcKRAQoHNF1HY4ydKzxcKKm3HjfDQSLj4e+PvfeVcdN04amcaNea4mJQGvvOLChamprFkZNkza2AAgM5MLDf3jH1LJuI/4eGDFCo607dhRGpkBAzj/9Pr1wH33uXBhSgo7Iksu07d1K/tq+pzbi4agILz7xk0gPEjqC2+1csnS2rVdzCuqlX8TAnv37kVkZGSVJo0aNUIjDzcVTc70epJug3GL3U4NhOEFRh1DO2VevVr9aacCNH8qyfk+rl1jxY3PLoQREUCvXkrKWI0cySar06ddWLSSk8vLvkjEnj08Np/l0+DBQJ06SvwAR47kOXvtmgsXJSXxGENDpY0LKL99n+WTdqCUnA4mKIhJJSW5sLwWFfF6PGqU1LEBPK7AQB8o0+cImspLcgosTdnokk+tjf8fAJjNZrsC4Lx58xAWFlbtx1lUrjYuWxk4IiICgYGBuHjxYoW2Fy9eRPPmzXW38Sb8AqAvID6eTXiHD0sl07s3ywgdOszEu7ocLACsW8cO602bSh2bycRrjE+aQjTExiJYS+EhEdreWFQUjdF6tI1WK6s4Ro2SfkTVTJ6+Fllahtq12Q/SLWdX1zByJM9Z3UEGFy6wBJ2QIHVcAMtVERHsS+WT6NiRw6kV5AMcOZKDqnTHBqSnszpKoqZfQ3IyMGgQJ672SSjKB6iRcino2Mb8m5eXhxMnTtgVAD01ARPZFwBr1aqFfv36IbnSHE5OTi5Ls6anjTfhFwB9AdpCk5golUxAAGsEdu5shF69dOwM166xjULRhhUWxouhz2L0aIiCAul5G5s25byNW7eG6zsl7trFmYkVbViRkT5QV7Y6jBrFu/2vv0olM2gQ521cv17nBVpDyXwiYj7Fx/uwyUrzdUhJURZYpftMsG4dSyOS/f8uX+ZX16cPvUKwFlCRAAi4oGwsKeHxBQRg3759AIBevXpVadaoUSN07Nix2s//Z+/L4+Soqv2/1d2zJZOEhC2sYd93DLIvomBAEIJsIstTQRQUH6KigsiDx0dRn8jvIYogCAFkkS1IolH2J1skQYgRZE1IQpwsJJlJMjM9fX9/3Lkzp0+fc+tWT9ekJ6nz+dSnlq6ue+ue7XvOXcot8cKpvb0df/vbTPz97zNRKpUwZ84czJw5s28Jl4svvhi33XYbbr75ZsyePRsXXXQR5s+fj/PPP7/vGSH3rCmqV/OwbtEWW9ixgCkDQMCCrH//G8jnDbbaCrjzTs/N06ZZbUzJYV17bf/clz/9qf+bxddem0pxA6fDDoNpaRkUPm2+OfD00wa5XACfpkzpd6gpUnu75U/ddis6cgFLynxqaLAY4Y9/DMxaTJ0KjB2belrutddssrGugQUAHHOMnayT8mzIrbays2yDgfrUqXaMZspf0/nLX6zc1L0+FQqD8lk419schDVdWo6M/9t+++0xrMYznqZPn47x4/fGIYfsjVWrVuGKK67A3nvvje9///sAgFNPPRXXXXcdrr76auy111549tln8dhjj2EcGTsacs8aI2NM3W4AxgB4EEAHgPcAfNZz7w8AdANoJ9s2IeXsueeeZo3TJZcY09hozIoVqRUxaZIxLS3GWO2x27Bh9jqntrY2Yz7/eWPWW8+Y7u5U6vP448ZssIExd9xh63Lhhfb88cdTKa4m1Pnxjxuz3XapljFpkjFNTWF8MsYYc8ABxowfn2qdjDHm0UdtXaZNq/ytra0t9fKDqVQyZuutjTnuuNSLuuEG2yZvvCH/3tcuxaIxY8YYc/bZqdfpJz+xdZo7N/WiqqK+Nlm2zJhCwZhvfzv1Mi+80Nq+1atjbpw3zzbeD3+Yep2+8AVjRo3qN6/1oEMvv/xy5cVi0fKqszP18tvbK11gT09P5Y3d3bZOXV1rpE6DQSIvegnAdFMDjFXvGcAbAHQB2BjAGQBujKJoV8/99xhjWsn29qDUshY0YUL/4OOU6Hvfqxy+tnKlvV5BxthI+BOfQMW3b2pERxwB3Hsv4DLhkybZ87r5vqxAXUceCbz5pt1Sou99zy4PQUnl05IldibCIHTTT51q51ekPIF14BRFtj0ef7yyIWtMLssWm1166SXLq0Hopn/sMbtEz+abp17UwGjkSLs21WOPpV7U0Udb2/fsszE3un7ilPlUKtnXPuqo1Mxr7WgQPwsXnGwk3/9Nk0ol2wlWV188qiHVLQCMomg4gJMAXG6MaTfGPAvgEQBnrtmapUQHH2xnBqbYbaV9eUa6np81y36fKGVDeMQRdt07ALjwwvoGfwDQ5WY/1AmfMG2atVIpA0BjgEcftePKmptTLao2NGGCXXQ55W9ubredXR996tSYG6dO7R+EmyItWwY8/TTwqU+lWkztaMIEu17N+++nWszhh1snHgvUXTe9MJasljRjhl2q59hjUy2mNuQ+C1cs1s9n4cjn39KkQcKZa4zqFgAC2AFA0RjzBrn2CgBfBvC4KIqWRFE0K4qiL6dbvRpTY6Ndw23KlNSUbMstw683ukxkygDwscdsMm2//YBf/nJQvrY2ICq5AUUpAsAkfMKUKXbhxPHjU6sPYL9X+u67QwhYHHGEnRE8COM1jznGjudaudJz09SpVsjXXz/VukybZp3WkOHTMcfYfcp8am21ycY//MFzU7FoM4Cf/GTqwOIPf+hPVA8JSjxDozoKSjay5V/SJDLPZK2kesa1rQD4cv7LAGgjc+8FcBOAhQA+CuD3URR9aIy5W7o5iqLzAJwHAJtuuikWLVpUk0oPhJoPOQStjzyCpc89h54UPkB+6aWNuPjiEVi1qt+4tbQYXHrpCixaVP4h8uF//COKu+6KDxsb7QzTFOjZZxtw1lkjAOTwrW8tQz4PnHzyCNx88wocfHCdfRezl5YtW4bhhx+O5ttvx+K5c22faI0pmE+lEsY89hi6DzsMK5YurXk9KN1zTwuA4TjggCVYtKiyf2bZsmWpll8NjTzgAOQmT8aH3/lOquUcemgD/vd/R+HBB5fj6KPL9WjZsmWIlizBmBdfxMpvfhOrUrYz99/fivXWa8T22y9JS20HTGWystFGGL3ZZig+/DBWnHhiquV+7GPNuOyyVrz00hJsvXWlDBdeegnrLV2K5QcdhK6UG++hh0Zh332BXG5ZH5/qRYdKUv9rPo8IAIpFmJTRUKEQobsbKJEvkNA6RcUiIgAmn4dJeWJKsRihUHBzJVItSqTUcUktBhJWswF4EoBRtmcB7A1gJfvPNwBMDnz+pQB+H3JvXUwCMcaYd9+1A5B/+tPUipg0yZhNN+0yQMnkcsbcfrtw07JlplQoGHPppanVwxhjfvQjY4491pgRI/rHFz/+uL1er9TW1mbM1KmWT1OmpFbOpEnGbL550QAl09SkTAB5+WVbj9tuS60ejg4+2Ji999Z/r4cB7BX0s5/Z9nnnnVSLWb3ayvC551b+1tbWZsxdd9l6vPBCqvUoFo3ZcENjPvvZVIsZMFXIype+ZExra+qTDN56y7LhZz9TbrjsMmNyOWMWL061Hh98YOtx1VXl1+tBh3wTDwZrNkRXl53f4SbHVEwC6egwZvlyO9krRRrEeSYirdWTQIwxhxtjImU7GMAbAApRFG1P/rYngFmhRQCox49V6TRuHLDzzqkOij7jDGDevAbcdVeEUgnYdlvhpj/9CVGxmHr37ze/Cbz8sh2g3dhorx1xBPCtb6Va7MDpsMNs5i9lPs2dm8fll9toWGTF5Mm2fyLlLxYsXmxX6hgy3YqOBql7sanJsuDRR5XB6w8/bBd33Hffs4ObAAAgAElEQVTfVOvx0ktAW9sQ5NOECf1rDKVI22xjV9uaPFm54eGH7VjslL9F6czGkOPTIC0H43p2xeUhTfnyL2nS2j7+D6jjMYDGmA4ADwD4ryiKhkdRdBCATwO4Q7o/iqJPR1E0OrK0H4CvAXh48GpcIzruOPsd0xS79CZPngxgKvJ567Qq6KGHUBozBjjooNTqAPQPhB5yhrC52Y7XfOSRVAdFv/7669hjj/f6ZgxW0EMP2c+KpfxJoalTrc0fcnzafnsb4TzySOpFHXecleWXX2Y/dHZa5h1/fPmX5FOgRx+1RQzCF8xqS0ceaSNA0RjVlo47zk6Sqfh831tv2ckoJ5yQeh3+8Adgs83sl5mGFA3SV0G8c05c2YMwLXeQ5pmsUapbANhLXwHQAuDfAO4G8GVjzCwAiKLokCiK2sm9pwF4E8AKALcD+JEx5reDXN+B08SJVvJSNIY//elP8atf/RCHHCJEw11dwKOPomsQ1idwCSyXqBlSNHGi/dLEjBmpFfHcc89h+fInMHaswKd337VlpzxuCrCiuNFGgOdrSfVJUWTb5y9/sdNjU6RjjrEDxTmfGp59FlixYlD49Ic/2Jgt5QRW7am11YLABx9MfZbp8cdb81oxa/vh3lzBpz+davldXXaeybHHDkFgMcjLwRgjJBvJ59/SpEGcZ7JGqa4BoDFmiTHmBGPMcGPMlsaYu8hvzxhjWsn56caY9Y1d/28nY8z1a6bWA6Tx44FNN7XGMGU67jj71YB33yUXn3wSWLYMXYOwPsGjj9rPaW24YepF1Z6OO84axJT5lMvZzNvUqdZ59NFDD9l9yhmL7m5b9rHHDtGZcBMn2pfwTv8cOG2wAXDggZXJxsbHHrMAJ+XPir3/PjBz5hBZVkSik04C3nkHeOWVVIvZbz9rbyqSwg89ZJd+2WabVMt/5hkbDwy5bDpgwV9Dw5pbDibr/q05DUWTvnZTLmed+tSpMetKDJycESpLNj74IDB8OLoOOyzVshcsAKZPH6KGELAe/7DDgAceSL2o446zTuPpp8nFhx4CdtvNLkSXIj37rO0uG7J8+uhH7UKTg8SnmTOBuXN7L5RKaJoyxY5xS3nxRAdohiyfjj/e2r6U+ZTP2zaaMoWMMfv3v+34w0HI0j7wgB0+nHI8kB4NUjdwLmd5VTYOsKfHgsBB6P7t7u6vw9pMGQCsR5o40S5bH/z18upohx2AHXckSaxSyXaFfPKTqSxvQun3v7f7QRhykx5NnGgXyHv99VSLcYsvu14qtLXZVMIgOKz77gOGDRuC48oc5XK2naZMST2gOu44u+/rBn7+eeTa2gaFT/fea7/+scsuqReVDm24IXDooYMG1D/8kHwVxM3eSdkY9fRYu3fssXbN/yFJblCcOEOjtuTmnBjTm+0LSMtdcMEFOHGA+ua+/rG2Z/+ADADWJx16KDB69KAYw1NOsb2+CxfCflJswYJBc1i77TaEHRbQ7zBS7gYeNsyOMbv//t51WCdPHhSHVSxah/WpTw1hhwVYoL5yZeoB1U472e2++3ovPPQQTEND6oNcFyyw2eFTTkm1mPTppJOAWbNSD6g+8Qkb37ogFA89ZFdgSHlWxjPPWDs7pPm0pr4K4rp/Y2ZlXH311bjjDnGeaDDReSZPP/00jj/+eGy22WaIogi33XZbxf2/+MUvsPXWW6O5uRn77rsvnmFfHwp5xpqiDADWIzU02DB18uRUIq077rijT0lOPdViifvvhwUyhULqA4nmzbPR95A2hID92Op++6UG1E888cS+aPa004APPujtBn7wQeuw9t47lXIdPf207R0b8nw69FA7MyLlgCqKLJ+eegqYP88ADz6I7oMPBkaNSrXc3//e+seTT061mPTJBTQp86m11QY1990HFD9st4HBCSekPq7svvss8BySk94ouS7YlLuB83mbwC8W0T8rI6b7d/To0WhtbfXeE0eu+zeXA9rb27Hbbrvh5z//OVqEXrF77rkHF110Eb773e9ixowZOPDAAzFhwgTMId/tjHvGGqVaLCY41Le6WQia0kMP2dVCp01LvahddzXm4INKxmy3nTFHHWWMSXdRUrc+7z//mVoRqVFFu/zwh/Zl3nsv1XI7OowZPtyYr31+hTFNTcZcdFGq5RljzHnn2TI7OuLvrYdFbL10zjnGrLde6osNz55txeGOb79qDGBW/PjHqZZnjDGHHGJ1eKiQV1Y++lFjPvKR1OvwwAOWTzO/e489eOKJVMsrFo3ZaCNjTj5Zv6cedMi7ELSjUskuxLxyZer1Wb3aLsZcWtV7wBeFJjR37lwDwMyePbvq8np6bDGrVlX+Nnz4cHPrrbeWXdtvv/3MF7/4xbJr2223nblU+YiC9AyNBmMh6HWgl3uI0lFH2b6/++6zg8BqSPfccw8A4NRTT+3dAw98fyaAN+3qzCnTvffa3pYdd0y9qPTpxBOBSy+1WYuvf72mj37ttdcAALvtthuGDbMrVHTd+6BdW+4zn6lpWZxc9+/xx1sxHPJ04onAbbfZj02nOKBxp52AvfYCSndMAvJ5dB5zDAaWj/CTy6b/4AcpFjKYdNJJdiX4OXP0j2LXgCZMAEaO7OXTppvaDwWnSC6bPiSztF//up3dRMkOzktvlsReewHXXYeGBqCz09i0nEsJKvTKK69g2LBh2IF9RvWaa67BNddc4y1uypQpOOSQQxItM9jV1YW//e1vuOSSS8quH3XUUfjrX/8a/4A6oKwLuF6ppcWOXbr3XmD16po++sYbb8SNN97Yd37qqcDZ+C2K+cbULdScOcBzz9ky1wraYQfbFTtpUs0fPX36dEyfPr3v/NRTgYntt2Pl2K1TX6T7iSfsF0CGfPevo6OOsl2xd96ZelGnnVLCEfPvxMpDjobZaKNUy1prun8dTZxo93eLn3CvGTU3A2dNaMNuc6egeNoZqU/3dJOphnz3ryPXXZ7yOMBcDmjI9SAy8d2/M2fOxO67744cA4nnn38+Zs6c6d0+0rvIKe3+jaNFixahp6cHG2+8cdn1jTfeGB988EGyF11DlGUA65nOPtsCi8mTU7XwO2zdjQ0Kd+GpkcfjyNGjUysHsHgWWIuABWD59PWv20UVd9sttWKO3m0eGvAXPLLx5Tgh5fFK995rx0ql/DXAwaPmZougJ00CbrgBGDEitaLOHvckxuJ9PDz2x0gXpls+7b67/YLkWkHbbms/x3bbbTYTmKKcf2X9e9CAIh7f4nNIc1WWIT/797rrKq8ZYz/fl8+n3kXQGHXDACjlG+CD6a+88gr22muviutjxozBmIDV0d3s38bGIbhId5WUZQDrmY44wn4z6Lcpf9Dkj3/EmGIbrltyFt55J92i7r3XfhJV/AbxUKXPftZOnkmZT03334kcDK5883O1TgqXUVeX7dH+9KdTX75ucOmcc+xs4PvvT7WYsdPuQHt+JK6Zle5XJebMscvXrTXZP0fnnAP885/Aiy+mWsyO0yfhtfweuOn5PVIt5y9/WUsmU1Gii0Kn+W1gY5DrKaKIAopFPyqbOXOmCACvueYatLa2erdnnnmmb75l6DKDG2ywAfL5PBYuXFh2feHChRib8uc5a0UZAKxnyueBM8+0i0IzIasp/fa36Fl/Q0zFJ1PtIZs1y36w/vTT0ytjjdCGG9rwftKk9GbGGQPcfjs+3PkAzOzYXv42cI3ooYeAJUuAM85Ir4w1Qvvvb7vs01yGoRdgvveRz+DFV1vw+uvpdS3ecov1w2eemVoRa4ZOPtkOgbn11vTKeOMN5F58AW/sdyYeecQms9KiX/8aWH/9/nUi1xoajNnAxSIiGPTkGtDdrfc4d3R04K233hIBYEgX8L77fqRvmGHoaIDGxkbsu+++mDZtWtn1adOm4cADD0z6pmuEMgBY73TWWTYvfddd8fdWQ0uXAo88gvznPovDjmzAr3/du9ZcCvSrX9n0+tlnp/P8NUpnn23XaWHGoGY0cyYwaxZGXHAWNt/ctmVa9Ktf2VVmjjoqvTLWCEWRzS49/TTw9tvplPHww0B7Ozb51pkoFIBJk9JJoRaLFgAefTSw1VapFLHmaORIO8npd7+zC+KnQZMmAVGELb51OlatAnrnxdWcFi60AdXZZwNNTemUscbITcoo+0Zljam7GyaKkGss9HXRSvT3v/8dALDHHpXZ3DFjxmC77bbzbo2NLeIqM+3t7X0gsVQqYc6cOZg5c2bfMi8XX3wxbrvtNtx8882YPXs2LrroIsyfPx/nn39+8DPWKNViKvFQ3+pyGRhK48cbU8M6trW19S858Mtf2mUQ/vY3c//99nDy5NovSdDebsyoUcaccUZNHzvopLZLZ6cx669vzCmn1Kysjo4O0+HWYPn6141pbDRm8WLzX/9l+fSvf9WsqD564w377KuvTva/eljCIojmzjUmioy54op0nv/JTxqz5ZbG9PSYU081ZtSonqBldJLSI49YPj3wQO2fnTYFycpf/mJf8K67al+BUsmYrbc25uMfN6WSMbvtZszee9vLtSa3SlTIyiT1oENBy8BQ6uy066YUi7WvTKlkzLJlprRypTtUV5658cYbzY477lh1UStX9i43w2TgiSeeMAAqtrPPPrvvnhtuuMGMGzfONDY2mn322cc89dRTiZ8h0WAsA7PGwVc9bHUPAP/3fy2rZsyo/bM/+lFrAUsl09VlzCabGHPMMbU3RrfcYl/h6adr+thBJ2+7XHihXaNvyZLaFrpqlTEbbmjMiScaY4yZP9+YQsGYb3yjtsUYY8wll9hnz5+f7H/14LyC6aijjNlqK++aYlXRu+8ak8sZ893vGmOMeeopK/O33FLbYowx5thjra52ddX+2WlTkKz09BgzblzfuqQ1palTLWPuuMMYY8yNN9rTv/61tsX09NilVQ89NOz+etChxADQITNp4byBUu8igKXubmOMLSJmKcCqKA5crikaDACYdQEPBTr9dDsm5vrra/K42267zX6O5rnn7Offzj0XiCI0NABf/KL9bOp779VWNH71K/vZt4MPrulj64v+4z/sGn2/+U1NHue6DXDnnfb7vxdcAADYZBP74YJbb61tD1lnpx0ed/zxtoy1ls45B3j33dp/Gu7nP7ddYr3dP4ccAuy0UxE33FDb1TLmzLE6+vnPhw9YH3KUy9l+02nTUPOZaf/zP1bAe2dlfO5zttf5hhtqW8yTTwJvvgmcd15tn1tX5D4N5xugVw0ZY7uW83mY3jVZnKzX+uNY7nmNjbV97lCgDAAOBRozxlr7SZOA+fMH/Lg+APiTn9hvDn/+832/9WJB3HFH7cYuvfyyndB3/vlr+fT6ffYBDj8c+NnPajIuZubMmZj58svAT39qF0b9WP9iFV/5ip2o4ZbVqQU98ACwaBHwpS/V7pl1SSedZD/j96Mf1e6Zy5YBN99sQcUWWwCwsv75z6/Gyy/byU+1oltusf7xi1+s3TPrks4914KLn/60ds987TUL/C+8sM/jt7ZarHnffXa2bq3oppuseT3ppNo9sy6psdEKZC2RmfvWMEFlboJGV1ftsKbDmaFr/61ttA6+8hCliy+2I2B//vOaPG6zlSvtN2W/8hVrAXtpiy3sdzLvuqu5ZmN7f/lLm8Bc62YrSvTtb9vPM9RoIdvNX30VmD0buOSSMvR8+OH2qxNkPe8B0y9+AWyzTc0/PFN/1NgI/Od/2hTNCy/U5pk33wysWAF84xtll08+uROtrbZta0GrV9ui1srJH5w239xOgrvlltqtgvCzn1ljxKKcr3zFAoFbbqlNMXPm2IDqrLPWsqWUJHLIrLOzdsisq6s/u0jIYc1aTTzu6bGr2KxLa/9RygDgUKFttrHLI/zyl8Dy5dU/58478bvnn8ekl16ymrTpphW3XHAB0NaWq8kqDHPm2OXxzjwTWG+9gT+v7unoo4E99gCuvXZg62PdeSc+c8klOPK666xxZdPfogj48pctfnn22QHWGXatsmefBS66aB2JhM891wpkLbKA3d02MDv8cJsFJtTaanDWWXZC64IFAy/qpptsJwD7+tTaS9/8pgUWtRj+snCh7UU55xy7LguhnXYCjjzSBlS1CHyvucbq6MUXD/xZdU9RVFtk1tNjt6amClRWKNhLteCRMVa03JKG6yTVYiDhUN/qfhKIo+nT7Wjlaj8wP2mSMcOG2We4bdgwe51QqWTM+PFdZtNNBz4w9gtfsJNX58wZ2HPqhYIGak+aZPqmU1dDgXxqbzdm7FhjDjpoYDMYSyVj9t/fmM03r34sdz0MYE9Ml11mZwT/858De85dd6n8bmtrM2+9ZUxDgzFf+tLAiunoMGbjjY05/PCBPWdNU2JZ+cxn7BICy5YNrOArrrB8ev118ecpU+zP118/sGLeecdOpPrKV5L9rx50aMaMGaanmlkWpZIxK1bYbaDTqTs6yqbk8vr0zg0xvXNDqqbubvuczs6BPScN6unpMTM8kz6RzQJeBwGgMcZ87GPGbLqp1YKkNG5cOahw27hxFbc+/PBSAxhz7bXVV/X1143J5+0KJmsLBRnpri67FMjBB1dXSAI+/eIXA8Oaxhjz6KP2Gb/6VfXPqAfnlZgWLjSmudlGKdVSd7cxu+9uzI47itMTXbt89atWFwaCNX/8Y8unZ56p/hn1QIll5aWX7Iv/6EcDKdSY0aONOe449ZZSyZgjjjBmgw0GhjW/8AW7GMDcuUmruOZ1aNasWaa9vb26P3d12YYbyNT0YrFiVjEHgKWSMcuXDwxrlko2gF6+PJ3lfwZK7e3tZtasWervGQBcVwGgWx/ryiuT/zeKZGARRRW3trW1mQkTrM1curS6qp52mjHDh1s/u7ZQsJH+f//PVL1QWwI+dXXZpSZ22626pbh6eozZay9jttlmYHa7HpxXVXTBBTZd8+qr1f3/uussb37/e/Fn1y4LFxrT2mrMSSdVV8zy5RaYpLEqymBTVbLy8Y/bpZAWL66u0HPPtQj8tde8tzms+b3vVVfMm2/aYr72teT/rQcdWrJkiXn11VdNe3t78kzgQJGZgsqkerjlB6vN3tVr9q+np8e0t7ebV1991SzxLCdWKwAY2Wet27TXXnuZmTNnrulqhNPpp9sRxq++aj9tFUrjxtlBedL1d98tu7Ro0SLMnbsB9tkH+N73gKuvTlbFV16xE1e/+13gv/872X/rmRYtWoQNNtgg/sbubmD8eLt8yz/+AYwaFV7IllsCc+dWXhf4BNivGJx2mh1redZZ4cUAdubjKacAt98+sEk6we1Sb7RoEbDzzvbj1P/3f+HfgQLsoL4ddwQOOgh47DFxFDltl//6L+CKK+zqS/vvn6yaV18NXH65HfO5337J/ltvVJWszJhh9enMM5N/Iu6ll4CPftRO/AmYUfzZz9qvd7z5pjhE2ktnnGFN89tvJ19KqV50aOnSpViwYAG6urqQGB+UStb2FQrJdKmK/7pxgNUs3zKQ/6ZJURShsbERm2yyCUaPHu2772/GmI8MuMBaoMihvg2pDKAxxixYYMfEHHFEskjrwgsrs0rC2DJj+qPR004zpqUlNnAuo9Wr7cdLRo+u/ZrIa5oSRekvvmgXBv7yl5MVctJJwXwyxmbx9tnHjuFbtCi8mAUL7JiyarOHlOohe1E1uTGbP/95sv999rN2gOsbb6i30HZZscKYjTYy5oADkmVb//Y3W8zEicmqV69Utax85zuWT3/8Y/h/enqsMRo7Nrhf9+237ZjNs85KVr3f/c5W77LLkv3P0ZDWIUelkjGf/rQdWqGMtRRp+XK7svn48RVDKbR2cZ1h11yTrIrf/7793+9+l+x/9UTIuoDXYQBoTP8n3H7727D7P/zQKti4cWZBU5PpcWPKFFDhlG7ePAsSdtwxfFzMBRfYqj34YNj9Q4kSG+mvf902xrPPht3/5pt2ANFHP2pWrL++KcXwydGLL1qQcNRRYWCuWDTmyCMtuK+295PSkHZepZL9hNvw4fZrHiHkvM/ll3tv4+1y5532bxddFFbM8uW2i3+zzewwtrWBqpaVVausIRo3zqLpEHKDZGP0h9P3vmf/dtNNYfe//bYxI0fayVTVDqUY0jpEad48Y9Zbz46BDu1G/upXbYO/8ELFT752OeEEO4LjySfDinn6aRuTx3yFre4pA4DrOgDs6THmwAOt1Xn+ef+9K1ca84lPWMl/8UVz2GGHmcMOO8z7F6p0Tz5px7VMnBifcLz7bitVaXymrB4osZFescJOCNlmG2Pee89/7/z51sGNHGnMvHnm1ltvNbfeemtwUTfdZNv+O9+Jv/fKK01NP1M25J3Xu+9aAHjwwRZ1+WjmTJve3m672GnyUrtcdJFt+9tv9xdTKtkkYy439D+hSGlAsvLss3Ys7Jlnxkc6Dz9s0cEnPpF4TFqxaGOChob4STddXRb4jRxpgWC1NOR1iNKtt1oh/+//jr/XzW5SBk762mXpUmsy11/fmLfe8hezZIkxW2xhzLbbxqt4vdM6AQABXAhgOoBOALcF3P+fAD4AsBzAbwA0hZQzJAGgMRZQbLONMSNG6Faqo8MOoI4iq5TGJAaAxhjz059aabniCj2oe+op60MPOmhofqM0hKoy0s8/b7vst9zSmH/9S77ngw+M2Wkn24C9vEwKAI2xY919CY9SyQIP50NrNQNurXBed99tI5399tP70mfNsrMxttjCrvcRQ1K7dHXZ0RtNTfr3Z4tF25UIGHPVVQneYQjQgGXFRS+f+Yy+GsK0aTYlvt9+VXv7JUuM2X57O/dEY/WKFcacfrqtzj33VFVMH60VOuSoVDLm5JNtw1x6qW5ofv1re88pp6iAPq5d3njDxmO77qr3Us2bZ/1SoSAmGYccrSsAcCKAEwDcGAcAARwNYCGAXQGMBvAkgB+GlDNkAaAxxrz/vjE77GCBw5139mckSiVjXn7ZepooKusqrgYAlkrGnHGGlZhDDy1fzmLlSmP+8z9tMdtsk3z5g6FEVRvpl1+2wGHsWGP+/Of+RayKRXu+yy52nN9TT/X9pRoAuHq1HWMGGHPqqTap6GjxYmtnAZvoCu1FC6G1xnk99JBFZrvuajN9znGtXm09/NixdiiFZ9wfJa1d/v1vY7bayuLNb3yjHKO8+67lD2DHoQ10fGa9UU1kxUWkH/tY+SKjH35oZ+APG2bMHntUP2u4l2bPtpm9ESPsklh01uiMGdb0RpExV189oGKMMWuRDjnq7jbmvPP6BZkOCJ83r38dzk9+0jsdN6Rd/vxnq0ubb27MHXeUJyn+9CcL4ocNG9rj/iitEwCwr5LA1QEA8C4A15DzIwF8EPL8IQ0AjbGj+Xff3bJz+HBjjj/ehq6A7cO4446y26sBgMZYX/ib39jhHU1NFmiMH2+XJQTsXIdagop6pAEZ6Vmz+htrzBhjTjzRDu4CbKM+8UTZ7dUAQGMsVrnySsujUaNs5LvPPrbIQsEOmq41qFirnNfjj9s1WwDLn4kTbeMBxmy9tTH/+Efwo3ztsmiR9Y9RZHHlIYcYs+eeVoVHjKhQ27WGaiYrt99uBRqw9u7Tn+5fQH3//W1WvQb0xhvGHHtsP/sPPtiYnXe2pnXTTSvUtmpaq3TIUalkU9huGavddzfmmGMs36LITnjr6PA+IrRdnn3WmI98xBa1++7W7m27rS1m110TqW3dU60A4JBYBiaKoqsBbG6MOcdzzyuwAPCe3vMNALQB2MAYs1i4/zwA5/We7gbgtVrXe4jTBgAWrelK1CFl7SJT1i4yZe1SSVmbyJS1i0xZu1TSjsaYEQN9SCH+liFDrQCWkXN3PAJABQA0xtwE4CYAiKJouqnFmjprEWVtIlPWLjJl7SJT1i6VlLWJTFm7yJS1SyVFUTS9Fs9ZY599j6LoySiKjLJV83n7dgAjybk7XjHw2maUUUYZZZRRRhmtPbTGMoDGmMNr/MhZAPYEcG/v+Z4AFkrdvxlllFFGGWWUUUbrMq2xDGAIRVFUiKKoGUAeQD6KouYoijTQejuAL0RRtEsUResBuAzAbYFF3TTw2q51lLWJTFm7yJS1i0xZu1RS1iYyZe0iU9YulVSTNqnrSSBRFP0AwBXs8pXGmB9EUbQlgH8A2MUYM6f3/osBfBtAC4DfAzjfGNM5iFXOKKOMMsooo4wyqnuqawCYUUYZZZRRRhlllFHtqa67gDPKKKOMMsooo4wyqj1lADCjjDLKKKOMMspoHaMMAGaUUUYZZZRRRhmtY5QBwIwyyiijjDLKKKN1jDIAmFFGGWWUUUYZZbSOUQYAM8ooo4wyyiijjNYxygBgRhlllFFGGWWU0TpGGQDMKKOMMsooo4wyWscoA4AZZZRRRhlllFFG6xhlADCjjDLKKKOMMspoHaMMAGaUUUYZZZRRRhmtY5QBwIwyyiijjDLKKKN1jDIAmFFGGWWUUUYZZbSOUQYAM8ooo4wyyiijjNYxqjsAGEXRhVEUTY+iqDOKotvI9a2iKDJRFLWT7XLye1MURb+Jomh5FEUfRFF08Rp5gYwyyiijjDLKKKM6p8KaroBA8wFcDeBoAC3C7+sZY4rC9R8A2B7AOABjATwRRdE/jDFT06poRhlllFFGGWWU0VCkussAGmMeMMY8BGBxwr+eDeAqY8xSY8xsAL8GcE6t65dRRhlllFFGGWU01KkeM4Bx9F4URQbANADfNMYsiqJoNIBNALxC7nsFwAnaQ6IoOg/AeQAwfPjwfbfffvuqKmOMib0u3SP9nvRakn0avyW5J2Qfd813LJ1r15LcE0WR97/S7/waPdeO6Xka+4HeU+0zffu4a6HtJf0Wd71aCpEx33maOu72adqDgdyb5P19x9K5di3JPSGyEqfvSXU/iW5Uq8/Sf9KwBwN5R99x3G9x16uletb1trY2rFixYsAvPJQA4CIA4wHMBLA+gBsA3AnbVdzae88ycv8yACO0hxljbgJwE/jCwgYAACAASURBVABstdVW5qKLLkIul0Mul0M+ny/bu+MoisTf+BZFkXoOoO9Z7ro71jagXLjdc5IIPBW8UqlUdp0bcmMMSqVS2W/0mrR3mzvv6emp+M1tPT096rl07PbFYhHGmIrrvr32m/Q8fh//TXsXd59rW9cO7jfenvyYtzvnDedfiKPj8qE5Bip//Nx3zGXZ7anMUx1x55re+HSuUCio92l77VjTaZ8++zbXBlTPpT1tX37Np+8hjo9THGCTzjW9T6Lzmm4ktQFxdsD3u6S3SXXat9H30fQ+VMcpH6je09/4sUaanFBfoem9JqMAyvSa67+k96Eb96eST/XpsuaLQ54buml+XGqHJL7c3Ud5lVTfJ06cGCsTITRkAKAxph3A9N7ThVEUXQhgQRRFIwC0914fCWA1OV4R8uzW1lYcdNBBohEOjVBIPSuOtYhZMhbOoAzU+PoM3q233gpjDM444wyvoY0DYEkBGf2PZPR5Pei70Xfh7caNMD/WDLG7L45/nHxyICm3z8BqxjaXy+HDDz9ELpfDmDFjgo1YnEH1gSIOuOj9PoDF9xy08d/igicJTGqAigZTGpCqxsiGRuq+jeu1psvSb3SjOuMDQHFgKCQwGmjA5NNr95vTO27HJJ32ASr+exz/OPmyTBJwkvRYC5AKBeta0wqQ4nRX0/daBE5a3TQgqOmyO6Ztyds4NDiSziXeJ9VhKls+f9zT04NisZgomNCCnhCfvHTpUlGmk9KQAYACOc7mjDFLoyhaAGBP2K5h9B7PCnlQZ2cn/vWvf6lCHCfAVEC5AAOVhkP63VGhUCj7TRNgd8wFWTKO3ID++9//BgDssMMOKrj0AcuBCHfIbyFRfJyj8W3Se/JrSaJ4CUxyQ0PJlcN572TD0YoVK/ruGahDkoCUFuVqzkk6j3MMSaJx6X4NIPp0lTvcJDqrOR/e9qGROr2H631cYEjlSpI5SUZ9gVGS4HFNbDzg43X3BX+8LbRz3u4+naW/JSWq17Se9DdNd3mgyGWTXw/V71CAxnVd00XtGfQ5IXWR9FTSV9pOUrsk1VuJVyEkyQXXX/4u9DcJYEr+RdPrUqmExsbGRHXWqO4AYBRFBdh65QHkoyhqBlAEsC+ADwH8C8BoANcDeNIY47p9bwdwWRRF0wFsDOBcAP8RUmY+n8fIkSO9TkMSRkAGd3TPjyWSolXuFCTjHycsGsBxkfzixYsTAbm4TJ4WwYRGOiHgjgNByYlJjk4DdFz5eHvz647inAM18lwO4hyAO25oaEAURRg+fHiZwQT6ux3pcRxYkwCYFu27/yXJANLr0rVQ4OhzKj5HIZ1r7T0QPdWyCFQu4oCItk8CzuKCorgsoKaPIUFYSF20c9dOod2n1QZdIQCO66lk2+l1LZCQAFlIkKLppaQ31eheaJae2wyewU8CAjVwFwfceHvX0pdSueEyExc8cFnVdNM35EHSm5AsvvbbypUrY2U7hOoOAAK4DMAV5PxzAK4E8DqAawBsBGA5bKbvdHLfFQBuBPAegFUAfmQCl4CJoghNTU1eAZUMgPsv3dNncgoRTkB2IiFRQQgYosLc0dFRIbwhBj/UmaQxHsenkDwjoDkQ3t6UZy56y+fzIt/i+BwH7vi5dOyM56pVqxBFEUaPHh0ciUsOhANBydnEdQFp9w7UQYQ4CUkneds7HvX09CRyGJy/WgCmBV78d5+OhgQtIU6EA6pQoBgK0KQAkp4nycRxfdSCK6ozuVxODLqASlDn9I2SBuZoGe4aDxI02QPk8XA8y+Xu8wFAn25wXfVd0/6vXUsK0Hy+kOsi5U8URX17zockxPVS85lxm5OZuCCDy7svOEuqyyH3SeVoderu7q6qTTnVHQA0xvwAdk0/ie72/K8TwOd7t0TU3d2NefPmBTk4Sem5YlHF4GOUAFQYllpEOHRz15zQ0HtKpRKam5thjMHWW28dBBg1ZxMStdBxEfQ3Op6IZwnpf6oZZ8jrKDk1n9JpwNHnwDh49xF1WtyIUvlxYwCpXIQ4Ic1paCAwbvyQtA85DnlOKOgMcZqSA+Nt6gOSSRyW5IwkXZTkKcTpJAF01ehKyLHvGv9Nq1uhUKi4nsvlxPfygcoQPaS2MYkeaoCRg0UqT/xY0sek43ZDxvdJuho3Zpf/J0QfKZjVQGytfKFPF0N0kB6H+kINAA4VX0h9yECo7gDgmqBhw4Zh7733rhDKJCQJIFAedXDAEWL4pYwAFSgaRWgGmp83NDQAAJ544ok+AdScgXRNcjS+bCCvq2b8Q7uEeHtzY8+zBhJx/koGX3MAWtTvM/4hXTncEM+dOxe5XA7bbrtt3+/8/qSgLA58hQZAmrH3tWOIsZf0zmfouc5JEb+T7zh9o7oUCrqSAKVQcOXLomv18G2SXlFwlQRocR6E6Fi1utbQ0KAGPFTvuGz6ulNDgE9aEyni9EkLbqh9Cc3Mae0eQhK/4/SNyqgvwOF6JulhCLjSjuMmKtFrPj1PkinX7IprS02fQhIKEj8AoL29veJaNRSFOMu1ncaNG2cuv/xy1cFKzlZS8pCIT4uWfA4TCI+M4jISkvD5oiBt82UbNNAoAUtpKxaLsb/7nsWjJl5nyUhwhysZLepItUwPN5R0LxEHP1JkTDdJbnxRugY+4465zLvsQ8jvvnvj9KoaB6sBU6mtQvXMp2uajkl6FadnmhPUgKQGEn3XJH3SdMhdp/+R9Ekr01dfp0Pa3umWpl/cdkn6JelaUh1zMgOEBXvSeLk4PeMZvBDditOruGdxHxWXEZR0zb2/pF9cp9w5b2tJt9w555/Pn1F5CAGemo5JQZbvOIkvq1YP4/zp888/j+XLl1fXt04oywACGD16NE444YREaWfN8VPkrxn87u7uYIPvM+yaALnn+65xYZT+4yvHZ/Q5sJI2DVBx40/bmra/RpKxrwZMUYOrGXvNsEtG3/1PMtCh12hWxGf0k4IpLRvBgRR1jDxw8Rl3nw5xXaomQNGAvKRXvBumWoNdLBaDdY//38m+Bqx4cEL/ExecSDo0ED0KAUtxAEnSKS149slwCFDy3ZMkiInTobhsY5wOaQGJZMNC9MgdSyCZ29dQcNTZ2VmWrfOBosEERpIe8WMN3El6w9uJ6tJgBfdcPuJ0yfF5oJQBQABLly7Fww8/HBs1xTlVn2N1x25mZ0hmAgjL/GkOVTMG3/nOdwAAV111Vawx4ABvoA5Vuq9aAxBimDSn6gOm1OkOpkPl/F+5ciWiKMKIESO8TpU7Iw5UpWyDuydpNiE0yyCVITl4n95IG4CyesQBk1AdknRJ4rXkUEMyDpIMSsFTaOAX6nzjhneEBHVUH5IGdtxGcL2hjsyXyfOBIm4zNbAVClK5TEqBnw+wxoFJ7Vq1AZwvqPOBUbr36ZJmv3z6I+kS1yMtcOG6JMkdt+VJ9MqnA9J4vBA98ZWl+VP+jtRGON2gtobXrxaUAUAAI0eOxMc+9rHEAs+FPSQ7SIXC5zi0BZNDhDjOMbz44oswxuDOO+8sA2BxoIsrCq+rz3FQ4dayGJJz1dpX4gHnk5SdkpwFdwpxIEQzvqHGPjQTOGPGDORyOey///6xQI2XQZ1WqIPQ2oJHqrRNQ7IWoU6BA22uG06WOjs7Yx2BT0cGkqlIEpSEOAnpHUKAFQdTErjy6QnXD2mfFEjxjHkIgKqmK9SnM3HBi3Yu6YmU5ZNAFAdUPhAVl92rxqeEZPac/HGfEgeOQoINLRueRGecfkvgyZfVCwFOkk/RMnk+PeE6ovkUpwP0OM7mFgoFNDY2irIoye4HH3xQUc9qKAOAAFatWoXXXnutwlBRgyUxkRsAX2aPOk+3Sry7Tklzlm7PlR/wr1AuRRt33XUXAOCAAw7wRk5Sdo8raqjBoBGMBA5DvzIgnccpfJzyu+shURUFQHTvc5bccXLZ0cDlnDlzkM/n+57vywaEHIf+LtUnFDz6nKLmIF07+sCkc8ZxeuLTlWocpxSESeBTC3p8gM8HBH1fzOHZOK4PUnBFnSPXBSr3UvAl6Qwnd71YLJbtqW5wfXG6RGWF/kblJdSRSnokAVDtWAJ9/DetHJ9u0/eTgqs48OjahesEJXou8Smfz8MY0/d+nK8h+iLJUZwOhWxcl3x6V+3m03v3nu79fICb6wptR643GjleUr658nt67FASn0ystcvArAlqaGjARhtt5HVcwMDGPQHlQpIk+xECfjSHJAGz1avt1/Lefvvt2EyfL3LTQF2cI4x7D8nYUIcVAuY0CgFw7lwy2PyaBJB8GY8k2Y9isYhcLodddtlFvaeazIbPcUnZDi0TyPWgGl2QQBvlraQHcY5Gkze359089Lom+0nuCc380d+1gMb9NpDAJlQfONhIkv2TwJEm/5pOaDoSJ+/V6oMU8PPsnxbQSPZC04M4ffAF+pLd8+kBD6K5LfbJZFIfQO+JmxzkC3o0kDeQDB+3LWnpQlyAEhJ8+ORauzZz5szY9wmhDAD2ElU6dw6ULw5cKpXKhESLhqVn83LcseTs4hRcAnyackkK78rXgB13TlyJuaHRQCx99zgFpIbU3V8oFGCMQS6XUwGg1s78uZR8xhvwr7UngSRfxiDp2CF67CZ8NDc3V5TBu3dDAJ5rD54pcO2Zy+X6+EzfNzRbp5Hm4OgxlWt3nct7aCCkgb9QHdEcUwiA82Us+Du69nOZGSdztD0aGhpUmed2xSf3WiZJAzAS4PEda5svMAoFY5I8+zLSVEc1AEezLpw3kqz7QJ2kAxJv4uQ/BOhr+uC7xvUlJKsWl2Xm57xcLXihGTYq/5KNl44lWed64eRb45Mvw+oLdrVgiPsJTTd8/oPf59Mppyu1oAwAwi4EvWjRIjVTEsckSTi4AAHhRsNnHCSjEAIO6bWdd94ZpVIJm222WWzUl8Z4Dy1DIhkePhmDGumQDCFtT37s44kPJFJAJWVBqLzEZeR4ZEczHu792traxN+TLLfCy+P14HLvAInP+dP394HEWsi743OIg+Py7gtqfPJK5TbkHt+zaB0d0NbqG+JI+bFrH61t42Seg8IQp+cDe1zOnNPyyWeoPGv3cbmTgjNJ/jVHzGWe2wFN7uPkncq6JPcUJPlk3hfwaMF/6MaXAXL2iNaNZub486WypWv8OZpfo3bAZ+ND5Z3z0p1zu67Zei1w0YL5uN4e31hWXj/3rFpQBgABtLS0YJdddqlKmUPBmgTUfBmGpErLnRRXYHq87bbbolQq4fHHH69qAohUV58ic6PGlZnueTQHxH85gxpnTYE1Z+XLwoU4plyucmmWJACNP9/npKTzuAyH5LA0mQ6RbQ7GKLiKyxhIgJ+epz05QyrDl/Gg7+FzSppD8sk0l23OF85Hx3efbEuBhwTAtGNfwDCQ4EKrky/DQYMpX3AdYqtD5Jpv3d3dZXZLknGfDY+z5z7AFGd3fccUkHEdi/NHSWS6VnLt7B8ANDU1Vdi0kIAjDoSFXou7l+sTDy40WdbsM5dt104hwXNLS4va9kkoA4AAVqxYgaeeeqrMaSdZdNPnqHmUSZnNhQFItu4TB1JcubWMIDUaPDuiLckSt+fHHHT6/qOBWMl5SxGnFj36DDxtT4kkhZSUNsRI+RwqlSNJ9ujaZnTvO6ag1PfsEKfODV5DQ4No4HzGrBZy7cuESIBSc6qSXDrZ0mTUJ7tu3T/tHq5X2gQqLs9dXV1qUMWdNG8/H0lZWglkaSCTgkaakfAFS1z2uHyGynWcPPvW2oxz9D6HDSS31ZwfGt84EONgTQu4fXYzie2tVr6l8iSfIk3u45vWNqG22ifXnI/UltG9b5a6JkvcxnJZpef0mib7VIZpUkGqQ1x7hFIGAAGst956OO6442KdlKbM0l5S3rjshk9xJaekXfMBsmKxiP/7v/+DMQb77LOPmikMiTLpXookJQXmgitF8JoC+xxSS0tLhUMKUVruZCRF5QrLgZbPUXFlpoZFckL0eNq0aYiiCBMmTBBB1mAArBAHVE2wQB2M9nuIE/ItvCxl+qhjAvwDyuNkV5NbQB5TpwUIzgFxABMKqDTgL8lzUlDFwRWvlyS3UhZEy05LshtF5eOrtUyUZnd9Qe/q1au9cizZQ80OU7nk8uzTAQ1IVQueenr0maua3PrkV+JXiPy6YzpmWZLbECAfB6S0/8UFA1JyRpJdHgj4ss9xpCUhfAkbn+11v9eCMgCI8oWgkzpxLlg0NeyMrgRiJAee1IlTw5jEib/++uswxuCSSy5JBEI1QMmNXi2duGQE3TVXdhwAjTOEGgitxolTEOp+C3XiHHj+85//RC6Xw/z58/t+lwBs0uiRyqtkBKkMS/IrGUDNmbs2lxy4ZgidLGjBVAgYDQGiXMYlhx7qvGkm0We0OQjt6uqqAKC8jXyyS2XYJ79Sdo/bK8mhS3LDnXNIxlmSV+n3OFtL5VbK3FAZjpPdgQRQ1ThyLZDioC8pKOW/cVsbJ7NxQZM754F/qVSKld9QoELthxREOb4CchaPByNcVkICqlD7rB2HbBx4uvrTc/cejY2Nquw2NTUFtWscZQAQwKhRo/CJT3xCBGFO6AD/R7KltD6NCDVAFuLE4gBYXHaF3/f+++8DAK6//vpYw8ABmGYMXPtQQMoNJ203R6EZQAmANTc3iw4szmlpgEnqdoo7157jMwA8m6JFoA8++CCiKMLJJ5/sjTzjZJTyg8so7WriQNsHtkKckhYY0N9CQBcNEOhvPqBF5Za/swSy0pJR6pQaGxtjZdTndAYin0mDAy6jEsDix6EyyuWU2gkpGysFBZzvXV1dXhnV5JTLX1xgKx1zGY0DWtwX1FJGJfl0v3GAEZLNc4sTa+Anzn4mlVVfkCABKcmGUlDly0JLoN/JqQRcJdDvC1ypjHLZ6O7uxurVq8sCR83XS8dLliypqF81lAFAAB0dHXj55ZdjhYwbQi0yocLmDD8Q//kcKdLkmS2tK8AnbPS8WCzi+eefBwBMmDChTLi0DB0XwiTRJc2K8OwdB5H8nR2wlLKdTol4ZkSK6KnSu3MtmuQ8lva+CFPKXvBroVHom2++iSiK8Pzzz3ujSAk8chmVDKBrJ3pcKOgmwZcJCZVTLcNAAZ4PfMZ1nWnBFNcBHtTwTI1UZyd7VA65o3bXObmyOTl+UJl1skiPub2RnLe7h2fFJFtGnXycg6Ubvd+XjeN1ksAj33NdlYC31H7ufeJ6SaqRVS0LpgUfvmMqd3EySPcaCKbnGniWMnEhWWX3v+7ubnR3d4tZOX6N89IHyvi55GMlG+ezy76ARbKTkt2kx1T/uP108hiXRY6iyuXiqMw6bBBiUynfm5ubVd4loQwAAmhsbMQWW2whgjp6HMJwboQo86gT0QxPqGPUnKJvULrbd3R0wBiDmTNnJuou4MbLN0ZFywzGOUpOEniLy7ZIxsLnBLXup7gMX9zYqbhMoM9oRVGEOXPmIIoi7LrrrmqWJVQWadtLQJvzjToXCUBRQO+TxyRdsNJ/44BdXEBB964NeGBRS1kE5OUiNDnkssjlx9f1FCqXPnn0OVvNSaYpi5ptdPz1BQdxMhgid9UOVdFAYdIgt1ay6I55gCsNW5JkUpMdyVZq2bu47tOktjGJLFI55PLIs6iaLIYmXEK2OHnUxpNS/0/3H374YayMhFAGAGGFoLOzU40ItAgViBcyKmy5nF3skiqtIyq8xWKx77inp6fsOC5yyeVyfff19NgFfYvFYtl+iy22QKlUQlNTE/J5u75ToVDoEzB3HAcAOfCTgAQQv05fHHEADvgXa5aiSZ6h4IaQ3sf5KYF6xw/pXrq59oyLUCWejho1ClEUoaOjQ5S7EIcr1YluAwlKuAEMCVYkp0mJGvV8Pt9XN3pNCjg0udMyQFpb8br4ZFCzE+5Yy3hIMidl5LTjuMCGZin4+zq+xGU/4hwrbRtJ/mi7aqBPC5LjnG+obNJjbrO0DJxkx9w7FgoFlEqlvvam8qaBWf7e9Nwnf7x9tayTxi9JNjV5lM4lG8rlhWd84wIHJ3+uzWh2jcsa/Q9/Z66TPtIyn76N20Iuj1w2uZ8b6EYDHV4noLyHYKCUAUDY7pnly5d7Daq0AagADRzcSUDCHYcInxZ1JI1+aWQxceLEimu+8S70Gs/6UGfvAC517A7AundOAgppe0nKrwFC31YoFCqcaVzkq0W1IccOxBhjygwefS9uZN3+oIMOUkGGZiSltvI5Ys3IUZnjzrFauZOOc7lcnzOm4JobQv58Hpj4stESOAwFgxwESgCQyp+U/ZNsSlzGz+0d4GhoaKiQGXesZWm0TGDS7J8GBkNkjsubFHhwXkv2TrI3bvP1evDMmrtPyrRIWT9N7ngduOxpGb+kMsflTgJ5fNOyelz+fNllDhxDbZ2WLQwdNqD5V03efLYuqX8N8a0+W0eHJHF5qsVYZ1o39znXgVIGAGEXoNx2220TG7eBALhQweIOUzqP+40/SzNqvH61cKRa1Eb31TpR39i6pAPk4xwnd6KhzlMCaxy4acTbljtJDbBJ4M0nZ5Ix8nVN+O73OWvqgH1GWIp6JRmTiLctzbL5+BOSWYsb3+lzqL4gwydXmmxJGTvNfnE5i8vMOb757FkcUAvdtGCS2ixNlkNkiwJNDYhxkBgqW+6aGzcbspCxBHy0wEADcL5NewYtk8uTlA3UQCdtCy5XlHh7ajLmluXhWS9J1qSsrSR7FESF3EePpb1v8wF8nvV1RGWMZjG1RBK3Y46XtaAMAAJob2/H888/XwYUQqNon3IldfzcEEmZmlBA6cu6/OQnP4ExBl/+8pf7ohMOGt3gXwcCNMApgU0eyThFdEpM34Gec0XSFMgRd/TUYAHyWCzKL1/mRAKPhUKhby210HPpmgRKJXDw5z//GVEU4dhjj/WClxD5koCUFrRIIFJy0iEBCZcp6ViTwZBAJgRochAgbT7Z4pvmOOOyfD65GqiMaRvPxLilUSTQ6cv6UYfvc/qafIXaL18WOYl8JZWxuCCZyhI95pk+KXBJIl8agOR2yx1z3sbJFpejfN5OUpSuUzmTzrUMYJxscRmLs10++6UFKJRHSf2jT2bcbPO43+NWOXA2i66DSnVBky/3vitXrhR1MCllABB2IegJEyb0nTsB1CJkDlioQaPLvnADQgWAMl5yjHTvjqlwSfdKAiiVPXv2bADAddddVwHItAhHA2M8y5DUkFFnGeIgnRGixogbMn4Pv1dzkhIodPWmEbsEwDgY8wF8asR84MutsbVq1So180bliMqVT6ZCZYfvJcOpAXsuQ1yWuBxxeZK6HbkMcflxclCNPCU9jnOIvqxxHJj3gSyfPPkydRQcd3Z2loH4JPYpDshr13gZPEgMBfBSRom2B5chHiDFBYdcnpqamoKBexyQjwPvSWTIHUvZO5540OSJ66IWDPKeBwlQrVq1KhhUSbKi2RxpL/lWX2aP2yInT1rvlUTcvjv5CfFvdGtqasKwYcMSyxLfrrvuutg6h1AGANG/ELQPKMRFPI75rhtAMjbVdC1LYNOniJJR58b8yiuvhDEGX/va11SwqZ3HRdISQOBRtAMbcZm/pJEzNYgSQOB7n2H3AYBQsOnbtKiZGviGhgZEUYT1119fzSbHZWOA8syXBNC0aHkgYDMOeEqgk0bKPjniMqVlYjSQoAUtdKMyxLvstOAl1Kj7Mnpx2T6fHXLHVIaiKOpbf1DL7sUBzjiQkLRXQuJ1KDiQ7tGyLVx+eV2o/Pf02HFVcYCTtodmk7h+crDgA52Oj9Vm9JICCakHRAtanD3i7yPpj0+WpIQKBZkhgFOyTxLfq+nF0uSHy5Fml0ql/q5tunEd8pHUu0PxRFtbm/f/oZQBQNiFoCdMmBCUveHpZ8mZJgVlcY6TOslagLL3338fxhhcf/31ato5JPtH20oCYwAqDEkU2c+2UaflM3Zx4EsDYCHZP6l8zThrWT/67qEyI2WOneGgC4QuW7YMxhi8/fbbsU6SA3Ytk1xN1ysHXBJw5MY8JNMnZYw1AO+OHZhJCt6Tyg7N8MV9Qk1ymlTeqdwAfvDOu7p4u/IMGbc7jp9x2Rdfr4ImQ/zeOIfpy+5pGWKqN3EyI/U0+HoYHM/op8oorzW58QV91QR+msxwW5O0i9TXsxAKzovFYp/98QFyTT58fomDdSmjR20OtzEDTRhw2y1l77i8cIBcKBTKssJxSQHNtvB7Q2SGHp922mkV71oNZQAQdiFoNwaQgxFNSXmGgGYHAH1WptatHBJRc2NKFSkk3e62X//61zDG4FOf+lQwaKTKKhkQX/efi6idMvtmMMVFPiFZGUlxKV+TRtY+50+zLj55icvAAOh7vjtvbW0FAIwdOxaAPKA6tOvPJz+SAdYi6hDZoL/z8njdKIDUMi9uT52Dkwmua1q3PJWdkCyMJkdS1i0umxLn+H1DDOJkhjs0OltY6uKTZIY7WCkrrNkgHiRoIFADHPyaK1Ny/HHgkWdWeObX1ZXbZK17T+pelfwAlSHOWx/v4+SDP1cqN0RWQrqDnT11/kvzU7zNedaO2x/ORymQ9GX4tMy/ZEuoHZFkhNoX6b0kmXHkgvPu7m5VdnhAIvVM8fM4edJkIFsHsIbkZgFTxmhpbR+Yk5yaphSSEQ0FcjSSCs3i0DIKBbtq/gsvvFBh2EMyO5qR5YZFAm++SF0zir7soC+KlzJC7j6fw5YctDv2dX1IbcCdMAfFkhFzx8OGDUOpVMLcuXPFLHJIhkc7T5LBkeqnOWOffPD2iYvI48CZLwuoZX1Dx2VpgF9z0kkAG5cPKie+YFACYRKvkspHtZO8ONDnWUkpCKTgPql8cDDPA0B3rgHyamUkTj60HgQaHFKbxxMHXDZCEgYc3PgyfHTiApUPbUiHJhdxY4QluZASBRysUflI4mM04E73GkCPSw5U41NCJ8m4MjUbEoJB7rvvPtSCMgCI8nUAkxp0oHIiBGCF0T07n7freRUKJOoqgAAAIABJREFUdiHRhoaGMkeqRT90T5XTZ+TjBmAXi0WMHz/eG4lrERxQvm4XV1SNNEV14NA9hztHWk4+n++rr1Os7u7uvr1k5LUMjebUqcGuxsGHBApuL3Xtcce/yy679EWe3Oi7Mlw98vl8haGkjqdQKKgOnDrxuKxd0iwMfW9JX5JmX3wRsgQSaR24o6H6p2VntKwLD2hCZIK/uyYTWsZFkhHu+H0ZE0nPXfmu7dyxkxkJ+DubQMuXgKv0XlwejKkck0llm2d3tQwLDxrc7xQkhsgPDVqpvDgdkcrh9YqTBS7/GnEQFCIjEjDkciKdhxzncjkYY8rALpcDV28O/H11T0px7UptiCYn7r64DJ0vAUB5ZIz9bJ4mo776umdx2XDXqAw46ujoSNxuEmUAELZhOzs7+5yHAxnc8DtyjkVjpHumu5cKvM+IS87XAb+GhgYR8HV1dYnZKi542nvncjnRYXMl1pw/gIq9RHHRvBY5S8DNle0ymY7o853Bor9JmSUNNIZke7RsoCuPt6mUCdSyPNSBS9lhmjlwbc/PqcP2BQhuQo4vEziQCN4nC4A+o45neKSI3YEWSa5yuVzZWBstoq92wpcEGqTIncuDBvok20ADMdr2UnDn+O6eoU3S0QbG83NeTrUZ4YHaBkkO3LEU6NGgiP4WMi5YmuXN5cEXLAzUNvBASrMNvoQB3bgNiJuwJckIDxolu+DK5jah1raByoTkM+h1qvPuWc52SON7peVwpIwxLUeSCV/SwBcMJLEN2beAa0iNjY3YfPPNRSYB8d2+PBKWovGQ7J2mlNIkEMmwaw6cj8+aPXs2jDHYbrvt+uoG+L/Zy8EUBb0UxGmZGp5l4wbVtybVQLpmQhVUA8w8QudROc/OaZkZKeOqZWjd9uKLL6Knpwe77767N7sbt0SCBCqlKJ3yn8o6Jdo23LhyQ82BksSLkGytj8fcCIeCd8kwU1DtC5oo731APiQb5wNaXI9917TnSxkdqQuRBycacXATt/ixZAc4OKPHErDTeBvCa5q54VkcSed9IM3tpYxsXMaNAyTfxgEWzc7za1JwoMkiB+ZSdlbjPdcNxycpy6ZlzzTw5gPVPv5qvNZ0nL6DBMw533mmmvv8np7+meNcnyT+a3ufjdBsSDYLuIbU0dGBGTNmqGBDAiw0Q+EcIXcojni6GJAFSnIaklPgoI/P3JNmftJt7ty5MMZgn332QXd3tzi7j4NMqS7GmIrlXHxdPtSA+NLzvI1p22sg0EV10jnfCoUCGhsbvTNBKYjkk4KoEQsFkBLvNYPh2njZsmXo6enBkUce6Q0W+Axfje9ulh6/7ssOcoDh5EJyKjzrI/Hd7TnvfZE936QZdZx/lL8+OdBmf/oi/LjoPpT3FIBLjpy2t6aXEi81/nM54DIkBRcSwAzhvXtXrvN00wJGnuXTxmAl0Xm+cbmJy/TEZXZooCjx3bWHBBa5g5dAvqSnGs8dfzV9983u1gITqb7uHUPtPW0nqu9Slp/upew919sQfQ+x975ML+c/B5WU/5z3XE80EKglijj/p0yZglpQJEX56xrttddeZtq0aWpqXsv2UeZxo8mVTDLC/DjumqbAbs+FRzLaxhi0tbXBGIMxY8ZUZHm4kdYiOilip4pJDay00d+0Y5/iSgrLu4h8EaDbS5mPUCXlm5Sp1Ryw79zx880330RPTw822WSTWOfsAATtLpDAuI/Xvm43X1bO54Al48zPuRHnz5YCMF8mV+teidPpUGMcot8hIFz6v88RS3x28upk15e1pe2jgS+q1xrwSgq6tfupY48D3FoGLymfadZO4zMH3JIDjgPcHKD5QHZ3d3eFL5G6UqmNqobPUoDF9UvTbcdfX7Dt+Oyz05TP3F5LuhyXlfd1nWu2O1SvJZ7ze0KXRqI8jrPZlMeSfwaA9957D6tWrYpfLT6Gsgwg7ELQkydPrnBQXLi17IBb3ZtmMiQDRUEHF9wkTsgHKt31zs7OPlDCgeXvfvc7GGMwYcKEoOyAlA3q7OzEypUr1UyAe09KErjk3QTUQEmGQ1rHrampqcwJuXvc9TiAKUWUHHC458Vlf3xBBHfYUtRP+Tx58uS+DKDP2UiBgwY0eWZAM1qOz93d3eKipiHOiPOZAw6N1xqfNR7GnYdE/hRsNDY2VmT6aPbC7SVe0yDCBy41BxQaRPiCQwloSvzm5YcEEZTXcRl9Djp4EEGzOxK/Q7I6kq32BYtU3tyakqEZXUmvQzN6vqxOHJj0BYlaUkDjMx3vy7PQ3F7F6bQ7psGilBwICSSSbBIg1UAm1Wde12oDRpq1D9VrDUjG2WeNvwsXLhTlMillGUAAe+65p/nTn/7kFQDu+Gi3DTWePmb7Mnx0r/2WRNmpAeKOe/HixQCA0aNHl72rz5Dz7hlJoanh9WX43Dk/9mWIJCWnESytJ4144xRb4qtrMxe1hSgx520t+LpgwQKUSiWMGjUqiK+hhjspX7lzlXhVS75KmT4ukyEGm2d7aDe2xldueKvhq8RbH+DmgCGua3Wo8tWnr6F81ezwQPkq8XIgfK2FvlZrh+uJrwOxwxlfdb4++eSTWLJkSZYBrAWtWLECzzzzjJj2plGq1F3ifufGS+pi9HU7OQH3dTH6or+4LIDbxv/rX5i4eDHGdndj0YoVuHX77fHnjTaqyARwp+Tq3d1ts4v83eg7064IaePjfFz7agLPI0UeAUpZgDhwwUGG1v3gyq6Gr75sbne3/0swWz/3HPZ79VWMXrECy0aNwl+OPBIzdt65jP/S/7js8OEANMPDjVapZL8/TPlLjbY7plluzlee8QntZuJZID7mR8vaUSckAUdfF3EuZycxNDU1ldkDzlue5YmL/ukwEO64NH5p4JA7HVqGBhwlvgLou9/xl/LU6avUdcizO66taZtrjovyX9NHSSf5Ne4oJZ5y/rpnUf5yfQX83+OWug99Xcaa/mnH0nMl3+COeX2pvLr/SzortU3cUABJp3gGN26v2VrJ7mo2WMrSRVHUZx9aWlq8eusbAiD5OkmfJf3mvpr+XzuX+MoBsCajDrwC9qMKtaAMAAJoaWnBTjvtVBGRApVr81Ahc0xzwuAcq6TUSVLBWnaI/+6ABHUg9BoX7qMXL8YX5s9HS+87bLRqFb766qv4YMMN8UjvVyfcO7s9dwo8Q+OMhBaxUIAWuvk+16VlidymjRWi7+X4JymbZAi44ZYAWByvNEAuRZuHzJ2L/5g9G829yr/esmU45sEHMX36dDw2cmTF+CD6PpSo7HI+cmNPnbWW3eUgOylfuWz4HD0HH1JwJfHRF1zRBXE1/oXyUtJDLXOgOQmNj3Qv8VHaNPCkAWzHA57xidNXDtB9QZWU5RtIkOw2Cqy1CS2+IFjjJ5cDek7tvGRbpa7UJHyUgieaYGhoaMDw4cPVLB/ljxv6ovGZy4AG2EOCKM03aoET1UcOohwvpGQHz9ZR3mu8pb/RoJsDdylw5+BaS3LwYFgCzz7/KPFH8of82rvvvotaUNYFDGDvYcPMEzvsYJnqBBsoP3bn9I80pe0EhZwbsjfG2GNy7t1o9BJ4rew6TUX37ndetgyNAr87owivDhvWV18vCe3BxwLRduPGomzjGZmY39WNl+3qpPFO6IqgfNR4yPkH2s4hPBX4JW3btLWhsVS5HENXLofZo0aJ9QLKnU1SHlbIPW3DAH4ihE8SryR+CXsAwfwDdD2M5SXhYxx/td/FZ5Oy+/jE+cfeQeNd3yFvG41/jL+ajqr3CM9TdU3iWRX84+1R0U5x7avwVuONduzb07qU1Vd4lz7i/BPaStRNsld1pcr7NBlS6xhIXJ7F85h21XQ2lj9x14Rjse4ekrCA2l6hbc15o1w7sKsL77///trXBRxF0YUAzgGwO4C7jTHnkN+OBHADgC0BvADgHGPMe72/NQG4EcBnAKwEcK0x5n8GVJfyivkNmGOiE6oossdRBBiDCOi77jXwvcYHvfdLRkvMXgUAjAZjcCdOx/dwDeZgS2yJOfhvfBefNXf3RzyuDsL7Re43IrCm7+eo7799UWGvAXIbB3Y56Vz5TQSIGgih9Y3jG2n3CkPieOhuoTwi/PHxocSPPed9fCqVZD6V7kZ3V1cfgKCAT+QbM0I8enXtZjRjE8qz3nP1N4GXGvDQgITGM/ruorOQnLoWNFXJM1/QxYFHKN+islPCN9Z27lpOcP4VupOUZ74ATKgPl59q+FbRdgoffXyJ45nEL413jl9BQF3SI8IreuyzX24fy6ekPON6xnhVNd8CeFcSfqtK77j+AfaYl8/qSOtdQZr9g65zql0L0S36m6SngkxwnRu1cqXOpwRUdwAQwHwAVwM4GkCLuxhF0QYAHgDwRQCTAVwF4B4A+/fe8gMA2wMYB2AsgCeiKPqHMWZqXIFdW2+N9ydNUruAKZUJcMxYr7hu37jlQOKWgKGpbamLgo8HOv8fe+Pb5pdYieEAgPewFc7Dr7E0yuOGcdODxuxpA1R93RJJugWl7sG4cXqOT45XZYab8MqY5Guu+bqXeDcE/w/tHpTGeGpjt85Zdiy+1XNjJZ9yedy20+y+99V4Fdc9L42/k/jhG7OldcPzLiNXzzKjJpDjEx3nxLtxJR2TxtnRTer2jRtn5xtf57psqVz5SGoHPp5NGlfHu4+0riR+XRuH5evSk7r1JD5RveLjq3xjq3zjqLQxjryLkP6mjduiNpnyhnfnUeLgTJqUIo1BlPRMGisXOg6u2rFwGlUAMNYdS3nG5VzimzZWUdIVSW84X7i+a/yhfHL61AeW2LAWzjONb5KdpDzkPJCOuaxo+hOnR3Sjto/zitvC5T/+sZf/oVS3XcBRFF0NYHPTmwGMoug82Izfgb3nwwEsArC3MeafURTN7/39T72/XwVge2PMaXFltbTsabbZ5s+9zHQMlJjqGEqP0XcuUX/zmt5ju7ftrkRLJXdcIuelvr10Tb+n/7zrw63QYUZV1HE4PkSp5Q1Sp/46K9yhfOp99/I2cu3E25Oe2+Oc55rvnlA+Ud7Ujk/9bUvbXeJFKJ/6j7uXDS6fpLZ07U6P+V7/rfJYz0iU16k6Pum8knUpnCcSLyv5L2SRgMR86m8L2iacV5xH4Tok8Sve5ml8onUeGJ+4Lkk6kYxfa4pPlFfV80niFz/28Yn6qIHyqb+d/HwK0SvpnhAelfOnVnyq3OLaPZlv4hPOIsKXgfOpo2M8XnvttbWvC9hDuwJ4xZ0YYzqiKHoLwK5RFC0EsAn9vff4BO1hvYDyPABobNwd661X6XB5o5cbLofUaSQcqgQO0Yfem0QxDKt7v1B1mZFiW3RgFIaXGZc4sObbhytGvFIArCOecqfvHWk78Ig3qfNPCqwl3pTLC41s4wyW3a/28Gm9psYB8afSoMU7fm6odN5Q2dMdSuVst6ROXwcAGkiv1OFyGSp/Bwqc6TtLbRIhn6ftp/PGD8D8OkKDquodCM08yLwp5w+3bfFO3w+6/TaM6kklX/qp/H1512p5YNPfzqGATOZFSGJA1hOdLzL4onzReCP7HI1P0n0SXxzI89mvJHypBF6VQWcul0c+X8kXGYRJ536e8KClelAsBZiUN66HKU4HkuuIFDz19LyjMyIBDSUA2AqAfwBvGYARvb+5c/6bSMaYmwDcBADjxo0zX/zinWhqaupbE0laF0nqopS6vPqNQb6XYfk+5e1P40a96XKgWCzvdpRmj7qts7OzYi9do/9xz3nmmTvQ2Tm2oi0KhfnYcMNTKtL25YIIABF6eqxiGWPfF7B7Y3LI5yu7qGg7uhlqtH2bm5vR2NjYd90ty+Hu4TzR+CJ1S9LZdXQ2MABEUQ7GRL3vlkOxWIQxud42AHp65OU7NL74+KPxRJrZ1t3djVdfnYzu7k0r+JTPz8OwYcf28aZXjnv5YfljTK5vn8vlew1seXcwnQ3K+eKuUX5wXlC+aTyhvNFmiFrwlO/lR8TeA338KJUilEoGPT2mrCtQmq1JedPZ2VnGJ8oDpzfd3d1l1+n9nDd002aB9httSs4R9QORfD5+SIVrU84jrhOajlB+8Rmh0qxPqYurX18sTyxAAIyxPLF8KV/ugq9OwG1a3Mb5xPkqDYfxzbSWeNNvCyiokLvlaXvxmZm83bWNLkpPeaQNiZH0RZoRX+7T0Msj9PKmclkTyc9w3mg+R9s0vnDeSF32ms6Ugz15hi3XGW0NQ+5zJH2hv2kzp/lsac4bDkL7bXQlb7QhY9zPuGEsnC/XXjumgv/V0FACgO0AeGpkJIAVvb+589Xst1gaPXo0PvWpT5UpFY24+Di/YrGIzs5OdHR0iIBNcvrO4XDg5s5Xr15d4ayk8WZ8LJlk5Bz1R1lWOFtarkZX17UwZhi5ZxU22+wX2HjjjSucDncwDpz5zjlQ4IaOLh9Bl24pFApl9eV8cO/IjQhVHNqGjg8UDLh25sCZXpOMHuWLNH7MB5ypTGnj9bgh23LLX+Gdd76DUqm577/5/Grsv/8j2GmnCRVt7wCZxAsfQJPAmTS+UuOFe28+bsvJ7MqVK0Wnr+kAPfcBZj7uko8Xk8bqUeLjvCTHQpdEaWlpKWs/KWjhPJH44ANlccGL02fHB6B/rJBPJ3jQ0t7eLoJiqe353ukXBwt8OZYQcEwzd9p4Yz4+mLZjS0tLGbCiPImzTxoIo2NiuX2iwJgTtU3uWNMJ3n7t7e0VPoLaJC3IlwIcbXyrNA6P2yfqJ+J0ggPghoYGDBs2rKydJZvU2NhYFjhyUExBlw8I84DegS3NZ/t44drR+fTVq1dj+fLlFW3O+eFLtEjjwaWx+a6ulCiAlMYnuvZfsmSJgGSS01ACgLMAnO1OIjsGcFsAs4wxS6MoWgBgTwDTem/Zs/c/sbR8+XI88cQTZZE2z/jxrF9TUxNaWloqIjIujNxZ+ox0NZEYNwK+aKxYfA5tbVfhgw++iu7usSgUFmDMmJ+gUJiMtjZ5QDt9P/fu0hpH3IhKoMMXefFMh5St0CYeOFApTdzhzpKCNWlAusYPKbskRWfa5B1tsoHNPPZntrq7u9HQcC823rgDbW3/iWJxExQKC7Dhhj/D0qVT8MIL9h19vJB4wp2dlJnljlbiK9cDOnDabe65mrPUAittApWWTeL88f3OM4Z0cg7ng6uj4y0Pqrhe0CBGAzHclkjty52g1PbaJBA+2J3Wjy52XZ6ZMBV64dqA64iPJ5Q3nE/0HsoHuue85/Lh7JJzxJSo7eVgjQ/21/SEtq9vL02W8k3wkLJ27je3eDG3UW4vDf6XJsRIgI9n4KRjvnG7JE0MoXaUBnmaz3D22DeJhsswbWepvX2bxgvuuyiAdLaO26rKbnfZZkkgk9py6ZrW9jyQlYJap6Nvv/12RbtXQ3UHAKMoKsDWKw8gH0VRM4AigAcB/DiKopMA/AHA9wH83Rjzz96/3g7gsiiKpgPYGMC5AP4jpMxhw4Zhjz32ULsLgXKB6Onp6TNmcel1CciFdHdQkCGDuXKB486Lgzn3Tk1N92PrrR9gafTRojGUQJoE4rRsk+TQJGWl2Y7QzJOLplavXl3R5eScjdbtJAFpnj3ke/psrtShbU8dQByIbmx8E/n818qyHY2NH+sD0hqYphuPqDmA0wxjaNvHdSv5ZJ5myrVMnwQkBtr27p05EGhubq4IXnzZcI0Pvq5wKvt01m/l0JHq2t5lKXztT+Wc8oHuNbnnmQxn+wCUAbUkbU+dvJP1kK7VuGAyJHCJa3v6Xq7tjTGiQ5fam/c+aHyQAkr+bK2L29WR7uPaXsqu0naTstY+u0/5RYNK2uvjyqMAkPrauLaXMty0jaiv5H7UZ/c5v9xzNLDGM71xbU+DdK3rmsqqa1OnB9pwj3/84x+oBdUdAARwGYAryPnnAFxpjPlBL/j7XwCTYNcBpDN8r4BdB/A9AKsA/MgELAEDAF1dXZg/f75oKGhUydOz+Xy+TFgpEJOMtOYkeQaPd13yTJOW1aAZDd4V5gTUGIN33rEDSLfZZps+oS4Wi2KXmBR9SQ6OG1/uRCWDLHVD8jGV/jEv/V1K1KC4Z1eTTZL44zMIPDKXMhda1zyAivd0POvq6sKCBQsQRRG23XZbMYvE21Q65u0sdXFJ41h5Fwtvcyk6pqDM8cRF105uJIfGZZhniJxc8+hYypTwoRDcMNP2d+8LlGdUXbu49y0W7WeXXF14EKNlIrjD5boldWn52t3tqdOpHFvcD1Sc7judcHzgGVDq3KQ21rIQvHxJ1nk3I8+gUpnjmTu3d23vbF5XVxdWrlzpzbzxbBMffkGzUxwAarZGk38pQ0T54cppamrqc/LDhg0Ts0mUf/R3qSxeF1pHTpwHdJNkkNslJwtdXV0VPJL4x/8vgW0u67TdpQSMjw80OUP1pFQqIYqiPhvY0tJSkdGk7cqDSemZnP9aHTWS5Iy3Dd2cnnZ2dqK9vR25XA5dXV3eMkKp7gCgMeYHsGv6Sb/9GcBOym+dAD7fuyUilwrmRlsCIY5pXBC1cThaVjCki1fq3qWgJDQyp/tcLlfmHGnGh0coUvehFo1rXbxSRoSCEdre1IhwJdG61LnB5QAvLgMrZWElwC21d0gWSooGeRbQXeOAeunSpSgUClh//fVjMyE8CteyT1LWlYNsqYuQdwu6NnDnmnzHRd2+rB9tb8lRxrV3XOaDtjsfghCXaYrLOEndUSG2ROqK5aCZto0m3yHZDs2W0DaXgDVvbwqm4rLbXMalrFNcr0KcbEvyHWdL4jJMki2JyzRJtt7XkyPZEm67JVviZFwKEml78TF3PJutyTfN6mnyXY3t1tqbyiTN7PE2lnrMuN2nyZIktlsLFnlAx2U7xFf67AjtufElSSZPnhyIbvxUdwBwTVA+n8eoUaMqInKgUog5+PB1AUsZvVAjzbthNCGOM87UCLrzQsGyfcyYMaqx4IbDJ7D8P9wI8QyI1MYUSGuGmXb/0QyoL6sq7aX7acZJMxAOVFO5cW0pZZN8444kcM3b0ZWxxx57iO3KnR7Pfrh6ORmm2RsAZbIjybOWpeNZUamrVgIUkuzSPdUzl7lyRpM6FQ6muIGUsnJS5llrT1/2TspuUDtB2zgOaEhyrbWZ1o5aJlrKRmvd5LR7jmdn+DtLWTct88mPJVDMs35atoiSMabPfkqZaC5XUptzICABEi37qbUrz8RRyuVyaG5uLhs3LmXHaCaaH0vtpoFe+l8ts0+PORnTP85Pa2NfW7vAUMpyUh3R2lOyB+6cEs8gOhva2tpa8c5ShlnKOGu/82fx4FmSV+fbaLvy93Ky7Msm84THWjEJJIqi/WC7cg8yxvx1TdWjs7MTb7/9thrxcOVraGgoAzCOJJAoOVItmqSRDT12g5+1jIqUMaFdOhTEAOhLH7///vsAygdPcwcqARMOCOlsL2k5Fyma5xE8j3B4VsoRVRCekdK62Hn78raVrkmRpesKicuSUJIMkAZO+NibefPmoaGhAa+//npZm/pmXkvjcDSgKBmsJDKsZbC1dvXJMHU4HOBwAE7b2NWdyrAv+6Rlr5PIMH9mEhnm4FsCHtXKML/uAhoqwxKQDJVhDgT5xjNMdPyYNlNakttQO8wDSUmGqY2oxg77ZoMWi8WKZWq0DB913poMc+Dns8MhMkzb3HVBp2WHfTJM2zjOz7lrrm15b0KcDBeLxYr2HagMx80w13oMpKQIDypp4iCJHb777rtRC1rTGcB3ABwA4MU1WYnW1lbst99+KgOogPPuFyqc3FBQxyft+fIK9HlSpE8FgpJPuJ3wUqFub29HFEU44IADyoyGZEDctRCQIQl2iPNz77lq1apYx8eXcaFtLN3Pn0eBMQXIcV2JHLy5SQMhBkNblkLqwqUG+bHHHkMul8PEiRNjQZszyBTwu3aly3fwzDTvUnHy6Otu8Tk63p5JDfHw4cPLMqNaN6G05BCXdZ5Z5Vkpnhlx9aOywCNw957d3d1YtWoVli9fHtz17RtqwLsbeXs6AOHqRMknq06WmpubMWLEiLK2ofu4LD+XTS3DT7MlUuYutNvVLSEkBRpaz4qW5Xf8okAwdBgB7/rjgUU+n0dra2sFOHO2Qere4zPutaEDPBMV0qZa1ypt12XLlqk9JDypQGWT+j0Oqn09JVqbFgqFvqydS6zk8/mydnIyq8mtpOu+wFdrT5qp04ZiSG1K22f58uUVbcfPpV4nKYtP25Ty2dHcuXMr2rgaqttPwQ0mjRs3zlx11VUqquf98jRC42l1oFwpaVcCNfAUoFDHy4Ei3yRw6VNcnpo3xmD16tUwxqClpaXMcVBnTMdGOcVqbm4uizTpnm70mhaRSgrLDQMfJ8Wjeqco1DhJoMa1EW/DuDblY0qkqF4COtTQccdBDb+UJaHtunz5cjQ2NmLcuHEVbavJqQR4+BgdV0faDcHblBo4KdOnBTZ0ozLNs1cc9HADyLt+uPOg3V9UjnhmhLaRJp88CJKCHVoG5SntDqLkAzla9pQHNJ2dnVi1apVoG3hWVdJ/CfDwDBTvhtR6AHiAI8mra7uWlhYxqJQApuSope5KV2f6HhSMh7arJq9a5po+j2ecJCApyaqU+dcCGxpwSzKqZaolcBkX7Dg5dW0bksmTsnearGoBOg2ApIy/NA5Salcp00+Bt2QHJNnV/JSU7HAAlXe503albSqBcuqrnI3VsqFcVt09DzzwABYvXjy0PwUXRdFMADOMMUHLtaRFo0ePxjHHHFPhFHmmqFgsVixuK4E3uq1atSoWxPHoVus64EaGdi1yRWhtba3IlmiATduoklBD5ZTAgQwOhinx8QsUWPT09KC9vV0UftpOUhtqiuEAtQYuQoy1A/jUSY0aNarCSIe2p6/Ly+cA3RYCfp1Mrly5MtbZaW3JNwn8cscHyGCCygUPoJqamtDa2ioCCF8bcjBBgxKtHbXgTAK8EoBYseL/s/euMZbmZdnvvaq6urovpqHfAAAgAElEQVSOXYeuQx+nBwZmMuAMBshGYsL+wGAiHwCF2a/v+8G8mryfjCbmVUwMGj/BJhjiNjBsso0GFRUMiOCAiICY7WAQYYIwzIk+Vtehu+t86OquqrU/NNe/fs+17mdVDRR7XkM9ycqqXl21nv/h/t+H677u+1lpMW57BWZczzpkv87ZreM60nGgU6b1kHzutX7urGXGjs6Do3pcx4ioDRx8HXWOFxcXU3msC8o8aKhzxto5YnX0C64ldWRvb2+MjIy0yGT273ZOg6f9OjvvVjRrLR1x8mDBM03ugEl3tjvXmVPhaFR2tjPnq136tF2A0N/fHydOnEjXzoMFT2s7oufcvLoA4YUGXWoAvZ9z7Q4b76F71lEq2gWxTEN7loP6T2f9IK4XDQFsNBpH4+4TPP5ns9n8v16UQXz/uvfee5vvec972iIqdUY6Ilo8/3ZoChEVdx4zg52hU1lKpI4T4QRaKp7u7u4UnaJSdCH0yDQzzO0cHVeI2UHWWtIJZwqeBqaOW1IXgWZIVKYMiZwQ6eNFR42y4UifR6WuJLPov7u7O5rNZhw9erSSBtkvEpUFM3WK0aPRLG1Z9163hkxjZpE9L+dAZWlMfzma5MGO/5yl3jKUxMndvDJHnKlFTwl5uj07t440uUO/V/rS02116Us5c34Ws/RauxRwllpvxx/LaAuUS3fGM0cyW09HpjMUNFvHujRbuyKDvdA86jV30DMEtS5l+ULSllxL6ct2vDzXna5D6+TP6R5ZWp3BNQOavagKft6zNfV1ylB5d7g9iPH1dCQ0Ay24pi6bOvcOMmRyV5dKd3tTt57MMFE2n3zyyVhdXf1PjQC+MiK6IuIbL+IYIiKir68vXv3qV9ciLzxU2YbvJ5Wzn2iiXYTr6UYaJxktCXp3d3c5DFQ6cji+/OUvR0dHR7zlLW+pTTG04/7tlWLgRUPFQ0S+334gcKGATDv472dKSofLjaXWMHM6GIG6cyHnba9UolJhdQY040657H384x+PZrMZb3vb21IaARFPIi1Z0FD3mTtwP4zsZahAb29vigpkqdcMnSLaJ6ct40rVyZ7z9xxZcTnK1nC/qcJMqTta2k72MsOnAIDBV7t0dh0q5XyzzFGLyKkBPEd1wRfRKK3NxsZGi7z52daaEUlx2SMPymWvHbLHs6dHlbVLqWZryO/JHN06vmOWqs7WjnZjfX29rb3IZNXR5du3b7c4Ee6I8dxyLnsBAD09PTE0NNSCNO+Vmq4LXB1Z9mxHHbKcZeCI4rUDUPzscj/qUtGZ7FFvZ0hoXbB/7NixOH78eK0M7heEevvb397Opdn39WI6gD8ZEc2IePJFHENE3K0CvnjxYq2CJCdNm9EuLVfHnZDCq0Op3KhkAurIiiN9KgQQqpMpyVu3bkVHR0c8//zzbVGU7DBnAp45hK4cXUFyTFIu7dZvP+iJG2NG/m5UtrbuVotlCpIpGkVl+m4nK+8XRfHI39Ma2botLCxERMTs7GxlfB5hHzlypIy7q+vu0xQyhUnHMUOrMhTU05WONGg8Ea1IHukJ29vbsbm5Gdvbd6tSNzY2UuTEHSD/HaYmiSbLmHD/MqSEjg0NTbPZLPdQENDb25uuR/ZZ3XrVrVuGOHHMcsgctZAMb23dLe5x5KTu35mMuQPoQUgdApoFxlrTjo6OEoB2d3e3IPh1P2eoHJ0oRzzrMleUQSK5mp/O8p07d4t4PBXn8pR95mvm9/LiAl/DbC01P65fX19fZR1oZxyZqlsztw8ue5lzqCuTh+3t7eKIqikxZbXdGu21bkTkGJDUrWW2ntIZarTtDmXdz3Vrlp1bt7euBymb7jjKxvMzP3tcT08Z6983b96s3bcXcr3YDuDzzWZz+UUcQ0TcXWhVcjEap3BmULvD63WIICOUDJUhEpE5MTQuNF4SPj84jOzcoCqq7ejoiImJiZSHVYckeETnXCFPjdfx1zKuEB05d4j3IhfXpSsyPoaunZ2dygHTHLTvmTPnafC6ddsLvcqoBET+NL6+vr5oNpvR39/flhdUF0hk68bfr3P6nJumdfOUN4OKLD1bx2Gpk7EMaX6hiJU7+fulC9ShpI7QE2nei3Khqx3nJ0tn71VkUVdYkTnNHsA698yDVznoDBb2ym7UoSy+7u3oFZJ5TxN68Mq5unztRVERGrMXukI5k4xpfJ5izQItrRsDfl+rOn6er5fe3dnLZCwi0iDAU9Bcr97e3j1RUJdPnvcsteqp/gwkaWcv3S7WyZvLZ7tzybRuRpPI0s8ZnaRurTL9307G2tnLLGtGPX379u346Ec/+kN4PLvXi+0Avujp34jdJp2Edf3wvxBBbmeMM2VJAXYDwzQCx+tkcU/D1fGlJJgdHR0xOTmZOnntDLGn3hqNRssYMwg/W68MyncDTfSUjg8Vr6MIEdXUpNaML43/haB57RDQOj4UZYnj01V34Le2tmJmZia2t7fjO9/5ToqGcv3quCaZY0fDK8Shu7u7Mi7nFXJenKfzc/w94+bQQGXKsNlsljlpjTLkzterDqmre98L6fTUt8ZPvVDnsDjyRlmr4ym1Q+q0B4407ezsFIciQ5UyHlOW1uUaZv+XoZn+4iWZaofI+d77nN0hq0Pp9oti7uzsxPr6eqytrbWcPV+nDHWjPnY5yVC4jL+VIZdcq8w5yXSXy9t+0EmX13aI2vb2dqytrcXKykpZuzrEzNH07LN2CJq+3y9HARuNRskWUUe1Qxedb5ghte3Ww9em7oytrq7GyspK2/XQPB3NztbD10VrofVYXj4Y3OxFcQAbjUZHRDwcEZ95Me7v1+3bt+Pq1aslOiRiI4VNlCgi99aVLqSDU4c27FWRqZ/dyHsaOFO+VAgZ2rC+vh4dHR3x9NNPp/yrnp6ethyjzAGigcuQUypH8vPaRYD7WSNGg7du3YqIqDiGWiceOF3Og8ki5gzx22/Fr39f5vTURX9bW1vx5JNPxs7OTrz85S+vyFOGKu+1XjJSjmLVUQooS0T9tFaSKZeH/VaX1/19HRqTIaQeYNQFFXutkVL92frul9dHecqcZQZY5JHWvfS3GRpTh8B7YOiIVbZOrp/ayVCGZhHpI2LlzlA7eeJcvBLUOY+unzJ+aRac0aGsQ9yzDMVe6FR2zuqyFZRNpvLrnOgssCcK6ugn10rIXqbH+XfOS3akvU6esqA+WyeBH3yv0+MOhjDQrws2mPFyJM+ROEfT96vHMwpPHc3Jg1UPTusyXVyPbJ1cnjY2Nvbt37S7XiwE8GUR0Rf/iyCAvb298apXvarlc0Z4dak3L1Lgu3/GjSSqpQPkDouUUwZXd3V1RV9fXyW1284A9/T0FMX5wAMPRFdXV7zhDW+oFHu4cXEh11gyw3Lnzt0mo55qq3NSRHimIq1LH2XGN6IaHXFtOjo6Wjh4bnjrlEEG57dbGykfXoz+NHYS3uscFE/bXrt2LTY3N+PChQsVRzdbG488eXlAIIeE1cWca10KUv/ndIA6A5I5uJlylNFcW1trm6qtK9jIkFBHhhl5a4+ycyWHRBxAl6HM8Pr/1aV+MqrEXukyOmySDWYL2qHm/qIeo8xkzofLTYZg9vT0tEXJKSPubGSOPteFiAfPuyNy1J9qyl03f39lXGFH99qdJwaPGeWmr68vRkZG9swm7DeLUEevqUvlM8i7c+dOrK2txeLiYssa1K1Jlk71NDQ5bjxPjuRybeSc9fT0xPHjxyty4XLiuqUuc8D7O0LnCLavC9dgdXW1ZW02Nzdbfl/6RTLDDJjLi+sZXxcHaKiTh4eHU3l5+umnW87rD3K9WA7gT37//Zsv0v0r1/Lycnz5y1+uOEk0hFp4PaXA08QR1aovChhTmHQU5Riur69X3vVz5jjyoK6urlbSEhJ2plyYimLkKMP17LPPVpxDzb23tzd6e3vLZ4om9f+MNnWQ/WBK4HkQGfXWRYx0nrke/pnWRk5nxgVxRe5G35VTHTrD9dF6cK24dpSZ7u7u4qRLbogiu+x4oEFEinLga0N54dq4Y62f2yFZrsAznpo7zS4rvl7q9cWeXxni5+eK6Sh3hrg2+5UdrVO2Nuvr6ykiSmPi6Usa+7q1aXeu9DnPFWWGDoFkR3uk9fFAg0EqHWYPTLM12kvnyCnwFLkuT1M6WiVdIVnIzlb2os6RU05EPUNA3TFyncwAou5cMVD19WGwlvHPiOZTJ7vxzzINPFdao+xctcs86Gx5MEYHyZ1G6YY6RHgvueH5YsBGZNiDMa1PZq+yc5XJDnWQ2yrXyR6AuBOpiyh2HRrsa5PpYQeCpGsIdGSUrzqdLD0wMzPzQ3g8u9fhk0Ai4lWvelXzC1/4QiUNR6Xq0TYVwdraWkUA9Moclzqo2x05d1SydJIfCHdQaGxceSwvL0d3d3ecO3eurSH2FInG6EY4Q0HdQeE68eWkXioKcuayNclSIm5gfA38PTPAXBNHQd2ppYKQAs2cEhkTrkmmJPT7t2/fLp8pJZwhWRG5YaEz287gch36+vpqDQsdHUdrdDHVnp2ddo5anaPvBsURCipMyoin9n8YJ82d+3YOLNErKXGinRq3r0mdg0aZcefVAx86H3XOWaPRKM4TkYYsfZjJiZ+ZvWSEKLDOMWVE6K/21LMF7mDUOWQuIzqL7pApSN5Ll1DH1gWClBkPGh0BdQTYuYmeTckcsb0c+Sz443c4Ncb5iZ5JcRmpyzDVBX1as70oDBlg4FkCpy3s5ZzWnRmuSZ3z5ecmS8M7NYNnol0go/Woc0rbnRvKyM/93M/Ft7/97R+6D+ChAxgR58+fb77nPe+pRbgYbWYH2An8WRrUD68bO/47E1jC83X8P09n1TkD//Zv/xYdHR3x5je/uUWJZYiW/k7fUZcKZYTpSq0dn6YufZ6tBdfTI8s6zlGGaDGtV6fss/csHUol72mbOrTGOZAZovWlL30p7ty5Ew899FDtWlA+qOR5DycWy0Hh/tEpyJSbc0EzVK8uyib3SutCdI8IsSv7bF2cM1O3Hu3QGQYWfnaY1iN/09fF5cbXq47fSEXvhHQnlzuHyFPA2UsBBD+js8gAzlN7XBOti3PRPGXpKfCMSuHpcv5t5iB5cJEFXXUcNNcx2asddcBTwXSStCZ160InkpzGTG74eZb+rKNU1PHNmGGhbvQ1qVsbfu6B1n5lxc+Q6DiOlFNW2q1Pliav4yrqRceWdrku7etoed25oePINalDNikrTjPJsgcefGR0HH3+u7/7u/H8888fOoAHcf3ET/xE82//9m9TREMCz4OwV/pgP6kVj1IVmbmy4cH3g5RFHxnq5am4d73rXdHR0RGPPfZYi7GmoY5oTcPtFY3VpVFeaMQeES3OrUdiGR9rL6Rrr0jM00pZZMpxZmn+DLloF53yb6Wwt7e3Y35+PnZ2dqK/v79FoeyFbh09erQ2MvdUiRtrJ8+7o+Jk8Cy15jJB5CZDxt3BpwNIR7Yu5ai9c+PiDn27NclQvr0i8zrqx35RTyLj7ZCKDPHk2chI8DSudUhWRmfg7xL98aDP02Z7per30pd+PnguMme+jivN4jeN2QO3TBb2Qjkz1Ned1czxcEAgQ/PqaC7MHHlwU6cvM6eDsuwBXYZmCgnn/2fOWTvqDwvasuKjvagtriezLJrkq85hz+wGQRHqvyyl3A7FYzCjtXBARDLB4N/rCSQL7bJoLg9f+MIXYnFx8T/1k0D+l7lu3boVzz33XGr85AhISCJau7zXcdrqUhn8jAqhDppm4YO+t66CNYskmN45duxYLC8vR0dHR3zta19riTBIvmXa2dNbEvbu7u4W/hoduizydiQvU26OdmYQfaPRKI7B5uZmHDlyJFZWVlqiK6JaddFmXcRJxMbJ2IzCG427LQpkbPr7+1sQvgydcNSCr+eeey62trbi9OnTLVGm5MEVXUQUB1LE72w9fK7Zv518nZHSedFJbjbvtk1R4NLX15eSr7k+/rOTrbOiDiERvFxWhejcunUrVldXW85Lhmpx3125eybAyflcC+6VzlRPT7XpuSN8ngrky9fAOa5u/PSu/drZ2SkI4erqamVOrkv8M/6eF25wzSmLLhtaC+navr6+CqqXIXxeMOSoDs+EozAZwKHxbW1txdraWqytrVXmwnd+7nOnHFAfO9rN9XBUs6OjozhalBc5Ts7V83Xy//OzwWCN68+1aTabxblYXFyMiPoWK0Q9fX0yufD3bF20JrJPw8PDxc56QYfPLZOFTEaIkjrCzntoHdSEn3vm4/fsUp2MZD9na6TvlwOayYv27+tf/3qLXP8g16EDGHedmZGRkZYNitittnLCbGbQM2ePXnuGfu0V2e7s7DaudPJwhgTWcREYufT29kZHR0c88MADxfHx1JTmrzX4/2v+jvb4/D2i3Q9fx6PYLJptN/+IVqQnS1HWRbP8d+YA181/cXExms1mrK2ttfBzftTz91RtxsnZz/yzn/c7f+4/Aw9G8XvN3392XpL+/kcx/3ZnwOfv/KyDmH8mB07rqOMg/ajmz8D3oObvaee6LMCPev5EcPYqzHIkK+PnSe/8oPPv7+//oeevuWeoZh1FYz/630GP/czf6U372X+tgSO6Pn8FZ55arpt/BnLUyf7KysoPNX/tO+c/Nzd3AJ7PoQNYLiIGnuLROxVAlvqqe+0H2dN9mWISslTH6/O0X50RcF4jHSpy9jSmDMVwYn/dvN3B83W4detWhYulaDIioqtr9xF7ddyJDMlyZyZDNTNkL2K32uv27dsVVE97nvGwMg6J80Q8fUc+jhz8zs7Osic0dhF3o00haPfcc0/L/B3t5dw8fZkhmTs7O2V8GVrDuWdoVR2Cl/GGHL3SXh85cqREujwDdPYzhIrzqXt35IrfJ0W7sbFRmXMd0kClX7ce2TzJydJ3d3R0FIPs6C0RtHYoVDs0ztEoR1ski2okS1n3udchLXX/dgTOeWKdnXefuNTf31/Rsxky5GN35HkvRMlRSKI7vtfZvmfr4Qidf0+7S+dyaGioBSnN5iE9wHRi3dyor/1dY1tZWYnl5eWW+bZ7J0rYbn7Z2nP8Sq97BsX3zZF010t+TrN5+GtrayuWlpYKnSbbR0dH95qjO6QZEqiz2tfXF4ODgynSl61btn+0wfr5fe97X1t52+916ADG3RTRjRs3ijFmCpTKVhtEpScj55wfRoJ1uXyPiskL8JQnnSIat3Z8wKw9R29vbzzyyCNx9OjdhtBKa0swaTwpmDw07bhOdTxIvXd0dMStW7fK9zCapBH1VFYd/0+clnacHjkZcqCU1iOHo47TQ2OmsdHZI3fD5611u3377gPayRt1QrEjv41GI+7cufskjOeee66Fu9KO0yVHg/wnvrh2WgsqpMzxb8d15JwjouxlO86nV99ljj/pF6xWdc6O5i0FLJSzjteXpfUjWiN9nUdHuTLultYoO9vOa8y4fHX8tXZcX1Vt9/b2FmQ/C/wo411dXZVUnuadBXn7Odva56x1Fc/Jfs625s2zqYCu3dlmZoMyvtfZZmaHQQ33uu5s672utVA7VJv61hF9Bq51fM2sldBe3FU6GB7YOn2pHSfPuzm0O9sZJ4/z9qA14yWSryx5z+adIXl0sKjTnIdXx2Ov6+ZRJ+N72W3q5HbZu7q2Wo5mHsR1WAQSd4tAPvOZz7QYAkc73Aj4wfCWJ64UHQGrS3W4oIiMXacI61pXEPWjwefh0Hx1MOSoOH+Pgp9VMHt6S/PlYXOj55GTjLOTtescPBo7ztUdeEZddc7NXuTkDMJv59Tou3kRmXEUj05dZuxdydMpomPMuVKWPX3lRRucS/azp2v191xHv5wLk6GWnC+Voc8zK1ai86Yrc2YYWGWFKhliT4VeZ8jpxDgC4AVb2ZwdxafsOg8vm6sbchLt6xDq7J2y61xLn6fOre+r0xJ8Xv7zfqo6fU8ZiLWbax0KTwfc0Vp9vy4Gv86FrEPg3RFpN086o55xInqVndmMb87PuC7cU+e8+Txdfumo1FWr1s01S6dme1qXXWq3py632Tmtc8L8rNLOMntCncxgoi67kqXO6/bUs4ecT5Y59N/Z2tqKL33pS4dFIAd1raysxBNPPNFS+SQlPTAwkDpOTIU4QkLnSE4hX6urqy1O4+bmZqWx8erqaiGwK52SoYB0luQ09PX1RV9fX/T29kZfX1/09/eXz5aXl+PYsWPxspe9rBI1CzkZGhoqSIEOTB0HLouasrnyc4+kZIip3CN2U+A6WI4OEOnkXPXzwMBA+Vn/5whpFilLOVIx8TDSAV5dXS0BgPaUc2YUKUfaOR80PkQFNGeNW2PmfPjq7+8vn3uFZ39/f1FiTC87qu2ol/ZI++b7yzl7f8PV1dWKcaKxpSzTsDr64fOlHHPPGRzo76nYmZbSetNoOKqZnVueWZdlzdURr2xvFexobN7w1/eWc/aAjwbKsxY6QxFRMTx0+tudWz+7+r3V1dVy7mmwmUrzYJZnV3OVnLocuyz73upFh0ryxGCWSKbz87i/lGH9O0P4yPuig6x70yl2pJ575jLs8pzZIDqQWlc6UXSeHLncjyxzjzNudoZgkpKRZSXq7I/PdWBgoPyNI3nS/ZJlOsaaqyPv+7VB4uZpveg0OzrvFCwGrkRmfX60QZL348ePt/CQubcZUuv+xVNPPXUgvs8hAhjVRtAZ0ZWHyQ3A6upqOUx+qBwqloAxl58pSoeEs4PEdypKKg4dJDoUERFvfetbIyLir/7qryrOjcZKBelOjR8kR8YcBeOVpXnaGb464+fGIEOFhBbI0HuKg4qRhiBTFG4IqGwZaZKvVKco3DnPHBrN8Ytf/GIcPXo0fv7nf76C4GYFKu6oZgbA50blTzTbEbC69BXT0lnKyh00zZXG3x02zXO/wYfm6Ug850iDzsAjQzJ3dnZa0B/O8ejRoy0IfBZ8ZLJKNJ7yqosIVxZQZo44A5CsUXKWgtW9nGNbRy/g2eS7N5t350xGvC7bUOd8u9PiPzPrQN0q54z8w0xe6TR725NsnjqzDB59H8UpjtjtTpDZkczp9v3LGsVnDlm7OWp8LPzSXnlQXBdU9PT0VJAup4lkZ9J1T5Yd83l6RkV6THLCy5HYdrqnnbxm6WOCPDyTdDh5JjM7wsDf55idSfkDL+RM9vb2xj/+4z/GjRs3DhHAg7gWFxfjC1/4QuVJCHockxZ8dHS0RTicK5QZXhpdjzTdWeR3rK2t1aJhdU7UXs6FPl9dXY2Ojo64evVqJb0spFMCSAidkf5eEWbdQdAh0AFYX18vxOQ6hNOjS6J3jKrq0C+PtPr7+2NoaKjCgYrYRXOJgHGe7hhnxknzI4x/586dWFpaips3b5b77OzslPt7YY9ePT09MTMzE11dXTE3N9eixLxnmV5ar56enhgZGUm5XnJc6ehl3DZHMN3wksuo/cw4Xl693Y7f5XNrp7B7enqKzEpxZqmluvS+O4V7zfHOnbvPu15ZWWkp4IqIFgOcGSgaKs6NSDzn2dXVFQMDAzE0NFRxMlxumQqr4zRlXDafowKcxcXFFj6T5IgoI9ODnknwF+fOddHf9/X1taTRGo1Gup9OyWk3V2VVOFfxd1dWVlqoONrLDOWqq7jO5ql1oP7p6+sre5mlC7NKY7ctdfOUjmJwdOfOnVhcXIwbN260IFucJ2lHfj597zhP513qXdmVbJ5O02Ba+4XKrfZTtiTLNNTpIdkV6lx/uZ6lgzwwMFD8Ai9Scs5hpod8L/fDs5yfn4+ZmZlYX1/f27HZx3XoAEbE4OBgvP71r29JEUnRUOAYubWDmj26cQRQB5FIh5yA/v7+FqcuQ8eIALqhJHeDJOBms1mKIkZGRooiXVxcLEKZzS1DN104mfYiKpY5rUNDQynyJ8dVzqs7PDQaVDBZWp7Kc3l5ueLE7ZXGVEqeHL8MDdPaEhHQ3g0PD1fGz3lliB+VUVdXV/zN3/xNdHZ2xjve8Y6yd5IdGgftw40bN1KE2lNb2mMqIy9EiYiKI85oVOM9fvz4vtI87sB1d3e3pVRkqTudu+vXr1eCjLo0pQdU5LjqYsqOhGwq/6Ghodp0nZPSyRXK0lc07gwS6HjOz8/Xpq2cW0xUiClJd9QciSZKMjg4GJOTk233rg71YpqZnGlH9Ry11ByzNCQDEaJ6WcGM86Qd6dKZm5iYqMinAkbOzZEgOi1uDxQc1qXQFxcXY2pqqgWhpcHfrz1gkF+XLenv74/x8fGWfWNGiPbAOdEMIpzak6Fbq6ursbCwkFIhmAnyogjtnebmRWoZ5UPzGx0dTdPkHviqeJF759QWOo/U+dSVKysrMTMzU0Gd+XdEsOvsAcGL/QA0IyMjlfm7PdC5azQa8eY3v/lAfJ/DFHBE3Hfffc0PfvCDLZE30zQsh88q5erQEz8g7kzyezx1EVHt+0T0hIpcgpUhKG54u7q64ld+5Vei0WjEH/3RHxVFkFW7epFAu+jTncEs9bSfyt66AhA5fx5xkjdCwytFoIu8r6wwwAsfsmIIJ5XT4fV9I4zvVZ10Nhw54eurX/1qHDlyJB555JEWArm3DXB+DOeXVfnVkcidOF5X9JDtXx1pPCOLe1EHzxpT3I5EexsWJ4I7ET4jxdPZdbI456fzx31ksRLHr8+U/nSCuL6njvzPPfTCDu4H98fn5MUqTGtne0g5ZaDGgM0r570Qgk6mz8/nyTk6f83nm82Z++bzpCOTXVlbGe/y4PNy2aQMZPPk5fMktcj31QsDOG+uT7afunR/Il3UhT4fL37wc0wdykBtL5n14iuX1axAxfeWc9V9OEefnxdzMFj1Yiz/Pdep7YpVsn3LilX4yopUnEPqesdtBxFL6dC//Mu/jJmZmcMU8EFcR48ejYmJiUp0JMOTVZtl0V+W9hRZmrl/VvKI36DoiAIpo+mIRDs+jke0rIZlRCsD29HRUcsBdCSwHQcnI75rTlSWmXOXcUURvnwAACAASURBVMV+ULK7onQShJ3sXjenOj6j9jyr1s5S8Xvxphw1YtDB/W80GnH9+vVoNpvxwAMPlLWtI/BnaXimcH2v6pAHj17peNdxiOrQIkXmWicvOpEi9DSyULG6OdWhKUL6pFwl4xk3igEU6QMZPyqjEsiI1KENdSmtjKPoqLrmpL+n4dGcnCZBfqJz2ohAZ+fKKxtdB+6nqGJjYyOWl5dbzhVT7B4ckuPFc+WIM9NvGvfAwEDlPGXIUBYYMqCo2yvnr2VcUp4rps6JClGvZxQIcrooa0TWnSrAYMp5pK4D6zh53J+lpaVaTp4H8gQonMJCWgbl79ixY5VsTh2XW2uScSqdh9euiIlna2FhoZIRqOMZug70gJ17lemHvr6+GB4eruh972KgdYqIFh3IwNXPleuItbW1kuo+iOvQAYwoKTWWiEdEGmXRQ88iLUe0vM2ASOYZ/4PC7yXnHp1lxGodehZBbG1tVSLfnZ2d+MVf/MXY2dmJpaWlWqXujiur/HTwZQA1V+eX+JyIAjElRYhcnyv61N5ERHGAiO614wg56pW9SxlE3FVqKkY4duxYy545P9GrOanI6Cx4QU6zudsb8NatW6ks7OzsxNDQUOzs7MR//Md/tMwrizxphLRmumdPT0+FUE3CsVdaUwF69OztD/T7W1tbRdFmCKX2L0MqHSlwlJa/32g0imHR/vCiQ0ZnWuN0xCdDdrQWMjRqmqzvJ5omucuQHv+3f6a/OXLkSAwODsbAwEDLXLx9Bc8yA7u6F/9+Z2enpLm4L5yPz41nIEPwMj0ZEcUQch4RUeH2+fh4pnX5ufAzorM8Pz+/51wo+375PX1cnZ2dMTw8HCMjI+n/142/bky+ttRhKysrsbS01PJ7juLuR04c3eXPCgpcpnw+Lid18k5kiw6bMmQMODmvbB8ylM/ta4baDg0NxYkTJ1oQ38xeRlR7+u4HtZR+Wl1dLTzZzMZTjugrOArrGRFHLhXsUH81Go3453/+59pz8UKuQwcwohw6Ro3aODk5XV1dFSdLjkVvb2/x1nt7eysFH3IC1tbWKpFaRLXBsL6Xwk8oW46Ropn+/v4WFCdiF/6Xw+JpG0UgP/MzP1NJh7oDKENO/qDGqsiDKBs5cnIUI+4eNBJvvfpOERPRGjqVTsz1iF57RIVSV3FHh1JEYSkmtrXY2Nho4R5xbzRGJ74zOpS8NBp3nw9Mvthe3A5dW1tbcf78+RayO9FZOZC3b99OkT+me5kezPaG1WaONjcajcrnjHz34uG04y0qwNC4d3Z2WlAX8seISDB610VukRcOOWIpQn6GYNJ5z+gEkv3t7e1KMJGhlirokk4gh8+5s3TQI6r0Dy+uqEOXHU3i3mhtnHfpe1OHrHgRmxd3EYmlo0CnhOR7ImBEKTMuos6Xo7COgGkuOt9aYyJ3e3GbnetVtzfS01nrojpubMbXdvRVzpfOTYYQ1XG1yTsk+spz43xR3xuirj4XzuP48eMt/GxWgZMnKlljYOd7w7PPvVleXq5kacifzfbGectZ5X4mZ0TI6/aGQEpWZJahxxmf19F+nR3SbjwdTPDkoB4Fd8gBjLuNoD/1qU9FZ2dnJULLDBZTvdpYbwNDgW7nTLijx1RHVhTBQ1hHziaaRBSQjsS3v/3t2NzcjImJiTQlRUK2lDt5DB7leNqGc8lSUVkKYK/Cjr1Sa0xN0zCRMM/0Ox1Vn4dD/+6IUoFnToNHa5qHDrZzR6gIOeabN2/Gxsbdp6fo98gpkSPCi06dV2jWIa5MLfk+ZGgzo345l0xZ1vEoOYcsXSalp+CFaDgNLQ2ulDvX31FzBguSJ+e7OmLsyLFeRF9ZfOFGiMiLp8myuWQ8Se6BBwiah+6vtXRUmOOmA+1oqyMydfvgaUw5c/w3Axs50c595J5TljP03sfOOfAss4Cp2WxWDDZ5jJk8cR580QmvQ+q9ctfHzzn4PJwX5kENkTCXI8+mZLJEGhDPAnWr1o0yRD3r54D61GWJ9AvtA+fgqVXnQGd8aD8P3FfXSdlZdv3quolzkRwSfSWSJzlidscpTdStmWPsQbKcfdkI0n3qKoY/9KEPHXIAD+paXV2Nb37zm8XJIuKkCioqLnKXyJtjldTKykrlJUdRv0MenQ4QnXFyAZ0r0tfXFwMDA5WXnEQ2nqSTogMUEfGud70rIu72AcyqvujQ+jxWVlZaqhKlEHT4eDFlSKRMa63Ul+agnxmZ6W+okHW5Q+XRpNa8bh5ZZOlRmAyIDi7RPM2lv78/BgcHK++ch37X0Quiq+Tt3Lp1Kz7xiU9Eb29vvP71ry9zWF1djeXl5cq/6bwTkWWkT0NIw0HkRWP2OXhDbSo0KXam4skr5X7wHGgOevfG6DSWTK/QUWewkc0jOx9EZGhgmFKlMXFknOfb5Ypn3M+35qEAU3Po6OhoQZHrzrjPQ/P2wEMyK8dO82CwpEpHPx/i8emzrLKTKUlHWzyAbTeP3t67jXIdqdQ8dNHJpTEnukJ58rn52aDDpYvpOe4H17tuHgzK9Xd0HKXX6aA4L5T7wDOhfckQSs6D6V86ieRIZjrX5corUIV8aR5yjBj47aVv/bwwQGcw6/NgoEG+Ll+Z/RgcHGxBJZlZIjpNR5HgAeUom8fS0lI5S7KfTAU7sqoz6siw2z3KlObR29tbceg7Ojrik5/85IH4PocIYNxtBP0P//APERGVSJQokhvg5eXlWFpaalH+dIrkTEgoJBCMgEhgpjAPDg62CDcPKCNAR5nonBK1lEC/5z3via2trXj7298ey8vLxRmSc0rUj+gGFYwbXxdojZ+f0TGlI5QRzhltMgXFA0lFScNLBekpaU+raTw0SO2UPJUj0Q06QIxA6exnBrcuMLhz507p3TUwMFCcUaIxnnqWkXLF4m1LsvRzhnx7uskVPMdO5JtpZ6IaRIKyQiCO33/WHCVziqY9PUsUhvLviL1knoGA5inDpvFLR3rKPHM+fR7ezihzdIjGeIqMZHCOn+vuBRck7jProPWn7iAaL1mhgfVATGuv8ZMvRoeEKAsDrWz8bE+kM890WERUAhinhxw7dqzt2L3lC4t49L1EhIl8eVFf9vI2L+7U6KormHC54XtdWlLc3iwl6ZSRdrLvdAWOX0GwdD/HT7RL8q0xS+9z7al7ZP+IqGr8zCZka5/pHW9Dw0CFxYhZoOJBfB2YIvnXuacuIx/SC1QyuaHu5/l1GoVsFgNGrf3TTz8dq6urPzQCeOgARsT58+eb733ve1s2nF43OU1U0uQxeRQn50rvNDbOndva2qpEP0zd0UlhtOCRHBWGEEwZSh22ra2teMc73hHNZjP+5E/+pKKkGfnL0c0cXCprGVtPWzCaPmgnS+kK3V9Od6YouO6s0pah5PjpZMlQSlkwde2Kzg1OHRIQUa3U0xwcmdFYv/KVr8Tm5macP3++KGymiKVsmMKTDDlnJHNUHBWj7BABaOeouLEhOky02DkwTLvQWGaVeJmjSOPusuOcUU9lu6PufFHnILXjiTElr/11R53Rf8ZBctlx9ELnt6urq1IIkHUmaMdzo6Ehp5LpO45fHFZyQ4lSOj0lc1bqeK4s7KnjuPrYvQpXzgodFl7usGQBk69/nQzVccActa/jfzlPL6tKpcMlPUoZki7iOmf6J+PlkZaie+gMECHOznHmsLseaidDdFqJSNYFS1nAoX10ahD1UF3AQVuQ8Vf1t/o+pn2ZsqVceLBBx91783oRoJzeLNNAHVMHGKytrcXnP//5WFpaOnQAD+J66KGHmo8//nhBoSTIzLvzELhzR7jYCd7kWBABlHIkyfb48eNFgI4fP14cPE/7KKVLx0IODHllEiR36h577LHY2tqKN77xjeVzpkI91ROxq0xJEPb0p5w5zcOdUiliHgTysfwQM71DR3ppaalyMDLSdp0zxGIAOp9c7yz6pjMRERVZkSNH5e9oMcfviB9Tz0TLxP8bHx8vytDTHxpzhrbSCLO1AmVF96bS5FiFdHO9qTzJ/2GhDMn+NFaOEPuaMwCQA07+XjuEu13wpXllKJMcT73IR6KTwLXmu/aDfFDylRQckaPnqVii2i4r3BtyiuUIZsiYULE6WXG9wjnLcHd2dlaMlIImOgg+ZpdxOQuOLEXsZhQoK3TOPCPici5DTLK+5ESpfMmnc7j3Op9ac1IRHFHKKDquD7M1p4wzSNR6s+CO9AONi4F5loViapIBos6nywr1YV9fX0UXcg9oq+gYUx9SVjT2drazLnsmh0w6kVW0DKjoRHLMe9lOrYHrQ2UOMrqHZ/3Y9sgDEi+20/3IJSeQkJ1P2iDy5Ds7O+PNb35zPPnkk4cO4EFc9913X/MDH/hAhRckfpNe5KE4Z4DKkIJehxpkTgo5D3JUHC3IoiSPtrO0mHNPfvVXfzV2dnbit37rtyrcDS8GoQP8QpAOj5SI/jlCIKGmU8h0sBcaMEr19B4jcCpuOQ1MJ0Xs8izlaDA1QNTMib1ZJaI4S44OsGCAyo1rm5GS79y5ExcuXIidnZ04efJkpVKapHBy+li0wvX1MbOqTU6Krqw4ggVE7UjhlJGM1M79ZUBBSgTHmRVH0FgSkSGqQWSGDjrPrhcZsACB30H9yJYO7Uj5/tL8WDnKtWDhAo0oU5JZ2x/KVkbEZ1qfBRjeUoN7wHPMsRPFy4qDsvF7gY3OAPeBKCblhrxPrX1Eta8oZZnr7eOnQy4DmhU3afwcO9fai4H4zkwOiwiIalN2fM15Rv28Eon3oiAv5qDT64UQRLt4fvU7fm619hkqnxVAtCsuYzGKFzQxyMjGzEAv0/G+/myV47rGixSZGfECPx8/dQ6LRLxghsEG7ZNz/91Gua4k+uhFJp/5zGdibm7ux9MBbDQaX46I10WEMP+pZrN5//f/779GxLsj4kRE/ENE/FKz2Zxv930PPvhg82Mf+1glgoyo8om4qUwpMhLQ5xkBnJvrPDQ6ep7m5WcUUhoVKjEJk1AGOnga31NPPRXr63fbyWQOn+bt/BUdBC+nbzdmRjxZOtGdUyIjGaGYSI7mpvnqEEVUSd3kCzl0r2jdOZaKupzrtJdc6L1OLvTaj1w0Go3o6emJsbGxFo6K80L3IxdCWV0uHIXyVAl5ZV4F3k4unDj/o5ALBlsHIRfi4raTC6eGvFC5oEyTkrBfueAaZ5SKg5AL6gxfZwa1P6xcZHwrcvVeiFwwY+C6g2fPkbGIVo5Yxs9zlF2o2A8qF9kaa/yUCyJLP6hccPx1NInv289KKx1WoDIF7GP2QpV2cuFrTA5hHYfZ5YLpYPJP9ysX+jfBBMpFRBTk8UcpF1n6OrN9zhV/4oknYmVl5cfaAfyzZrP5/9jnr4iIr0bEmyPi3yPiwxHR0Ww2/0u773vggQeaH/nIRyrkTqY/6tKTGamZUHbdQfBKRqVG98NN8aredhXK5BVIsTMacfKsV8GS1+fKhmgZoyiN0XmIjP6IehAt8CiK0V6GlLG60jlwWeRHPlk7hIxIARHJiN2qY283wLH7iwgII3VdjLAzVKbuxQjR22xorEKCiCgRMcpeRGE8ov7+OWtBNDJUif/mGLOxarxsNsu1avcizYLIhemGChLjL0cUibAoKPQ2LD5mvlgpm730t3UX71X3817XXrpd38V3viKiZZ98z/x+2RrobGb7RIeG+oZj5N5xn6hPiIyxMr2dnPm4eEaIhpEqQHSVZ1gX7+/nmeea/yayShnVxXE6Ouk6yFFhZpu41kSDvd2So+91SDx1pCOpXEci7nVIJBFVrTkdR3LyMl6kZ25ct9e1kaH+4DrS/hC5y+ylxsyMgcba0dFRyWzQkZUz7txf584yaJMM/NIv/VJ897vfPWwDY9d/i4hPN5vNr0RENBqNd0XEU41GY6DZbK7U/dHOzt2O8q78dTC9rJsH1NMeNKSKpqhMFF3o4PGAfn/MBaVgFSI/98PpHCkpB42f1Z2bm5tx5cqVWF9fj8HBwRbieF3kxnRLX19fUX4ippPbyMieB5LCTCI7q77kHMoRFT9N0RtJ1FIiWaWpw+b9/bsd77u7u1sQS0aZUnJ0AqjQnE+kfSeSpuiNioNpLnKV6JySXL+5uRn9/f1x4sSJWg5RT09PSc1IVvWiM82ARUFMHeKndaXhYNDi60r0zKN45w3JwEjWiVozWPGAKitG8lYSt2/fLmsqXhy5ts5DZOU9gy1SE3SGpCec9O9FL6xOJ8pHfu3mZvW530wlM6ByBNWRhqwNDIMrOglMRzkK6cgIqQleoMPAVbqACI7zgcntzApbSK/Z2tqqpPzIeSOC4zy9OvQmQ8i84tPHqXcFsc61Jj9PiJ7zxfgif9aLh7S2kgGltb3gKRvr0NBQS9GNI2N0xBhUO8JELt7NmzcrvDam6bWu0vWyRbq/ryvP1tDQUIWHRzqHxkobdfv27TIOZn48u7KwsFBBHQlg0M4y+Hckmmi/3oeHhyv8UrcH2r+sW0jd2VpZWSm9XdfX1yvOPGlJWhsW9LgMrK+vv3DvKLn+MyOAr4iIRkQ8HRG/3Ww2v9xoND4VEf/SbDb/T/zuakS8odlsft2+439ExP+IiDh16tSr/+mf/qk4S4760SBxY726lwZUwhGx25S0jhhfp4zqCKBCJ1l63g5CpnLf2NiIr3zlK7G9vR0PPvhgSuBn9WWWOmUlLEvk5ZAyCnZ+Bw2npyGJThLtq0MkfZxaVx4cKh0hp1kKwSuysiiQaRk6H2wZ4siej5MvR0uJRHZ1dcWXvvSl6OzsjDe96U0tCp2Rv3NlvMKQStzHKKVGB38vLiFTGo7qOo+KCp3cQY+gZVCJOuKsVviaGWdQY834gjIyjKJ9jHUoLj8XMkEnm4iZc6TISWPAxrWW08Y0Z11QyfFlHLSMA8h9JirOsTkHzfeXaVgaa1aR+j47X5RzcI4f+X10uLSflEVmTMihq1tD5/IxKKpDoWiYPX1ZV5nbjgOXoU9EQbNqblbZuj6ks5sFlK5zWPxGpCmr1va9zqqdGahlHLcso+RBhM6y9lljcqeH6BgDM+15HTVAutArsWlrnIvnPHEGvOS2O4WI3Hyv/PW0L/ea6+dBGWlmLI7Rvj///PNx69atH1sE8J0R8Z2IuB0R/yUiPt1oNF4VEf0RsWS/uxQRA/ZZNJvND8fdFHG85CUvaT711FOVBpJ6SXGSYCsh54YtLS3F4uJiLC0tlWohRSnkBUoBSiCYUmXad2hoKI4fP155MbKWohkcHCzCGrELEXv1GHsXfvOb34ytra24//77K9Efq1Kl/CS83oZATquPb2hoqOLI6tDS4YqIymHlIeA4uZ5EALWORFpZwUl0SpGor+Xx48crB5dONo2I7kElxzHyla0lHS9XKqwGUxUYx7e6uho9PT3R0dFRCQaEoOgiIuGcF45zcXGxUoFHh5ZONlFUGgtWq0k+NebBwcEiv0RRmcqiE0NkLxsjgys63DJIEbu8LSpnIpBDQ0OVc8TqOq8ehV6oOKh1a8k992r0zc3NioNIh0bGzfec68k+ajrnMnpMnepebBmRjZGy6a07iJzrHkQgM7nUOIeHhysBF503Otl0aDTOujGqypIVuCxoYeGT7s0qZ41vaGiofHbixImKA6QxSq/v7OwU54vBvvY80+s3btwoATWrhCVDdBCJ4NSt5cDAQExOTlYcCBZoMdiX/SEYUbffMzMzsbq6WglmqNdZzCGZy/Zcel3Io/QXkTHpIs+S7KXXFxcXY2pqKtbW1lrAEwVWrPwldzvT6319fTE6Olr0ugJ/Is5E7/TSeDQm6iLtueyA/l7OtzIOCk5oI11ncs/Hx8crIA+pC8wmyvZILt/5zne+IIep7vpPiQD61Wg0PhcRfxcRb4yI/7fZbL4X/7cSEf+7I4C8XvWqVzU///nPR0RUohx64wsLC0UoFhcXY2FhoWKoiLqQ5yND6EIh48QXS8EJlesQMA1JgiuVk8ZHRcCIZ2trK6anp6PZbMbExESJxBRhZYaJY5OhpzKV0ySlz2jVlZSPjZEZkSCiAnKQZeC5hlQCTtonIqmLaAAjLCokjTNrNSNFr/1llRZTYdpHOkkkNbtjLJ4h08yf/OQnY3NzM173utdVWuDQOZKxzFJ2rDplWokOulr2MF3D1KL+PqKaAtV6tGu/krWl0PwiouK0a3+ZmmnXisKLBLQfdDJ5RryFEIMzGR45ReTxEM3T2nhaVu9Mh3qqUygziwEY+HjakMVD3hpDLyenZ/vLfaXsySjK2WfmgwhKli5mOl7jk27J0sWSdWY13Nnl/tLpZYpQ6ycHnbpjP+1cZLSZ7ZCOcNnzsyFwgLrFnR/XLS57dNSy1ieiBjjthtwx6mHqaafdMGOU6ZbM6aEOZMCgl76DQaIHDJmT63atv7+/gjhKt+zs7FTax/iZcEecwbbbDtlM8usJWsgho1NG2orOL20HMwXMrmlcbnupn+kb+P66860x0Teg09jT0xO/8Au/EN/61rd+PItA/Go0Gp+NiM9GxGRE3NNsNv/b9z9/SUR8NyJG23EAz58/3/z93//9Et1IkTOidQSQ3jgdw0wIxKvxCIycj2PHjlX6542MjNQKgA65lJcuwvTkpCiS0dj+7M/+LLa2tuI1r3lNZXxyIDza7urqqnC9/PDovb+/vzwgXAZJcL/GJ3SS/A53vLR+hMKJTApB1Xcz/Zs5sHQWfX+1x0wZMU2t+2dIBZ0ccnuYYmWqjelzOoaO7mp/H3/88ejq6oq3v/3tJW0kB0JGtx1CRSXl49va2qqQ1FmRR+dB46MzkaEp4h/qjJCSoDVyA8MefTLSOiO8mJ7MHMS6Ho5MwzKl5fwtIpGZky0jwzMiVIppZsk8EQo30kTwtebaB5LSSUHInGzvz+gcXjrZjUajkkb1PfYX029ZlSgJ85mTzfNBCorzSyPuGn6eQ3Jgucc8v/qMKVfXM0TLnJ/HM+LOLPWM0B19t3PIHIXiXjMYUBCh8akoS0GH6xmeY2Y/skBU31eHOEruMvTJkTymp4VusXjQETIfI9PApCOwbQ/PiGeQfIzkjtJZlH0iRSsDGgTSLC4uVuSQtlgyzTPCII+2ji/qamaP9H0Z0CBbTBu3sLDQss+kPskxJu2lr68vnnnmmR/PKuBGozEUEf9bRPxT3G0D83/E3VTuT0ZEV0Q8EbtVwP93RBzZqwr44Ycfbn7uc5+LRqNRQQBlWIkAzs/PV+BhKWH2JdIlgj+dE6J/w8PDMTw8nAqVNlvOiQ4lkTUJjcbGdxcoCWOj0YibN29GRMS5c+eKQSBMPTw8HMePH684oUQQyK9iZM7UhIR6YWGhBT0VwqGxKbKisiWiQWfO14ypXClbRb7krhBtkeLS2Obn5ytKTUaNqAEr+8h9pPLi2PQZf5epJ7ZZoIHi2L7yla/E+vp6TExMlM+lkGXkhbbIgDItJqM5NDSUBhREcsnlk5PJVC0DCMqZlK2Mk6JxOYNMLdJ50xh8P7XXDCLISaLzRseNY9P46IjQARaCS3SUCIH2kntKA69xOUJAnqPOHo0R3zOFr7RXRBRkzwuW9tIdRC9Y+eqonhtKnlEiL3LKiawQ1aOj2+58amx0yFmsJN3hKbhsbDLkQmzpqJFjrYBU6WruK8dGJ0MFSgqEmcqULGX7KWdSeo3UDzpATgHgGZCc6RzQAWIhis49AxgGz7RTGi/RUHKBSaNgU2UiUBybXiw+EdLLDgnkdBMh434yUGVhhBwfci39fNaNjedTY5NTy4IoyVC7scmplQzQIfPAXragbmw6I46+6xywEJLIovZRNkF7+i//8i+x+uP4KLhGozEWEY9HxAMRsR13Eb53NZvNf/j+///XiHhPRIxGxBci4r839+gD+NKXvrT52GOPlRRPpkBZGMCInLwlR4WkCORIeKUPDY/zWHQIvepTSkr7xmpPpkWIaBDNWF9fj5mZmdjc3Izu7u6WSJetaYiWEb0g8TWr6iLBmaRrogVePOPVh2zLwKpsRml7tczJime0Zlw3OohSDvy3k4WVBqFyJ7GeKS+2xyEBmwR8fYfGpnW7fv163LlzJ7q7uyvtcDgmktmp2FWtR6SZBQlSREy5eZ8t8WZYASklTydxryIUolIsQqHMZeuXkeo5NiGPPKekSXjhyX6I/97KgSiprx9l0YtQSPonp5RBSV0RD1EeOmBE52UgXfayogSOz8n+NJCSm6zAaD/tmJie5t4y1eUFW3T0M9njufCx+Xkgp5X981iIwHPBIhKdVxYi+PllgZafWVawKqjzPnR0irh+dHoUaDIA8IpgVit75bqQPEdDHVF2PUznlXssXbe1tdWCdjuaTNSMNoJnhXQX5wsqAHA0j1W/jsRLbr1LBQMARxqdoy6d41kCrRmzLHRc9bN8Ae2reP66iM7y3nU0MDm6GUdZFKu3ve1t8e1vf/vHzwH8UVyveMUrmn/9139dFlqHkS082pHVs0jBWzzQIcgIy87joLLRxQpVJyo7R4KRsvMj5AzRCWA6LSP4kxcmBagxiejNVAGjZU+3UAGyNxUrrqTgmP5xvoZeTFPRSakrMPE0MwsiqJDr9s/XhvvnPCZWGrMSkUEEAwinDtSl9WhQM+4N0TQGEXROXAnLOFGmiCbrRQPB/dN3s2qcKVpG6jRiclzYWqmOi5vxSBXYKOCSTLIAgw4494/rRIRPa+tpMaKPLLrwPdTeChGVE6KLKVlH+LJiJRk09o6jw1ZXGEDkjHxlOcN02PTd5MNlxQBCp7hWWcAsp5u6yosVFhcXK3w5Oe50ItkaxekJWfEMgyx2TSDx3/nTWUqzLuWacQcdncp4yUQZ5QS1S2USnSJ/lZxaBqLaPzpAdcUcdIC8SK+uqCwr3GE2R+iismD6Pgac5Fd6sY6nzpkxkU5msE5aCfUCZT2j5gj5FyKr/cm4gU7Lkd4lACNHnnzezCZrrdi6iO1qlIbO+J4u67/9hYbxVAAAIABJREFU278dzz777KEDeBDX/fff3/zTP/3TOHbsWOHiMBUmh8uVvgtbxsPRIaBzQ+PIKCUj8kqJMWpSKpjVyCS2E1ljLycdzKWlpWg0GjE+Pl4pEGD6kIiatwuQUmWbAEa/3h6AQu+EXSlBpeQI+5Nz44iBxiLHT2OSkibhvh3Ksp82KUxjessMvhiFEjmLqKI/3trDkR69LywsxNbWVvT395c18gauHBs5m/yMvysjzXYeHBsRNL7YDoKVqLrYDkUIBFP6bA2kz+kc03ngmnlTYe0N5cZ/5t+3u/yeROt8Phyjj5v/R7Qvm4O/2GpEa8wiMqKRujfRNb3r7LCCW+NkCp1rxz2V46Vz4//m7/qaEbWSzJF/SaRNv8OxcQ9fSGse8sx4PjQeR/iylyPxfmZ5VjOkmy+OTfupFmCO6tHQE2kkGkqkVjLA9eG9vdUJswRygPS3zKhQz2YZH7cB0nNa5zpesvfFZEAreo1sZB0nnsVLDNI80M4oLAJLnOPrAI7sZsZP9SKmjIrhzqLWS3apq6urolO5NhnVgUGI7KXsttapu7s7rl69GhsbGz+2bWAO/GJlJ5WaKp6kUGnAInYNO2FjOoDsM8doNGK3v5lXaZKb5eXrd+7cKd+hqiIKO/l3UgZ0ADs7O+Py5cvRaDRKVMompEIDFZnp8HmkL4eURSc6zEJtPDXOljgs9edBc2XDiCwbj6B7XXRCl5eXy9rUoTRSoOz7lY1HvBdGrYym1XKASJbGIyUshaP1EbpGJItcmPn5+djZ2YmRkZFiTIkiM71RNx5HRukMk1y+urpaOCYZWiQnQKkgjocRNKNUL7yhk0wHiEbM0cd26JXOLVPc+xmPfkccW1Eq5OQoWNnveLSe0gdqr6Hz5IRyR47Zr1KVixG7XC8i2UQTxF+lofVuBLq09u3Gc+rUqQo3tLe3t6DNcj44Hq0P+b7tDCudWTf02XjUAF2onlAh6WA6ijKqRKrIgZuamiqBcaafhRLXjUeyMz4+XiniYQ/CbDzOgdN4ZmdnC2hAx5HOWTvUWuMZHh6Oc+fO7Us/s9DEeezLy8sxNzdXnEZWTNeNxxEzoVWnT59O9TORfelnL6Skfr5x40ZcuXKlRT/LXskp6+npaTue/v7+GBsbq6Sh2+lnjsfR16mpqZaKd+lYdgpwFD8bz5kzZ/aln7PxfPrTnz4Qv+cQAYy7KeBPfOITJS0m4Sf87aXeXsxAJIncIVY9ZalffS7Uz9O+KhSQ0qWSI0dCL3Kb5JySh9Pb2xtf+9rXorOzMx599NEWBLKuwpNRjEeddCJYtSuYWwdFion9kthagWldkozJCYrYjcxZicgUJiNgcoG8co6IGXs/kvfjjT0Z/Wp/JC9M5bB3XYYQEK0ikkce13PPPRddXV3xUz/1Uy2IRV1T2YxblvHeZADZr06oDJFMIr9EANjnT0ZLSszRV92faI4QHDqWEfWPfNK8JTcZ144IMCsZHWUlZ1LjUBCnohrKqvP/6lBfBodE+bJGzkSfiWgRYfW1oGwQueGaON9VsqH9rkOyKCNEBYlsct/JLyRqpbNC9FkX14L3Z4bA+XtaE2YJKJssFmPwzCBSWQutJ9OJGgN1B/U+HWoi4Po+7gODD1JYiFRp7CyCoUxSnzLQYCZF68eLZ0P33k8FulMcMt6bp8YJLshZVLaLHEHtBVPiHswzNd7d3V3RIV7kQvvrBWiUG8matyCiAz0yMtJStEc+JTNKbP2i/Zifn4/5+fkShM3Pz1cCecovsyDk+Kkwb3R0tLyIKJK/y0LQjY2NeOtb33rIATyo66UvfWnzgx/8YInGJTTM81MYWZWjzadjqAOkqFfKXAqJ3Izh4eGKMIonRWGkEyZHgzAyq1kJJbOST9y2zs7OmJ2djUajEQ8++GARRt5f42FURYNHw0aOHSvQNBZyIOQQCflrNpstbXC8apWHghA7kVGmn53nxzVZWFiotH2QQaSTfPTo0XJIWaGnNRkZGakQndkPTAaCDjIVFhUGK8jJp2NLgrW1teju7o7777+/rMvo6GgZi9aF5HU6p+SmkOvk+6PHKclxlmMYESVC9fSHyyyVOrlOcta3t7crAQPvzTOkdaHD6i1/WOWsaruRkZEiM5JbGrqI3RQaK+m1LlwTyq24Rs63kpNDng5lROtCDhgdIxaqsMULZVXjIdVEToscVfbxFDKj/eB4SIBnc1zpOK/E9fMjAyeDy76icjKZ3vTKfe2N3mX8VR3P9J3mqLPKakjuEak3pJQo+BZCJFnJdJz2Tgi1ECs54a73XcfxM6ZdiXZmTojLrPbMK/NZwZrpfddvqviVE8IuBuQIttP7fpal44jas40XU5jz8/Nx8+bNCgeOPEoFFCwkYUq1nd4Xh3m/el/7I1lhYaYCHNf70u1+jkdGRipOa6b32TWBlcUvVO9r/5lpcr3/zne+M55++ulDB/Agrocffrj593//99FsNksxw9raWhHqmzdvlteNGzfKxrJylfwsKeP+/v6KAI2OjsaJEycqxpucNu3F9vZ2iT69BY3GQSfUD5hQAzkGXpr+sY99LI4cORLvfve7i1FQNC1OjlK5TD25QJOzwMIOXR0dHRUElBEghVkKgGgTI0EW4ngLF2/IzcpEojgaB2F5bz9CQ8DDHREFgd3Y2GghsSsNx7SgFHbEbjWpr4W3qKCC6e7ujs997nPR2dkZb3nLWyoOt3ifPg4Wj0jRCflRBKpeXGz3kBHEFZmzCk0OEDmeRKKzwgzyK8mF4hlheouOiheuEImmbLI4hDJCFMdTSHIQSJb3QiMGHEJdmSUgpycjyXuBgxsecbCEePv9WTCjtcja9RA5cnnQS46sDI4QeRmyrPeZB4FKVRP5JcJK5MpbpMgxkTNF7h45vywSaNdWidSViCr1Q+cgayPjjhqLYBSMUh7oRFNXSLeyspUOCc/F/Px8sRsaE6tuiRYLyWWalfaDDpvkpq+vr4LkEU3U/tN2UIezx6XGoADHHfkTJ04UpIrOqxxKouDMQHDtb9y4EdevXy8O2o0bNyo6i1kJcsAZ1Jw4caK8tC7kz0s2Zc+1FpKDGzdulHHQtkt+WcjJzhO0GWNjY+WldVFAoSBLZ1Wo6ubmZsVRvn79enlpTAQpyEnXGHp7e2NkZCS++93vxtLS0qEDeBDX+fPnm3/wB39Q0B0JNEnhLLjQIZZDeOPGjSLQN2/eLEpuY2Oj0nOMEY8EyQWZh0qGR06d4F8eKo5DY1C0TmUrh6q7uztmZ2ejs7MzXvva17ZA0DrYUjAah/MmMoSLY5CQcy2EFLCpKqt8dZAc4ZKiI7+FaSRWzOkw00llCsUVjNAQGTnNn+vAtZARprJVZE7n1NdBhkdrIePHBqROMJ6amoq+vr5405ve1IIM0/CwqSw5a45qyVnmWmg9nJPl6A2RChYJMQXMljqMyNshjlS25PKR/J0ha1oLOctyLOmQsPKbZ4MpJK8e3t7erlAUuCcZWu9IlorISAmQQndkT4GcuHRMp0XsVlWzylvn1ZF6OS1EGZUx0Fo4cuTIBA2PqBtKWVI+jx8/XkFqJB8MHlgMxeppR/X40lldWloqvFMFckSvdB8/p0KNyBNmxbvQMK4F9TblQ//PAIZ0ABl6OSXS31oPOZADAwOVAEZyxt6fPga9s3JUexKx+/QhOYkDAwPlvtTlkg2loXW2WPTD6nqCHBoHuXmkKjCAYfCke0uXnzhxohJQkToiWZe+WFlZqYyB79Il5HCSHqE9HxwcrNx/dHQ0xsbGKig423ARiScfUnadL3JcWcTCtLfGMDw8XGw7nUSimiyoIe1MOsvvL4d1cXExnnzyybh1AM8CPnQAo4oAyslbW1trQf64AYqeFOlIGDs7OytGgwdBL0bW7MtF7hYjNxcEHQgZc1bnsWJJRkqHYGxsLIaHh+PWrVsxODhYUsB0MllmTyWt+Us5yJkQGkQeGfl8PIg6DDwI7K9FniEdCCoDHkZF+uQK0Whqjf3e+jeNN506OdlMyfEQUjkqgpZyjogSwff19ZX1HxkZKYpAMqCIUegrq6rX1tbi6tWrxTmRHGoPbt68WZAVtl+QoVQrHUWtlD/Nn8in/o7N0JnK8DOgfytyV7AjB1/BDhEMKkM6lWyUrUtR+/r6epG/hYWFEjELySASL/RZe0kkJzuHMgpDQ0MVbiN7HNKpz86hnAWhFwq2hOToHMo40hhILlj5r6CTVZ5E3v0c3rx5s8K7ZWGD0Obe3t5KcCVdoHM4PDxc6CZC3+U80kHwc3j9+vUKwsfKXHJrdQ6J3mgtRkdHK8g3+YtM1TPI9HMoGVB1qYJdncOenp5KIEM5ZHpP/F/yalm1SXvAPZDTKJRHxXbiK9JBcjkcGxuroL3kk8r5ZLGN1n9ubq7Ig2RAqDubPAvJc/RK95bTSNBBRUiNRqNSgERn3c/h/Px8BU0U7UnnUGeRcj8+Pl7OoewBOa0sqpE9yM7hjRs3KhxF/Z1kQI2khSRn53B0dLTSf0/7R867AobFxcUKcscARsCIgjjZZK0tgR6eQ9kDIcs9PT2V7gykidy4cSN+8zd/My5fvnzoAB7EpUbQ4gDq0LBPGlutMHJ2Z4QR9NbWVqUNgRSdlCGRLqJMLGtXw2KSUOUYOV8p4+ZERIXM7/3rvFM55x+x28aChQ08bEw1SVGynQnbUijtJ+eMUbTSKt4DiqRtOSXe108/s0eWlJCQJRokjYNROp9WoLXS/RmhMfWYtd/xptFs6UECPXt1ZcUvLDDwnlPsRcfqPhlB0RLYoJfUgKzBLJvMkjvHfWBBAQuTmOYkOVy0BO2fAgTyYb3xbt0YWEHPprvZGLgOTuCP2C26yQj8fKcsMO0pNIl8T7ZjYg9AVsHKKO3s7FTQJJ4LtofiWpCfpqwEA0bOneeBjpEcCl1aA1VPE2nkuSACr3XwqkWie0zJcwzSSxqDHHXK4V50AKbiWeSje5B75rpJL+9dqRQ4iyY8/SzkSEga9YICRjmqngWQnuS5YLERW3A5J49cRY2BXGwFHUQ1HUn0QIVnkrZBc2XASIeFlAhWGtNB8oDZ0TOdCSG7EVFpQUOqFFOszMRIb0REkSemuxUg8KWAWfxiBXksrCJgMj4+HmNjY+VdjrrAHQIGouVo7XX/2dnZmJ2drQQsPI+SIQWKcpB1v4mJifLSuihQeeSRR+Lf//3fDx3Ag7hYBcwu/eRxeGpRB4opPV1S2k6yZaWPDjQNvZTZnTt3imOTpUrEOdNhIo9FaBJJ8kwNDA0NxXe+853o6+uLN77xjcXAyrBKkSnVzQiPaRpyJcQxY9qMKUQpEnL+SHSWMaFjxXQZlZgcXR5iVvN6IYmny3RvKTEpIXLrvDiB9xbSwaIatiSQQ8vUGO8tY8rejkwV6j7z8/PxzDPPxOLiYkREST3I2LoBkfPgxoPypuhWa8V0KXl8zjVlGr1O1tvdW3vCSlUZLrXE8XQx06QsUHFZz2gEvLc+p1NPWWdRSrbn5Jayj5oMB50VzpspYiJLEVECKucaZ/emwdrc3KxUHdNZcl6WEA032AzmyIvKZF33/kFkXf8vrp6cdzb3zmgbdNwo6xFRKQKSs6T5EkmSI9dO1qnXXeaIatfJOikBpO4ws6PsTp2sc8/93nRUteZC9SXXRNNc1qVbWcyi7/X7ZnpdjpUXSTCr0o4mQ8oQ9TrnTR3jss40s+5Nvc57E0XXvaVbHUUml57yJt2qc+6yLv2WUYRYXS2wQ7LuulUOIukwsuWUNw+MaMNGR0fjD//wD+PixYuHDuBBXPfff3/zIx/5SDHMcgoUWSsN4xWdckaWlpZSfoTgf0W2LMbQQWKLADkkFFx/soYEiWiDV7MqspfCYqXiwMBAPPbYY9HZ2Rnvfve7C0+GES17RxFVkGNIEnUW0StCY2THl6JYcbciotLXi+kv3csbm4qjxFYVRDQ4BqJJUspCMjhvFhgI2RDfka1E9HtCEWRcue9smcGWJUoLeS9Jos1ah69+9auxs7MTr3zlKystU/ZqyOtNor0ZtH6XDrA3LJaTpHmyTYo3M84uNiz2sbGfps7bXuPhvLVOXC+hvfwbzpONktmQWO9cH2+aTKSFrWMYCJBDyDWhXEgOslYykg+NNSLKfNgUme1TvLWPdIDGrXmxbYtI9WyVIueCbVu03mwPw1Yk+pkoK1sbkTuogIsIJ/WBzou4ell/SG+LQqPJAgbvwciekEQUvVemnCXpXukbZjkyRJHtlNiqJ6sOp95Xyl2ZFt1bTlrGGZXNkT5mZTqftsGCP/IBSbmQLLAVjb5XgT+pHjdu3CiOM+kOEbu0IwYjRPAYkLGfo+4t9GxlZaWS4p+bmytpVq0/C7pYSczga3x8PCYmJkqKeXR0tNJvU315d3Z2Kpmsubm58pqdnY25ubkKD1II+tbWVqX9ECkNk5OTBbUTekkEW/slgElOodDCmZmZ8s4MH+1sT0+P9v+wEfRBXOyWryg5ovUZj+xbxZc4YBJM72PGFgAsbhgaGqo4JjJu2mhVfXpKQVCyR6msutXh8Ejx+PHjheNw/vz5ks6Qg8EUIiM1J84rnaP7CsYWlymL1KR4ybFQY1lx7sQ9JNeQFZXaBxp1FgwIdWUxCdPLdPCZRhWn0ecsp5u93OREilsj595RmKzKW/LEytEsIl9ZWYnvfe97sbm5Gc8++2wxrI56EekkCjE4OFiMkYj4Sll6L6vMwLDZckRU0rVEP8hhIflejkcWCddF4ULaxOVzDpET//ViqxX2R2QQ4e0hZmdnW5Auomwy5hmiLC4t+Yvi7EREIbYrJUmZWlhYiEuXLpU9kDHVGZZTSM4SUfTx8fEWVJOtKLzohHs8MzNTScvKgVAlLttPkLeodT537lw5S3La2PPRC5CI8MzMzFT6tUmeyRnW2SWyMzo6GqdOnaoUl0h3UCcz9cr7zs/Px4ULFyp9OOWUZ4UtRG+13tpfBosMzllkJYdpZWUlrly5Ukm5suKZLWEcMR8eHo6xsbE4e/ZsCdDpkEt3sMDLUUQ9752V7wq8aX+4v9Jl586di5e97GWVNk5MMxM5pYMqnuTly5fbZgmIZOm+LOB4xSteEUeOHGkpKiMfWWdJ53pjYyMuX74czzzzTAksu7q6KgFApjv0/+Pj4zE5OVlByYlaSpZI+ZLOkuP63e9+t9h8Uox8vqJdyRk+d+5cnD17NiKitJIhBz+zSU888cTB+D6HCODus4AJ3fLpER4RaRPkLLWrSHKun6paSfx3wjGFjkLONgpEXxjVO39G92Oxw6/92q9Fo9GI97///QVVIHeIrTwUjUoJiFwsBarIhi0keD9F44z4yBtiB382YRW64k8LkYFkGxFyyOT0CE2J2H0snMrxiSwK4SWKwka8dOKz5rd82gbTuuQtEkl01IZVyUTJjh49WhpBv/a1r21Bi+joEB1jI2jeW0ba0UONmyiUAhJvcqwAieig349IOBsbR0QFAfT5MD3s6CyRPm+mzIar2jvtn84j5Yb7yObS5CcS7ePesdksn24hpE7nQlxTtrGQEWQqWmvBPmK6F1E2NjoXAsHWPryfUDZH3rm/Wh/Nz1E2NnbnGlNmWGzGs08usTh72kvRHeTESMcQXWNgq/1m82cadDqq3sapu7s7InafpsJMBh0npgC1v06vaJcCFJqnQir9HdFLOmjigrGFFhuzU78RxWPRhuwJ5Ulysbm5WbEdQtJYuMQ11hlTtqq/v7/CQyMfTbZFxTqSXcmpiiNu3LhREK3Z2dlKcKmzGhGVAhWhaBMTEwVF05yHh4crlCFWc2sfhZoJQdOcZdN0z0ajUbiuQ0NDBa2bnJyMkydPxuTkZIWyRD6+7OTa2lpBCmdmZmJ6ejqmp6djdna2zJ9gBTsssADp9OnTcerUqTh58mSMj4+X4E5nlpXrLMa8du1a/M7v/E5MTU0dpoAP4rrvvvuaH/7whwsiJyPENGDWv4dRENOT5KQpivfqM1YASlky/SHl6IRaRphy0PgIOyn+wcHBgs7wvsePH4/f+I3fiM7OzvjoRz9aOHgaP0nEEua5ubmioNkOIGIXsSDfMav2FPKoSkeWveu7WeHIKI+VVXJE2A6ClZ2sspVxYlUbu9yzok/vbL/AnmlCwHp6eiotF7yiUpGdZIjoMWVICkpzVPsgyVtnZ2esra3FkSNH4vz58xVkgJWU4npJhhRMsJJ6aWmpIrOSIRYKUIbkRLB6VqkVb+vAqk05KWyvQUI2+V1aF8oQeYRMITF9JcK+igAkQySAK3VDvg1lSI6eAhdvHUHZZRNnXWz46mkyvWR4yFvU2FWVqnucOHEixsfHW6rjyV+T47SXHmJalg6pnCSitnoXwV8Ib0QUGWJQyH5lTAu6DOkxk6K+ZGtLp81lSAUlmlcmQ4uLi5UgQ86Eesax4nh8fLyCPGUyxF6nWcW7dLyyPSxmYgsp10Ne5U0ZIpLHe2YyxKc6KZjwDgNZhwVybiVDpDFlMiTHX+dT30FQQbqA+iGToe3t7QqfPevBR8ef9AmvHva19S4Gsg3sG0s9JJ1Lx99liJX7ruPJd5S8dXV1VVr7UIa0tsx21MkQ28cwda13l6G3vOUt8a1vfevQATyI66GHHmo+/vjjBYWTwiMnQMKj9KtQMUVQbKwrgVEFkSsCPvuPkbAOvzeG1OEgcsNI2A+GlABRv4go9/vlX/7l2N7ejl//9V+v9FoSusn+geKpuUJnYYnScOwZKLliVaQTrhWNKoJkM+26tB9Jv1Q2EVGJRpWaYSpIKKocFd2rs7Oz0hCYhTtsk8HyfCk3Vh86qqC1FGqkA6+0tRx1pRfZ8Lanpye++MUvRldXV/zsz/5sSR2z8pj3k8x6qkvonpw6b4DtBRpK67GzPXumecNnPmVGc1OKmmR58p/UA0scUCFarCRVOk9IiZxZoZkyhjKI/pQSkuIHBgZK6o196divkKksIuBCLJS+EwrFalEGOUJRtJYRu9W6WWNeIkHqIKAKSfbfk1zIyWDKjn33+DhAPhmBRp5om5BxNkL2Ai7vYaYUsGgr7lgIeWKrGDlQGhv5clxDr/wcGRmpVCDTkaEucR0tPS3EmM2mdS85h9LT0mVCEY8dO1ZB0oniCd2iI7O6ulpBfXg/raPQLTpQul9vb285uxsbG0X/z83NFYRJvLjr169XeNFE0rVfSmuePHmyck8FjJqfzoLkY3Z2Nqanp+PatWsVPhx5n5K33t7eYgcmJibi5MmTcerUqco8pevY6kvBk+537dq1mJqaKgje7Oxs0T0CVXTWtX+a2+nTpwt6Nzk5WaEaKcOi+y0sLMTMzExMTU3F1NRUTE9PF+RQukDBk4JhyaHuderUqYLcTUxMlHNImg+R3+np6Ziamopr166VFykoorzIj5C+PHXqVOV+p0+fjomJiRgeHo63vvWt8Y1vfOPQATyI6957721+4AMfiLGxsUrVHlOHdFokMFI66ssnx4KFAHJapGxOnjxZFJwOh9AbGUKiRYTTWdauw7ixsVF5sgEVt5NRx8bGoq+vL5aWluLYsWMxMTFR7idDoftRoep+UjaK2HU/VeHpwFOJMwUlp0n3E2JDxab7yQnd3NysIDbsp6XUBKNfOTp8dBGf3qFIkHtHR5RPU2GqQPejU8/ITPeTky2nSfvFtWQahu0QGEQIaTtx4kScOXOmRKDkQkVE5X4rKyvlPryf+DnqkaXLUVQZCM1R9xsYGCg9+hqNRsWpZz8s7SErCpk6dGTR00ziayqi1z6QikAZkSGmA6z57ezsVBAE9l3T/NgCiek7OvXeUkL913g/BXPksmktdT+hUDIWRL0YtCjwo4yylQhT+DpbRPtJgNf9hoaGKvfj81V5P51DOvrkJyoI8vOge7K3JQtK2FdU9yPpXQ6i5FOBNTmnOnNsjcH7sfm1+HnSn3TYSK7X/BS8yCli1kTpwfHx8eJ4KKBQOl3zY59KEvoZ7MqeiJOnJztofnJkhAjzfnKA2ZdP92P6k8CBHG5RO9QTUmdO95uYmKhkpdgTU0EsizOU+pQOVdpT9xPKeuzYsUpLlZMnT5aUp+ZNB58UCJ45pVl5P7VYun37dqF19Pb2VtLWk5OTcerUqbKeJ06cqBQhsiKesnLt2rVi4wWQMLgmtYP3Y0pXNoJUIXaboKxcu3atzE+ooYoP79y5U3mK1NjYWHzzm9+M+fn5QwfwIK6HH364+dnPfja2t7crHBEdqunp6QpUzubHgsdZbOF8AhlzpUBZ5cs05MzMTIm6pDTY2FIRl4RcClGRj5S+OBPiJbK1ixQ9FSIbaZL4T6KyoklGdmyeqSo6Qf5S6lpD3UfpZDmUEVFJl0sJMXLVGgqpIvKwsbFR1oqKkGvITvosZnBFKGdSayiCvQoK5PTQOFNJaA3FN2k2m5UKRBovGU4iw+Lz7ezsVBAwGhPJoSoRxeMRosLUBeVCayj0TdGxUj1M62VryCczkINJh0qyMTY2VgIRoW5KafsaUhbpeBDtZr/IbA2V7u/v768002aqS/PhGiq9tr6+XuGYknfFNdRnSgkLQdFZdmeKlYRKQ2sNyddjxoD8J60hexASkfU1lHMjh5jpNPKCOSetozIGAwMDJZDZ2toq36fAV7pQ9yafTIFvd3d3JTWZySEbT7Ofo4zt3NxcMcJses02IZLBgYGBihGW7mVBByvvtYZLS0uVfZLcE5Vlpb+CheHh4TInVn2ywphV/SwQ0RpSHyrAWVtbK7pQyKEjeQyw2bNVPM21tbVKQKY1ZNNi9uckd1zfrTU8efJkpXBBXEwFZeKqaw2JprE/rQKkzs7OSuGLr+Hk5GSlPyyLMAmI0E5K7tlrkrxtNp0WOkmHVNxCyaGyHq4Lr127VsnMkcMtx1OBGPfr5MmTFSqUxhYRlbY81PFCX7mGWosjR47E4OBgXLlyJdbW1g4dwIO4XvrSlzbQ2vsnAAAgAElEQVQ/9KEPxejoaOWROSR/k6MhwWN6keRsNvllapbIA1NTQjqkTNkvyLudy4iyzQUbTIuzwMpQQv7NZjMef/zx2N7ejte97nXFWfM+hzJY5DUyvedVZEQ3xI0Qj5JFNV4FymbObFBK0j7TlYz6hWCJ66TO9TImvB8LadgQVvJPEr3mIbRRaS8pC7YIIdmbSJTemQLW3wltyJ6FKwSsp6cnpqen4+jRo3HvvfcWThZ7ajEVK6eABSZCtPhkEsmJUoeeklWxjgpKiE75/bRn/kgk8fmYKlX/RyGYCpxYdKEUMKujtWcykjL+6vLPijumnOkUinYh6gUbq2s+TKez3ZGKWNh3kHMit0zOiGRE6WbJB58koIwCZVKpoIhqc1zyIv1+4gqy35v2h/xEVomy35nWRWtHSgm5cwy+IqKca6bP2bqDfDLtmdZQPD1vn6HAQfw1oZaSQ/F2dQ9mDrSPXEP1YlUAy+BBTpSC3K6urkprEA9g6fiKyyXaCgNYBeZCnuT4am7SaUIqhazJoVaKcHp6uuh9tvri2WLwzzSoKBDsdSoEdmlpqZKSZKpX+kuFN7ItAjZOnz5dXnRuJKcdHR1FHygtf/369bhy5UpcvXo1pqamioOzuLhY5i8KTk9PT8V5OnPmTJw9e7bitCkb0d3dXYCN5eXlmJmZiatXr8b09HRcvXo1rly5UtZQ9AM5otQTZ86cidOnT8eZM2fKzxMTE4U+wrQ1+aFXrlwp89Keibe5uroaEVFoADpLk5OTcfbs2Th37lxlv4aGhsq5EOK6trZW1mp6ejouX74cV65ciampqZibm4unnnrqQNrAHDqAEfHKV76y+clPfrIgPVI2nvqZm5urcPJIKBdxVCiF0AmhBlSgeki1SN0kjEqp6bAKaZRTpMPmPEOhSUwRSqGx0m5ubi7e+973xu3bt+Onf/qni2PJ9ip0vMjvYKWSFIyQCRlUQumaD0nNMgjkWOj7mE5i42o+BUGwOJ1WRe8kictJVrd78gkZRZMXI2RH66Y0mfeIYvEGq0FZ1cbCCc2JVeAybuT1UWlev349/vVf/zVWVlZicHCwELOF9qlzvYyO970SgqmiG8mb+GeeZnTkaGfn7iPdtC5Cc7h2UshKDWq9lXIX94xVeQoy5IQIRZdjRbI3UW0pZFZac938nFLeiJLyiQdSzJI3rR3bq7BxMeVNQSA5n3zyjIIulzetIZuCs/+c1o3UCK3b9evXC+KoNJQcRSIdcnDI29M5bTabZd20P5qP1lBGVeum/poM+nQfFgEoiOrq6qqsmyP0fDKDp7Ulb5Jr13HsMqBzqp5q0gl0DLVnrN6WHu7r6ytjZ3qZHFlWwDv66jLHghitmxxD8Td9fwQ6EA1VgQgd+KwvnbiwcmoUvGbnlN0gInY7I+jMU65ZiMdqVj5BiMVEWjfKm6PW4i37mkln63cidrsFsIAyWzcVRXR3d5eKdmbVSDXgurH7ArnfbhcE1jC7IJljpTUzC7ovK+gzfiYpU0L7Zes0fz4v2tfs4x//eCwuLh46gAdxvfzlL29+5CMfKYZZh0pRpRQlidQkGYvToQPCIgIaZCmv7u7dZ++yr6D3VRKXSpFSxG7bDgktq2+FssjZ1O+SOL2yshK/93u/F9vb2/Hoo48W8rmQKjZWVqqPzaSJ4rDaNWK35QL761EpikOo/mNCVvg4JrZ0EcKhSEoGLCIKN0IorRAU3oPNSuWoK0XF5rhsScJUEYs9vPkvU1Hsp6ZCFh1i8kH5zsbEEVFpByS074knnojt7e146KGHKs2+pbzocLDhMRtOs40LK+W0lkpxaa34NBrNjQ2Y1WyX7SP0YlsV/ZutX9g8mm1jvDXO5uZm5XfZziVrx8P9yx6VxqbmbP/DeUpG9LQJcm7YyFXyqPXkOnnTdLWKImVE8ss5EHH2RzIqYOQjKf1pEjrD0hX6XbafoFPD/onsj8nCNH0fjanuJ+MmlFltl9g/T07i6OhoQUwVjCggyQJGFr9JLxKplEMwPDxcoQJI10q/qvJV+02OKluF8Ok+khs5oENDQxUqCqkvSvtFRKVIRIZaqNDMzExZw5WVlXKOuru7C/que3hxgXS5uHhbW1uVbgnXrl2Lq1evxtWrVyucZj6PWDQD8u+Idkke+vv7i2yyQOPatWsF7RJyyIKQ7e3tIssqlJicnIwzZ87EuXPn4syZMxVqDQNfOVEzMzMFTRPKdfXq1WJ/19fXi46RozY+Ph5nz56Ne+65J86dO1fZIwXiEVHp06e10nwuX75cqUSW7pCjplYt586dK3NRUQa55rKdi4uLZd+vXr0aly5disuXLxck8vr165WMgs7LxMREQQaFep45c6Y4wAMDA0VXra2txczMTDz66KPxzDPPHDqAB3E98MADzb/4i78oFZ4yHHLGmPqVIhSELSNFArgXC8jDp9KQ0WA6SEgWoxW2AGDfLSfQKz3Ex05JaIT66B4f+tCH4vbt2/HqV7+6RF+6h1DGgYGBkp7xkn82m9Y8WHLPNDnbx7CVQV3PKd2D6S0a2Lp7EL2SwdA81D6F8xCHRqgFD5intIiQLS4ulnt0dHRUql29AEYGSeslp1SUAnFaMjRpa2sr5ufnY2dnJ86cOVPS7tm+s5BI91CVpBS50xbUk0vyy5Stz2NsbKzSb7GuOIpGqO6MKIDwefiTAvgEDHKoMvoF+/H5GeE5FBKie+gcyhixRY+Qa32u/nC8hyge2TmUbNFJZNNvoq+sktU92FidLXjYkoJPVdA9hIiyWlWFOUpvZ2eEFbiUXzq9bLvj51BnREGrnFCiyH4OWSgmh5ctNRQI8x5sHK57sP1Vdg6V6hVSrTPCyl4hhwrytSfs08nOCy6/uofOogI2nUPJb90TJhRoeuGSy6+Cg76+vjJ/nUOeEdoqpf11D/ZudaRQtkr30DnUGaEd0TnUviv44Bnp6+urFJfxHGocfNa6bJXS72z/42eEGTGeEaJ32nM+O1pUDKF3LF5T4CEwQ/QZ2kPuh2xVX19fHDlypMyDjbr3OiPsCsHqdLaHYjHlnTt34tFHH43vfOc7hw7gQVwPPvhg82Mf+1h0dXWVzuNra2uVakodWvI/5MzoWYXy6L1nmg6tIgw5TOwBxYPE5qBKLUrYxe9gTyIqUCkroX18BqIq7f7u7/4utre34zWveU1R0oriWLRARIINiIUKMYoXKsGGqkpZig8loyguGp0KNdwUQqD5+yOo2EeJFXxEiPR9dPAUQUbsPmZMkTK5nmxmLHSMxT5sCu3NkskN1NiIannDYqGFfEQZmyN/4xvfiCNHjsTrXve6gvDxEWhsls0x67v5eDNvvMxm0kK9Zey0JuyDKfSR6Xs2dNaaaF24JnXrrciZrWS0flwLosLiaTJVIkSRT+AQGsfG4pJbVrHru4Vui38ppyciKlw+cezEiWTzZCl/8RPJKfWnAGVPABKnzp+LTJ4sUWZWActwkCOo7xdfSkgoe0+SX6xzzKbYGq/6k4lqQpqBzp7WnkU+5AWyHyORf1bditMmJ2FkZKTwDlV8INRO/Cjy8xR0iLcsfaMUorh53sKD7Y9YHa12IeLJyRmRMyFUSsU1QvHOnDkTp06dKkieuGvs5yh+IVEp7QmLJ1gQcvLkyYIWaR4nT56sBP5KHd68ebOgaVevXi08MsnV0tJS0T29vb1lf0+fPh333HNP3HPPPYXHODk5Wc51R0dHpWvE5cuXy0tzka1ZXl4u+qG/v78gnWfOnInz58/HPffcU2mELJ0UESUNPTs7G5cuXYoLFy6UuUxNTVWKCaUfBgcHKyga76E15JNzWEhy8eLFuHDhQkE7r127Vul7S364kEe+KE+k3bCQ5OLFi3Hx4sW4cuVKOSMKyhRkiM6hPbjnnnsKT1FPpnnkkUfi61//+qEDeBDXfffd1/zjP/7jUnEnp4V9tFhmT64M+REyJuJk8aVGmar2lINARcx76PONjY2SqiN/iWRZRSVscqrmzoqkWbX653/+53H79u24//77yxykCIQGsK8TWwRIuGUUZQg5fiEPqtpT+ospL3LjpJjlbIrfIweK3CsRY8nDY6sKNtRkhZmQBz6RgU9BYJSmajZFaktLSxGxy03R+mRVoiMjI4U/duTIkcJ9WV5erlR66V7q3bexsVEhyY+Ojsby8nIMDg7GG9/4xlJRLmeCj4diRKuxC8Gan5+vRJsyWKpYY1W3Ils5zuLECoWT3KtdwczMTCVNw6cnKBAaGxsrRoR9I1mUISdK1bqsyvR+Z3Jq1LqGZ+3kyZOVYiQ+ek8BkFKAchp09ljMIhlU5TbXaHx8PI4fP14cCzb69jYZjuYrWGk0GpViFaYZhcSp115XV1cl0NJ3s3qV6dKIKI4Vm8qyipS9Jpn2E0qoM+aImAIY9s3UufICC6FhCqrVl5DVlSwIEDoZERVdJ9SbjqGQF/Z25NMoKD/SGXLS19bWStDW09NTQTspp3LW+RQT8gxZKa/UtYqUNjc3KxXKkh2ukeg6Sr0qMPRqfMmpkGMGpp6q1otooQIAFhd4pbp0BQMZtdURV5adGSYnJyuFVgoY1N5LHEmuESuQ1WlCAZLSoL5GpBrJXio9rTmww4Tk1zNNQlOpqyVHBAgY4GUV9pLRlZWVStcANoz2p4mMjIwUCk5HR0el7yjPsfSGCpnkU7C1jfTP5ORkvP/974/nn3/+0AE8iOuhhx5qfupTnyqHXAeQjRtZtarIjA/eJo9DB1CbL2NNPoL3EpShU3Uee26xPF+HhOiZFCwPB3tfSXCXl5cL16ezs7MQnlnkwQaz6v0WEZWKWgmrHDBF3U6spZEmOsHGtVJMclRZKKAoVVFzxO6zcFURzGdZyvny9DSNm+4hJFfNW+UA67v9GZ1ELP1pG3yAusas9SZiw/SOxs1u9uKWKYoXz5T8UO2DnsUsZS15YbNsORns68XH07FxL79fjrGQLH/SC9ebaJkQMD57lxwyoUNydoUgshk2KRNC5YSs8skU5OVqzRUMqDefZFApO1bIK4VKJ0tOBIu/iJARWZKcOYXBUzZ6zjYLlzxY8ga0khUaXukXVsGT68gGzKru5PjJh2WvSSFIMoyTk5OlKEH8JjlWRNzEb2NRnFDP3t7eyuO1hF6Q7K7HiG1tbVXoHKysVNC3ublZHFU+WeTs2bNx9uzZOH36dFkjoXldXV2Fg7y4uBhTU1MFZROiNzs7W2kKrz3jmHkP6QW2rllYWCg24sqVK3Hx4sW4fPly0WNLS0sFZee6nDlzJu655544f/58hb/GZ6RLVqanp+PixYtx6dKlyvqQ76yxnThxoqBG4sidPXu20tNTsr6yslIQu0uXLsWlS5fi4sWLMTMzU/RvRBQ+r+zb6dOn4yUveUmcP38+zp49W2xHT09PsRdMRV+4cCG+973vxaVLl0rD54WFheL0Sm+Njo7GuXPn4vz583HvvfeWtae9azabRedOT08Xnp3W5/LlyxX6CfmJp0+fjrNnz8b58+fLPVjtLnlcX18vzvfly5fjwoULceHChSLz09PTJfju6OgoNu7kyZNx7733xvnz50ul75kzZyotj9j/Vmuufb106VIJ2FdWViqtoiiL58+fj/e9732HHMCDuu69997mhz/84SJoMnpCb6RA2D1cfXoEQYsr0NvbWw6KFMnp06dLBZvalijFxkhDyml6errSikVOgZTf+Ph4nDp1qigmOViDg4MlYmWnfJXGqwxfEZ8ievW4EieE3celxIUKHT16tBJts5WAxs7WHXxEk5AUrguNQsRudZqM1+zsbEljsKJYqbZms1khnut7T506Vam4VBSmCG9paamMWQdbik8OkDrPS3Ez5aLWBOyDqNQmEWOuD3mdLOaR86A2AQoixIujodeezs7OFkNAroxSU81msxgFrQtTX5OTk6WK7tixY5VKPY1Xxn5qaqryuCalwXt6elpaUWjdhVQT7SZCxrPEHm+6xLkZGRmpkK8VDKmZstJqUqwM2ojYy2BH7FaaCp0k+V6yrkcIisrge6rxK9W1srJSUBPJi4yCSPdy6EZHRyMiClLBVJdknXtKbpUCE5d1nS2iPuvr68Vhpqxrf1Xluba2VopfNHYae7XGUKqTwZVkUU9XkJ6RnCrdq4p/7Z/0l5qcK4PBrgLUX/puIZMzMzNlDM1ms4JqU140DyFJPT09Zc5LS0uVJzRw7Fo3oYyuA2SQ1RVhdHS0UDR2dnZa9LrSitJr4q3qnOq80+Fkf0EFKtIv0r1yCJkKZ0ZBgavGrrXR+kh/6bvFi5ScsHCCSK3GI4d5YmKiOFhnz56tVOtqnkI4hcIrJa2MwtzcXKFp7OzslGBWeyqnSjZJBT9uq7WPepesK5hhP12tg5w1rTtTuQIg5Ixr7AxUMlvNlPq5c+fKusjmyVazcplFN0QIRfnRd1+9ejVWV1cPHcCDuB5++OHmpz/96bhz506lWSerkuSQqSJJZeo0fhJ+doxn9Z4UgjZZVUhSdBKeY8eOFcRPnA/l/3Wo2PCWfbEoPBIqOXmNRiNGR0djc3MzBgcH4w1veENxONglXRfRUK2FGmJev369lO03m81CYJVyl9FjGwI6pxqbODCqZJOC1NyOHDlS4VXqe5m2ZKsGEm9pTLX2QippTOlk8LFJ7B+nthkMBoQiiHvEJr9ygPj4IBlpNUgWaiZUTw6AHon07LPPxsLCQnFyIqI46mNjY0WRM/2jSLOzs7PSD479xeTI8LmxMhTDw8MVp4jIitIlbIrtToWcaBkhteCREaJhFn1BylitXdbX10sqnuOenp4uKXuS2UdHRyvGXi+hBnLkNjY2SpqcjuK1a9eKg7u2tlaoCgMDAy3fefr06Uq7JXED19fXy/zpIAo9VNpIAREbxUr25EwokBPHk2lrybTGrNQs077aM50VVRSKL6zAT/3TFEyIJ0YusuSf6JWcNxk0nU+2ZdGZk2N1+fLlIhs3b94slZCdnZ1F101OThZDSZ0knmRE/H/tnX1s3dd537+HpMlLie8iLVKU+HJJ6s323DndCzos2ZC02YoGBZIZQxOkCTCsSIqgA4q9dEOTGOmCAQGMYVu2rAmcNDGcpPHi1o23FtiwZXUzO5gyL0Uli+9vokhRoiiKL+IlKZ79ce/n8fO7oqQMocVE93wBwQkp/3zuued3znO+z/f5Phk5i9ed+TQsFyZf0IRujqCNSwR/99atW7avzc3NGSvjbUR8T2cvb0AD5iUnMHkxxkyA78dMIIv+dWdnJ2Og3dPTo97eXh0/fjwT9BB8wXovLS3Zc71+jbWxtbWlQ4cO2UXbV8+yrpln0sX+QjUzM5Nh7xYWFmzfYE9CLsG6IIjt7u7OFCv5tqecgV6n6M3ZfXtHzkBfKeu9N/2exFnltZWkuFdXVzMXe+8ByBrhstPU1GRr49atW5kWcsw1Y0Yr6DMDfH5f4est1EiHw677uIDYgO+B95uA+4c//KGWlpZSALgfyOfz8Utf+pKOHj1q6cPq6mo78K9fv24L1Gtj6AdMypZD3zNo3gSUFxc/NjZgvnyfWvXmqTCHXmPgq4RYHN5M2msMfIC4ubmpN998U7u7u3rsscdMOAtjUW6SCkPEIbe+vn6HBxYvlz/0eRG8EXZ5lZmvuvaeiKQPCI5hWanKQsfmb4jcoH15vu876Tsa+ACWgIPgG7+zck8671Dv7WFgExFEU3nH94ulUAjBNjRvtutT7uiZ0DS9/PLL2tzc1NmzZy1tzedgPnZ2dkx/SncTxl2ect+rlZQ3CmbDgZ2WlNFt+u+Siwjrg44zmLN6LZM/nH3vUlIc5WbpsOV8Jpge323D+wui1eRCgljdP9d32GBz9+wk0gCkHwQwfLdo39AVseYI4rq6uizIolKagJbx+sDTyye4vBDM8p771OlebArviLfR8Cy572LhU9QEQl4/TMGLpEzloj/oCLzQGnr2ygcsPT09JlWBRQkhZNLrpEunp6czmlXerbq6OhsbYv6+vr7MpYR35fbt25ngDTE/+/XCwoIxSzFG2zu7urrU19enfD5vViJdXV1mi1NVVZWppp6YmNDExIQFLfPz85n2db5KP5/Pq7+/X/39/RYIoH2lKw5jnpqasmcT3C8uLhqbz1yQ3s3n8xoYGMiwebDPsI/Mxfj4uMbHx+07vHTpkhV17e7u2ni7uro0MDCggYEBCzpJX3JusXdeuXJF4+PjmpiYsIIGbFt4V5FdtLe323P7+/st0EIeVVdXZ7pE5oJnE8AxF1x02N+Yi3w+nzGN5nK9u7trwffc3Fzm+2PMsHu7u7uZTl48F9bxxIkTmeyg7+BFinh6etrGTFetra0t06QfOXLEUsR+jpEucVFFkkNqe2pqygJPJBHXrl3T7du3UwC4H3j88cfjSy+9JEkZobJnevZKacBq+FRPd3e3CYgbGhqMut3Y2Mi0zOG53p7BVz163Q8BZWtrqzEPW1tbFpSxyXm2xGvcSE83NDSos7NTr732mmpra/XpT3/a2v2Q5vBeYzBSPkVK6rhQKJiYHbrbp4tg0bzYn8/qU4BeAOuLOfxNmIPQFxL4A8tXM0L7w4Bi1swh6w2NPYOGBi2Xy2VsDzxry1z4wgduzU1NTRac+/QZBQWHDx+2ABn9ik+1zs3NZYyL6+vrdevWLeVyOT311FMZ933vHl9fX59xxN8rBbq8vJzp0UkRAGwO7DLsJIyqL57wKT6vj/Nefd40nO/MB+d4K5L29Cl+vjPYJ1+tyyHFc71bf0tLS8YGhYDIrzGKVVZXV7W9vW3Bp2fL/KXNm4JTqQxz6L+zy5cv2/e1ubmZsSLxjKRvUk/ADBtRnoZkj6ClYKFQsIIUnstY2XtIyyJx4ABhXc3NzWXsNHxRGRcFnwbz3qWcDzCR6BfR1PnuQb4lIRdf5hapCvuF9JaXpxfa8555bSSVsHV1daZT9BW3XiOGgS7ef+Xf2cLCgrHIOzs7GTsfLwXgv+GLBBjP9evXM8w3Y6fVHwFQebcJAm6K0Vjfvv2oZ3h9JxC+M8YK682zkQU0NTVlepL7Ygn2RvbFmzdvWtUt6Xnm1z+Xz+ILnjzD6y81rGnv2en3Lc84cln3/ZQhRMorvHkPvZaas8G/Y0ePHrX3m0JODKj9mcMf77bhjc79mcNa9s4D3j7InzlcymBHd3d3rXd2e3t7Zi1wxnkzeApQOMuYW95p5EMzMzO6detWCgD3AxhBQ/myKLyNgQ+AYI0kZUSmvEC+CwiVPNJbTdF9RSU3FFg/FqM/TLx4neCy3KEeDQTPohqrvGH94cOH9frrr6u6ulof/ehH7Xm+ao814Wl1z2rBBPpOKLBQaEpgWdgEKIQhRQQb5E1nvV0KN3DST96yAxsQ2EDPyMFCUX2MZYxvIUbVGhtFuY0JN14CFukt30ZvBO2tXRDYU2DDM3h2uQULY/CWMQT13ErfeOMN7ezsaHBw0AJjDnqeywHFP/k9BuD+AsKFgT++faE3rPatDL3/FJ/Jm0V7w2MfwGIb4y1oYD+9oTj+WTU1NZn2WL6FH5s4Po+IrxkbOkfPpno9rC/Agqklbc87wrqQZO8E7DLVuZ45LG8PRYqPdCH/PTrRYBDvq2Y5XPgcNTU19h2VtyEjmPeFYt4nsNxEGGsQvjdflAMTRMBJ8ILO0Fdwk94kRf3oo4+aTyOXOvZHRPik3K5du2aSGaQcXGgpVOjr68sUKxFMLy8v27NIQU5OTmYqMblctbW1GatCEUFPT0/Gj9F3JIGxgb3CtoQ17YvXYK/y+XxGAsD7i2aWQo2xsTGNjY3Z97W0tGTvY2NjozFV/f39GhgY0ODgYKYCn/d/dXXVCgQmJyeNyaOob3l5OWNUDsPW399v7COBcUNDQ6boa3JyUmNjY1aYMTU1ZdmhQqGQYV/7+vo0ODiowcFBS8N2dnbaZ8LmhIIMPr/XJfIe5nI5C9j8c32mDA14oVCwVOj09LSxgl6OReHOoUOHjH2muKOvr0/d3d22z/jGDpcuXTKWmDU7NzeXsS5DGnTs2DGbV5/WZi+MMdpZfvnyZVtbBIQLCwt2NoQQMo4FsNpo4Ts6OsxL8Pbt25m+wD4Nf+PGDb355psqFAopANwPnDlzJn7jG99QLpfL+Od5XZP3gELoi90FTIJn6o4cOaLm5mZLBXmNiWfrOIRWV1ctIIFZ9CwgNilsaL5NjL/dkhbzqQnsAtiAX3nlFdXW1urZZ5+1tCkbum+E7RlFdGrr6+sWzGBx4DVoXV1damxsNN0HfmzlxQXMg/f0Q4MG08GBdjd9Imm78nFiSurTxb7hOMwMBy9MD4Ut/hYKA+gDSYIMr5ej9L+9vd1c6EMImQDB6wZhQ+kiQycFr/0cGxtTc3Oz3v/+91tw7i8RPjjgu/eeegRCbGSeNWpvb7d2TVVVVcbkMJ+kG9hs8YuE6aOVkQ86fEEMgQzWLv4P40QIzyZNWs4HG2yKeIOVazyRTnC4S291ffEtxDzzQOCAdcXm5qatR/+HAgyYB9h3XzzCONvb2zO+hOVstx8n1ZVccPg+fAEW+0dTU5N1oikUCnc0jYfpZl1gKsseAvtC0Okb0vtuDJ7t91IG4Lsj8L0fO3YsE3RzMdjc3MywerxLaKgR4nPB8Gl0WEPaiSGZ4QIK++h75d5tnJ5x8u0EuQyiNWWs7CEUsW1sbFgBGx527MUc5BTdkL5jnyzf65DJkJYn3Y+3X/k4yR55/TiBtt/rGGe5btXvyRR8cWn0BRN7jRONLUQE7KLfkzn3kDsgh+Gc9HIHfzHksopvYvkfXzHrL20+qwEpw6WF9DvnLoE6fZHRXEMUeAcBr2vkMouWj0tb+dlBpTxyMd+SknF6o3ev8WScMM5kM5hjgsoYY6bFpX/uysqKzp07lxjA/cKZM2fiN7/5TbM9gH0ob+eDLYdPAfvD0L9sLGSvi/KaOTQP6DG4eXrbClg/b/X1rTEAACAASURBVMvC4vAehZ71I82IhUe5SXVzc7O+8IUvqKamRp/97GetFRkHiH8u1XIIqzmwOGB8OzpeBgTbBEsEqb4lFixnjDGTKiBoxFYEWxQvLudAJoDA4Z7nlbclg9GDZZKUYSL5w3cA0yYVAzgMk33LM996jdsabJo3qPZt0/jve5NkngWwUWFMP/jBD7S9va3Tp0/bhgzz61lLb/IsyTa6vdhQvxnBVHonem/M7Q14OTBJYXNp4fnb29sZM25vrOy9C/nM3oPRFx2w6fuWhP45vkc0awVzdW+1wh8CRubVazHb2toyrcQ4fJnD7e3tTB/bvfzA9qqI9l5maDvr6uruqECHJWEvwAePNJfXwPliDgJY5p7KRw4zirV8eprPhnYKUbovHmK+vQcdGqRLly5Z8EqhE8UsPAsWrru729ZTCMG+37m5ObME8Ya+vvc0QdaJEyeMfSN9DEPEu4+G0GvoOMiR1PC9wrgNDAwYq4lEBU0eQcvExITGxsY0MjJil8ClpSXbU5qbm429QY83ODhoWuHm5uZMgcbo6KgxbswnZv/b29sWBHZ1denkyZM6efKkBgYG7GcUG6H75Bnj4+P2bG99QqFYd3e3jQ1msK+vz97t6upqC35nZ2c1Ojqq4eFhY8QuX75s2YHq6morZoC9GxoasmKMtrY2SW+1XpuZmTHWzrNi3kqJy1lvb6+GhoY0ODhoBtpdXV2ZHtlo9mAZYUNZk2SIGhsbMywgTDMEAlXgyIYYGzYvXAAAhXwwdjzXF3py5l6/ft0YZWxdZmdnM++hDwCxdPGX6Pr6ejszfA9onodM5o033tDm5mYKAPcDQ0ND8Wtf+5paW1szh0m57xVVfRxcNTU1tiH7yleYB14Kev76PpFsgBx8N2/etDQeL2+5RQXWFHV1dWYm6RedvxVvbm5aoMJBB0sJ09LR0WGVrjB1Xnvg2SXvI4YG5ejRoxk7BDYC38bLzx9z6BkLAtQfZQ43NjYyYm/mkM1vbW3NDiaeV24HAeNXV1eXac3l7RS4CRcKBZtD3/ScMaLramxszJjRer2JZ2Z95xJufOhtqKhkY0EX4+0HfBUobAo6RIKkcjsDCir4HGtra/b5qGbzWlSeRyVzeRUsInlEy/4S4ueQoAbrAvyyyufQ9zylUIBCo/I5RCdKsMmNH9bQ20+0tbXZfFPBt5eFi9dcorWjyhXGlDHyOShy4XvGKcDb/BCYezbb//HaMu8NhojcZx28gfReOi1ftYgB/Pr6eka76vVUXFRqa2uNISHoOnHiREYDyByiz/IOCXxmWCiYsubmZvt+/Rw2NDSYtKPcYcA/Dysm5pAes14PfezYMQu8q6urjXm7evWqPW+vOayqqsqYh/t3j0uG72fu1wxziCGzn0PGSJrU2zj5YrRyrSPFE7BIvm3eXjY/XjvoC8980Ea7PF8d7guWPEOG9U8Iwc6hxcXFzBxybnGxJHjznobofT0bTGDkNa7eQo0LfS6Xs/Xm1yHkCiw4Ug6CVm/1Qs95vw69/tLrBH33Gn/B8/sXF0d8Vsvn0GuRuTBLypj+M4e+EJM5rK2tzRTgeKsoNJe+E5avG5ifn9fLL7+smzdvpgBwP/DEE0/Eb3/72+ZVxmItN92kowVRvC9799WzvqWW3zQ5eGDqNjY2rAqX6mEvEOXWAsVOSolUAAJkbhe7u7vGvPhbJUJvRN0cohxivqKS6qVCoWCBFOkkAgpe8lwuZwcEh6I3ol5aWrJbrk/1tbS02HzhiI9lDoFjOYtDAQ5zQbqDFCcHPywWqSNS+p4N4pCRZHYUvl0XLBPMJoUhsFZ+XBw8vu0d2k30mxwQBJ4+WFxZWbE583IAWENSw96Umv6TsIleq+m1pWyKW1tbmWbqvpKXf/piHT4THm/0YOW/AZOZy+UymlffN5hDyjN8HHy0Pyq3TihPt8AQwEDEGDNV4hwA3nyb9l/19fUWgPgDFP/A6upqS635IIQLBR1a6PjR1taWsbfgIIWlDiFkOsmg3/KFLV63CJviDWPb29vteyAIpg0W7BGff3V11bSx7e3tGbaD58Kk19TUZArQYFB8QQ/G9vX19aZRg+3ArJjv3Kd40ZONjo5mdG9cxFpaWjLaNPRpnsFl/5mfn9fIyIhGR0czjIwPZDAh7u/vNw1ZT0+PBU6+InhyclLDw8MaHh42VnR+ft7muKGhwcY2MDCgoaEhDQ0NWTDY1NRk/bqvXr1qzxobG7NLt88IdXZ22nzB5OXzeWNfffHA9PS0RkZGNDw8nDExBnV1dcYQ9ff36/Tp0zp58mSmpRmSlevXr9u8jY6OGkO4vLxsLJr3GBwaGrLx+XOG4OXy5cv2LLSSk5OTlt0IIdhlvbe3157lbb/oGb2+vm7rbXx83J6F5x9VsqRXYRbRRuLjSjB59erVDKvI87w2lrPl+PHjtka8dZgkY5HR1sEko4fkTOX96ujoyFQdM7bW1lb7HCsrK6ZZ9ebaBKabm5sW6B49etTWCu/+0aNHMz2Lfczg7d02Nzc1MjKSNID7hXw+H5977jl1d3errq7OUiEcWIhGJyYmjCK+efOmWbQ0NTVlXlZuWB0dHZJkzIYXdLKA6S6CqJngrb+/P9MHENaA53G7R3iKoNlXzhHUdHd3ZwSnr732mpqamvThD39YkrS5uZlJH/EiEESsrKxYsNXa2mqfk8MLZg14DUj52FZWVrS7u2upbgTB2CVwWPMdFAoF27hnZ2fteT5gIhXd0tJin5M+llR18jw88ebn5zNj41m0AqOq17+kPI/vAIaTNcIhTQXjysqK6V182siPzXd0oKsAqbdz585pfn7egjI2S3Q+5WPr7u62IJQNxK9fNl4CPJ+q9SkTnuV90rxeCGH69PR0JgBm3so3N75Xn4L3Y+N7uHLliskvPKuJuJt0CZX2fFavuWJsU1NTmUDOSyv4rLB8iLoB34GfN6+DxQqmsbFRvb299jyehdEzVYgwR4yNYgbYQt+uimfBCHR3d1vRkKRMNoL1xp60srKi6upqex7pOt572H+etb6+bnuIT6VySVpfX7fPydjKxfDslyGEDLPl0368W6Tfm5ubM/ulrxL2vnwwHnvtlysrK7a/HTlyJPNusd4OHTp0x37J2Mr3JMyk2ZNIl/JudXZ22vpgvywfm69chgmlQAVLEdYI+yUXB7+XT05OamZmJrNf8ll9UYJPlwL2y/Kx4Ue5srKSqSrnWf69b2hosOd57TLzNjMzY8+6ffu2zRudMPx3yn4pFYsrPbni543nQYjsNTb2S4A9m9/Lp6enbd7Qa7Nfls8b+6VUlN/wOTkDJyYmTPdPZw6+Bz82LoRePoKEDE9JxgfLWigU7FlcHHi3YBi59O7u7trY2C+ff/553bhxIwWA+4Enn3wyvvTSS7bpsOl7Y02Yp7q6OlvYPT09diNAq9fU1GTVVteuXTNtCtE7nlQwMl1dXXZr55+dnZ3GFiH8xpMLDY33m/OaJhaSD1hqa2ttIc3Pz+tjH/uYtra29N73vtcMWtHokXJBo8BGCGvU1NSU0Txwg4KKn5ubU6FQMC2dF2JzkPvqNHRby8vLtinMzMwYZX7z5s2MyS23WK+L8pW1zBUb4NTUVMbPjSKb5ubmzJzDOnnzbm/L4Jup035pd3c30waIAwg7ldbWVhUKBUuv+7kiyIDtk5QRhPf09GhiYkKtra16+umnLU0Dg3D58mXb8Lz3G8xLQ0ODMUHeL4xUZnV1daanKc+C1VhdXTVrBNhp1pa3RvBaT26pzNPs7KxVd6N1Yp79OoUtDSGYjQSfj+ALLZZvlebNYVlbFCDV19dn/AJhHy5dumRzFWM0Fsq3iPJpQbSvFIWxofNe+xSoLwijcvTEiROWeuLSUCgUMhctDtUrV64YA44VFAcXWi7Y4MOHD2fSTRMTExofH7e5v3z5slVLV1dX21o4ceKEBgcHlc/nLaVIsRapPjRw4+Pj5nwA40hfUj6fZx49i+zfGSpDCaJv3LihQ4cO2XzBvOXzedtbfbEGujf2Pv747i4UvPT09OjkyZMaGhqyop+Ojo6MdpXx4AnHRQZ442gYpP7+fguGq6qqbA3Nzs5qZGREIyMjmTSxr3z3zCf/xG6ptrbW5Bgwb8PDw5qcnDR5CgVdGCJ7/R3BONXxGxsbFsz76mHfCcNLgpj7/v5+KxTyxWaXLl3KsHfT09Oam5vLZGJgsD0ri99rc3OzMeCLi4vGFk9NTVkwyHlYU1OT8Xxkrkj3U6RJGn1qakqjo6OmByTIpRitra3N1joXvhMnTpi2WZLt7zMzM/YOsS9joUbWpa+kfRwYGMj0dKZ6+vr16/Y+w/6V7FpMX+7N31kXx44dsz1we3vbgmGCUL8HLi8vq7q6WvPz80kDuF/I5/Pxy1/+sgVL0OYEEQSF09PTTLzp19hg+EL7+/sz2gWE8wRL3vmcZ0lFVsRX0nKo0ZECITCLwwckpDzwdcLrbS/tUS6X08c//nFJ0rPPPmvpNK9X4AaFoP/w4cOZZ3htmSQ7uDlAOCjn5+czljG+2pONHt0bJrfok7xXE0EKFdje2R7dCLd0b6qM4BgPN68JWlxctO8ZfRrdRnhJ6Zxw+PBhFQoF0+n46nBScphJhxDU2tqa0WqRfiSFBmtFNZrXRsK+oA2rra3VU089ZWPymjRSmXfzrvTN3X1vZq+/I2XDXHjZAjdOWsGRAift3tbWZkGF9yKjty6pQt9FhnnjwOIShB6LYIJ+t42NjXZr9hus79LgK75pp+e7HZC+rKurM3snujIQTCBZKL9QeUNYPmcul7PCI1hz/6y5uTmzOXrkkUdML8UBOTQ0lCk+IYBcWlrS+Pi4xsbGjGGdnp7OFDoRmPT29mpwcNBSb7B+3q9vbGxMw8PDGhkZsQsH9iH19fXq6OiwA5uALp/PmzYxhGB7zNTUlIaHh3Xx4kXToPIsDshTp05ZqpIguqWlxeySrly5YkULBAKjo6OZXt/eJuXMmTM6c+aMMZft7e0mKbh+/bouXLhgn4/Dln2LdcXnI+05ODhon589Hjb1/PnzunjxotnYLC4umoNCW1ubTp8+bZ+PIKWxsdHeHdiZ8fFxSxdPTk5aKrSqqsoYu3w+n/l87BG+e49POcMg+e5LBF39/f06efKkTp06pd7eXpPt7Ozs2FofHx/XxYsXNTw8nKn69d10KD4h4O3v79fhw4ftEuGLG0hfexYV8/UjR44on89bati7ORAsLS0t2fviCzEIlnZ2duyMIOgdHBw09p/UK0w6wdvExETGO4/AEgkHFxa+PyQ+VVVVmTaVjA094NLSUqar0V6MM/6WMICewWZ94t7hu750dnZmyBHW+s7OjrUX9V3J5ubm9Morr2h1dTUFgOUIIbRJek7SL0i6Jumfxxi/fq9/54knnogvvvii3TY9i4FIHq1bXV1dJoonZUPpfkNDg7E0NMKenJzU7OxsxqrCa4FYTDy3ra3N9HA3b940LSIB5MzMTMbN3RujMh5vy4FubXl5WVNTU/rc5z6X6TCBFhEzUJ824uXFwBf2gvHAGvlWaCxqnwrs6emxg1WSHSBsmqQnscqAbfCpU1J2x44V3d7ZeEkP+zEtLi5adaqv0PTNxmFs6+vrjYWEAUGDRHBdVVVlzBIvPgwUhs+SrHzfMyDMkWcs0Lj5VJi/Na+uruqFF17Q+vq6BgcHjUWG0fKVmLDIuMsjTgbc5Cn6YEy+itf3I+bG3N3dbeO5ffu2XT5goknb4qnHeJAJEDQxppqaGvPl8yJ4LlesRUyVCVCZH7zdYE9gwEgBcVMmMIQZbmhosGCQ8Zw4ccIOt6qqqoznFkEm2l/6QnPZ8xo5ND1UflJIxTMQ0i8sLFgnm9raWhuDX0ft7e122GLv5JnsqakpCyy3trbsoCCg6OvrM7P0trY2bW5umuWHZyZ8K0BYOzROzBHBLtY/29vbd0gKpqamTBu6sbGR8SyF7aLTAUbg3roKlgo987Vr12w8jY2NNhb/T6rdYUG4qPhncVGjuKCtrS2TqqcDT0NDQ6b1IPPjPewkmc+pb3PGd4cna21trV00r1y5YuMhQMLMGQa3u7vb1pA/9Pl+YYAIHAjafHGf70ns29Gxxm7dumXvgn9nScl71vrRRx+18XAx52KIXt3r5Qhubty4YRIAPpdn5fnOYN3YL7xMgBQwqXMu9oynp6fHxpPL5WyPgAX0fXkXFhaIA5TL5WzvYX4I2tmrvM+vrwbmgl5VVZVpocca8p1u2NeXl5dtfgjWLl26pO3tbRsTmRgyKr29vfa56uvrrUhvcXHxDjkLXZqam5s1Pj6eAsC9EEL4hqQqSf9A0s9I+s+Sfi7GeP5u/87Q0FB8/vnn1dLSYgUNpLR8r8nFxUW71SPO990kfKonhGB9en03CehlFg1FEVSR+Yo5tGtsCqTsMAPlpWE89BCGgaTCz2uIVlZW9MlPflI7Ozt6+umnTVOD+SYNvv3Nt62tzW7NpKUxF/YNwrF78WbLeKeh56KwAvaKXqpY5fDH+9lhfUKFNI3dvQ2IVAy+vNkxN2+CZW99guGxbxJPdRiWMoVCwZ7DzW1tbc3YK0nG6tGT0heO5HI5SycTzPt5okAD+M/X3NysCxcuKJfL6T3veY9p1EjVY/rtU5Cw0oyJw48NDCsWLGV8FbQvRMGc+pFHHsnYFngjYKqpvVE4BzKpGDZQuszgx0fw29HRYSl3XPB5hq8e9xXonln1vbe5wODHhdCeIIzPiY4SXzcuF75aFVG3N+NlU5+dnbU1HmPMHMI+pcMaKBQKFhSQWh0dHbWCpPX19Yx/GUwVzGVnZ6dVkK6vrxtrNjY2ZsEBaaHq6modPXrU7E5gcwYGBmyPuH37tn2eiYkJY4WuXLliKVI89Lq7u3X27FmdOXNGAwMDNve0SFteXtb4+LguXLigCxcuZITqvFPt7e3GDA4NDdm4crmcmazDepL+vHjxYsYyBLlGT0+PTp8+rccee0xnzpyx4KWurs4CnampKZ0/f17nz5+3jM38/LxdJltbW3X27FmdPXtWp06dskO4sbHR9pzZ2VljFn1hBev00KFD9l0PDQ3p9OnTNh7sbzj8Z2dndeHCBZ0/f16Tk5MZNolgeWhoyNhA76/IfrOwsGBsIJfTmZkZs86pra3d8zunTzpt3CA1RkdHNTIyovHx8Yzcg6Cmp6dHp06d0smTJzP+pjB93tpmqqRBJZVL5bZnJ73lEFhbW7P58Rccgrutra07TJMJljlH8Ie8ceOGMZP00F1aWtL6+rqdERTBMA4YeKRR5ZkP/rdv9+ptqLwPKGej9FYVMM4N5bZR7M2NjY2ZhhGsZXxE8RJlvbCn0V701Vdf1crKSgoAPUIIhyUtS3o8xjhS+tnzkuZijL91t38PI2iaVfvqO1K2HLCS7IVA19bb25vpFMDht7S0ZIcQVDLVpwR4/rZE1WNTU5Ntfjj3cwvkcJRkqQ6vh/NaLwJR2kIhcP3617+uQqGgoaEhMyLmpaAoo6enx4ynDx06ZAECNxOYBBY8ljiU3vMMAshDhw7Zhnb16lWbW1IRV69etZQzzu68sN58E989b/fgW35JstZRvKS+2bfvbcn3PD8/b8wYFWCYlpLS4DvisMAAlpQt66W88pOXHX8qNkPfgxWan5s6utM33nhDq6urZobMevEWG+WFS95yhwOZamxSwWysnjlgE8MUmg2Mz+SbqfvLgWce2cRgetbW1qzAAF1nOVvY0tKS+a5han0nE6/D9C0U+a691Qrf8/Hjxy39E2O0ixOMGoETBw4eeTBhCM99JwFYYnSA09PTmbZa/F0YUJ7Bc7mg3Lp1K+MXxju9trZmDNiRI0fss3jrE7C9vZ1hUUlF0+6OFBOXUs8QEZzu7OxYuhiPtenp6T11jfi1oXHlcukvNqSKyzuB+JQz4+Cy29nZaeOFqYY9gW3EiFuSXSJgmlgzzBvMEK3PYK34nguFgn0faFHZe7l8weZ4jTPdQsq99giYyLoQ1CETwhMPIgEZSqFQuEO24HXb7e3tFiCvra1lChRg8ulvjLYM+RDspO/V7SveCbjm5uaMIfbspM8okJVobW21998zXbTtW1pasvaCSIb4rr1ejssw3pX+u0a3Cnng2TK+a4KkmpoaM+1fWFjI9Pfl7MTyh5Zx7OEE6g0NDVZFDWvPOUtgGULIFFP6eaH1nu+tznrh0rewsKCqqirrBuQ17N7ejQulL7rxc4OzCNKG119/XVevXk0BoEcI4S9L+l6M8ZD72T+W9K4Y4/vK/u6vSfq10v99XNJfPLCB/nSgXcUUekIWaV72RpqXvZHm5U6kOdkbaV72RpqXO3Eqxtj44z6kZj9G8hOEBkk3y362IumOiYoxflHSFyUphHAuxvizb//wfnqQ5mRvpHnZG2le9kaalzuR5mRvpHnZG2le7kQI4dx+PKfq/n/lpwprkprKftYkafUAxpKQkJCQkJCQ8BOJhy0AHJFUE0IYcj97UtJdC0ASEhISEhISEioND1UAGGNcl/SSpM+EEA6HEP6GpF+W9Px9/tUvvu2D++lDmpO9keZlb6R52RtpXu5EmpO9keZlb6R5uRP7MicPVRGIZD6AX5b085KWJP3W/XwAExISEhISEhIqCQ9dAJiQkJCQkJCQkHBvPFQp4ISEhISEhISEhPsjBYAJCQkJCQkJCRWGig4AQwhtIYQ/CCGshxCmQwgfPOgxPWiEEOpCCM+VPv9qCOH/hhD+bul3fSGEGEJYc38+edBjflAIIXw3hLDpPvuw+90HS3O2HkL4w5L29KFG2TpYCyHcDiH8u9LvKmqthBA+EUI4F0IohBB+r+x37w4hXAwhbIQQ/kcIodf9ri6E8OUQws0QwkII4Tcf+ODfJtxtTkIIfz2E8F9DCNdDCFdDCC+GELrc758JIWyXrZ38gXyItwH3mJd7vjMP81qR7jkvHyqbk43SPL2j9PuHdr3c6zwu/X5f95aKDgAl/XtJW5KOSvqQpC+EEB472CE9cNRImpX0LknNkn5b0rdCCH3u77TEGBtKf37nwQ/xQPEJ99lPSVJpjfyupA+ruHY2JP2HAxzjA4GbhwZJnZJuSXqx7K9Vylq5LOlfqlhwZgghtKvoRPBJSW2Szkn6ffdXnpE0JKlX0t+W9E9DCH/nAYz3QWDPOZHUqmLVYp+Kn3tV0lfK/s7v+/UVY5x4uwf7AHG3eQF3e2ee0cO7VqS7zEuM8YWyvebXJU1I+j/urz2s6+Wu5/Hbsbc8bJ1AfmSEYt/gD6jYN3hN0p+FEP5IxUP9rn2DHzaUrHOecT96JYQwKekdkn5wIIP6yceHJH0nxvinklS6tb8ZQmiMMVaK6fgHJC1KevWgB3IQiDG+JEkhhJ+VdNz96v2SzscYXyz9/hlJ10IIp2OMFyV9RNJHY4zLkpZDCF+S9FFJf/IAh/+24G5zEmP8Y//3Qgifl/Q/H+zoDg73WCv3w0O7VqT/r3n5iKSvxQqoWL3PeXxE+7y3VDIDeFLSToxxxP3sh5IqjQHMIIRwVMW58ebZ0yGESyGEr5RuIZWEfxVCuBZC+F4I4W+VfvaYimtFkhRjHFeRST55AOM7KNxtU67ktSLduTbWJY1LeiyE0Cqpy/9elbnnvFN3mvO/r5QiPh9C+PhBDOoAccc7k9ZKEaUU5zslfa3sVxWxXsrO433fWyo5APyR+wZXCkIIj0h6QdJXSzeKa5L+ioqU8jtUnJsXDm6EDxz/TFJeUreKKazvhBAGVFw7K2V/t2LWTmlTfpekr7ofV/paAfdaGw3u/5f/riIQQvhLkj4l6Z+4H39L0hlJHZL+oaRPhRB+5QCG96Bxr3em4tdKCb8q6dUY46T7WUWslz3O433fWyo5AEx9gx1CCFUqdkzZkvQJSYoxrsUYz8UYd2KMV0o//4UQQkVsQjHG78cYV2OMhRjjVyV9T9IvKq2dD0v6M78pV/pacbjX2lhz/7/8dw89QgiDkv5Y0j+KMZp0IMZ4IcZ4OcZ4O8b4vyT9G0l/76DG+aBwn3emoteKw68qe9GsiPWy13mst2FvqeQAMPUNLiGEECQ9p2JBwwdijNt3+auk+yp13URJQcU18iQ/LFWg1am4pioBd2zKe6BS10r52jgsaUBF7c6ypHn/e1XInlNijf+bpN+JMd6vNSfvWaXB3plKXisgFFu5HpP0n+7zVx+q9XKP83jf95ZK25wNP0bf4IcRX1CRUn9fjPEWPwwh/LUQwqkQQlUI4YikfyvpuzHGchr6oUMIoSWE8N4QQi6EUBNC+JCKWpQ/UZGWf18I4W+WXsLPSHqpEgpAQgg/p2JK/MWyn1fUWimtiZykaknVrBNJfyDp8RDCB0q//5SkPy+lcKSilum3QwitIYTTKqawfu8APsK+425zEkLolvTfJX0+xvgf9/j3frk0HyGE8Fcl/Yaklx/s6N8+3GNe7vfOPLRrRbrnOwQ+Iunb5fvqw75edJfzWG/H3hJjrNg/KpZS/6GkdUkzkj540GM6gDnoVfEGtakijcyfD0n6FUmTpfmZLy2wzoMe8wOalw5J/1tFCv2GpNcl/bz7/QdLa2Zdxc2n7aDH/IDm5XclPb/HzytqrahYqRfL/jxT+t17JF1U0Sbnu5L63L9Xp6LtxU1JVyT95kF/lrd7TiR9uvS//f6y5v69b6jYt32tNG+/cdCf5QHNyz3fmYd5rdxrXkq/y5X23Xfv8e89tOvlXudx6ff7urekXsAJCQkJCQkJCRWGik0BJyQkJCQkJCRUKlIAmJCQkJCQkJBQYUgBYEJCQkJCQkJChSEFgAkJCQkJCQkJFYYUACYkJCQkJCQkVBhSAJiQkJCQkJCQUGFIAWBCQkJCQkJCQoUhBYAJCQkJCQkJCRWGFAAmJCQk7CNCCN8PIXwrhPCZEMJ4CGEzhPDnIYR3H/TYEhISEkDqBJKQkJCw8VyzvwAAAcJJREFUTyj1Ml2VtCvp+5L+taQaSZ9VsYdyPsa4dHAjTEhISCii5v5/JSEhISHhR8RZFfuY/qmKvaNvS1II4bqKvTvfqWJT94SEhIQDRUoBJyQkJOwfnir9818Q/JVwsfTPIw94PAkJCQl7IgWACQkJCfuHd0i6HGP8XtnPj5X+eekBjychISFhT6QAMCEhIWH/8JSkuT1+/vclbUh69cEOJyEhIWFvJA1gQkJCwj4ghFAl6UlJ6yGEmhjjTunnxyT9uqTPxxjXD3KMCQkJCSBVASckJCTsA0IIZyWdlzSrYhHIVyQdl/QpSUuS3hlj3Dy4ESYkJCS8hZQCTkhISNgfUADyi5JaJH1H0uck/RdJ707BX0JCwk8SUgo4ISEhYX/wlKRLMca/kPRLBz2YhISEhHshMYAJCQkJ+4N3SPrBQQ8iISEh4UdBCgATEhISfkyEEIKkn1EKABMSEn5KkIpAEhISEhISEhIqDIkBTEhISEhISEioMKQAMCEhISEhISGhwpACwISEhISEhISECkMKABMSEhISEhISKgwpAExISEhISEhIqDCkADAhISEhISEhocKQAsCEhISEhISEhArD/wNx3M73ISUtIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 648x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "i1, i2, crop_i = 100, 101, 150\n",
    "p1, p2, p3 = 22, 60, 35\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(9, 5))\n",
    "ax1.plot([p1, p1], [-1, 1], \"k--\", label=\"$p = {}$\".format(p1))\n",
    "ax1.plot([p2, p2], [-1, 1], \"k--\", label=\"$p = {}$\".format(p2), alpha=0.5)\n",
    "ax1.plot(p3, PE[p3, i1], \"bx\", label=\"$p = {}$\".format(p3))\n",
    "ax1.plot(PE[:,i1], \"b-\", label=\"$i = {}$\".format(i1))\n",
    "ax1.plot(PE[:,i2], \"r-\", label=\"$i = {}$\".format(i2))\n",
    "ax1.plot([p1, p2], [PE[p1, i1], PE[p2, i1]], \"bo\")\n",
    "ax1.plot([p1, p2], [PE[p1, i2], PE[p2, i2]], \"ro\")\n",
    "ax1.legend(loc=\"center right\", fontsize=14, framealpha=0.95)\n",
    "ax1.set_ylabel(\"$P_{(p,i)}$\", rotation=0, fontsize=16)\n",
    "ax1.grid(True, alpha=0.3)\n",
    "ax1.hlines(0, 0, max_steps - 1, color=\"k\", linewidth=1, alpha=0.3)\n",
    "ax1.axis([0, max_steps - 1, -1, 1])\n",
    "ax2.imshow(PE.T[:crop_i], cmap=\"gray\", interpolation=\"bilinear\", aspect=\"auto\")\n",
    "ax2.hlines(i1, 0, max_steps - 1, color=\"b\")\n",
    "cheat = 2 # need to raise the red line a bit, or else it hides the blue one\n",
    "ax2.hlines(i2+cheat, 0, max_steps - 1, color=\"r\")\n",
    "ax2.plot([p1, p1], [0, crop_i], \"k--\")\n",
    "ax2.plot([p2, p2], [0, crop_i], \"k--\", alpha=0.5)\n",
    "ax2.plot([p1, p2], [i2+cheat, i2+cheat], \"ro\")\n",
    "ax2.plot([p1, p2], [i1, i1], \"bo\")\n",
    "ax2.axis([0, max_steps - 1, 0, crop_i])\n",
    "ax2.set_xlabel(\"$p$\", fontsize=16)\n",
    "ax2.set_ylabel(\"$i$\", rotation=0, fontsize=16)\n",
    "save_fig(\"positional_embedding_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "id": "wLUQDsVOtv8h"
   },
   "outputs": [],
   "source": [
    "embed_size = 512; max_steps = 500; vocab_size = 10000\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)\n",
    "positional_encoding = PositionalEncoding(max_steps, max_dims=embed_size)\n",
    "encoder_in = positional_encoding(encoder_embeddings)\n",
    "decoder_in = positional_encoding(decoder_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kkXbid7xtv8h"
   },
   "source": [
    "Here is a (very) simplified Transformer (the actual architecture has skip connections, layer norm, dense nets, and most importantly it uses Multi-Head Attention instead of regular Attention):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "id": "KMNOJwcKtv8h"
   },
   "outputs": [],
   "source": [
    "Z = encoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, Z])\n",
    "\n",
    "encoder_outputs = Z\n",
    "Z = decoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True, causal=True)([Z, Z])\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, encoder_outputs])\n",
    "\n",
    "outputs = keras.layers.TimeDistributed(\n",
    "    keras.layers.Dense(vocab_size, activation=\"softmax\"))(Z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ORq4ECstv8h"
   },
   "source": [
    "Here's a basic implementation of the `MultiHeadAttention` layer. One will likely be added to `keras.layers` in the near future. Note that `Conv1D` layers with `kernel_size=1` (and the default `padding=\"valid\"` and `strides=1`) is equivalent to a `TimeDistributed(Dense(...))` layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "I5LYzX1Qtv8h"
   },
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class MultiHeadAttention(keras.layers.Layer):\n",
    "    def __init__(self, n_heads, causal=False, use_scale=False, **kwargs):\n",
    "        self.n_heads = n_heads\n",
    "        self.causal = causal\n",
    "        self.use_scale = use_scale\n",
    "        super().__init__(**kwargs)\n",
    "    def build(self, batch_input_shape):\n",
    "        self.dims = batch_input_shape[0][-1]\n",
    "        self.q_dims, self.v_dims, self.k_dims = [self.dims // self.n_heads] * 3 # could be hyperparameters instead\n",
    "        self.q_linear = keras.layers.Conv1D(self.n_heads * self.q_dims, kernel_size=1, use_bias=False)\n",
    "        self.v_linear = keras.layers.Conv1D(self.n_heads * self.v_dims, kernel_size=1, use_bias=False)\n",
    "        self.k_linear = keras.layers.Conv1D(self.n_heads * self.k_dims, kernel_size=1, use_bias=False)\n",
    "        self.attention = keras.layers.Attention(causal=self.causal, use_scale=self.use_scale)\n",
    "        self.out_linear = keras.layers.Conv1D(self.dims, kernel_size=1, use_bias=False)\n",
    "        super().build(batch_input_shape)\n",
    "    def _multi_head_linear(self, inputs, linear):\n",
    "        shape = K.concatenate([K.shape(inputs)[:-1], [self.n_heads, -1]])\n",
    "        projected = K.reshape(linear(inputs), shape)\n",
    "        perm = K.permute_dimensions(projected, [0, 2, 1, 3])\n",
    "        return K.reshape(perm, [shape[0] * self.n_heads, shape[1], -1])\n",
    "    def call(self, inputs):\n",
    "        q = inputs[0]\n",
    "        v = inputs[1]\n",
    "        k = inputs[2] if len(inputs) > 2 else v\n",
    "        shape = K.shape(q)\n",
    "        q_proj = self._multi_head_linear(q, self.q_linear)\n",
    "        v_proj = self._multi_head_linear(v, self.v_linear)\n",
    "        k_proj = self._multi_head_linear(k, self.k_linear)\n",
    "        multi_attended = self.attention([q_proj, v_proj, k_proj])\n",
    "        shape_attended = K.shape(multi_attended)\n",
    "        reshaped_attended = K.reshape(multi_attended, [shape[0], self.n_heads, shape_attended[1], shape_attended[2]])\n",
    "        perm = K.permute_dimensions(reshaped_attended, [0, 2, 1, 3])\n",
    "        concat = K.reshape(perm, [shape[0], shape_attended[1], -1])\n",
    "        return self.out_linear(concat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OrDV_52Ntv8h",
    "outputId": "19b1c5b5-87ec-48f4-c66e-5a403a090028"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 50, 512])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q = np.random.rand(2, 50, 512)\n",
    "V = np.random.rand(2, 80, 512)\n",
    "multi_attn = MultiHeadAttention(8)\n",
    "multi_attn([Q, V]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ql_LgBBXtv8i"
   },
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gVxTeZyYtv8i"
   },
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a0CiD550tv8i"
   },
   "source": [
    "See Appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2dul3AEmtv8i"
   },
   "source": [
    "## 8.\n",
    "_Exercise:_ Embedded Reber grammars _were used by Hochreiter and Schmidhuber in [their paper](https://homl.info/93) about LSTMs. They are artificial grammars that produce strings such as \"BPBTSXXVPSEPE.\" Check out Jenny Orr's [nice introduction](https://homl.info/108) to this topic. Choose a particular embedded Reber grammar (such as the one represented on Jenny Orr's page), then train an RNN to identify whether a string respects that grammar or not. You will first need to write a function capable of generating a training batch containing about 50% strings that respect the grammar, and 50% that don't._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JQKIcaNOtv8i"
   },
   "source": [
    "First we need to build a function that generates strings based on a grammar. The grammar will be represented as a list of possible transitions for each state. A transition specifies the string to output (or a grammar to generate it) and the next state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "zAPnSnshtv8i"
   },
   "outputs": [],
   "source": [
    "default_reber_grammar = [\n",
    "    [(\"B\", 1)],           # (state 0) =B=>(state 1)\n",
    "    [(\"T\", 2), (\"P\", 3)], # (state 1) =T=>(state 2) or =P=>(state 3)\n",
    "    [(\"S\", 2), (\"X\", 4)], # (state 2) =S=>(state 2) or =X=>(state 4)\n",
    "    [(\"T\", 3), (\"V\", 5)], # and so on...\n",
    "    [(\"X\", 3), (\"S\", 6)],\n",
    "    [(\"P\", 4), (\"V\", 6)],\n",
    "    [(\"E\", None)]]        # (state 6) =E=>(terminal state)\n",
    "\n",
    "embedded_reber_grammar = [\n",
    "    [(\"B\", 1)],\n",
    "    [(\"T\", 2), (\"P\", 3)],\n",
    "    [(default_reber_grammar, 4)],\n",
    "    [(default_reber_grammar, 5)],\n",
    "    [(\"T\", 6)],\n",
    "    [(\"P\", 6)],\n",
    "    [(\"E\", None)]]\n",
    "\n",
    "def generate_string(grammar):\n",
    "    state = 0\n",
    "    output = []\n",
    "    while state is not None:\n",
    "        index = np.random.randint(len(grammar[state]))\n",
    "        production, state = grammar[state][index]\n",
    "        if isinstance(production, list):\n",
    "            production = generate_string(grammar=production)\n",
    "        output.append(production)\n",
    "    return \"\".join(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nC0EgwRttv8i"
   },
   "source": [
    "Let's generate a few strings based on the default Reber grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rAdE926Htv8i",
    "outputId": "5622b6b2-75a5-43d9-89fc-ef7281698384"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTXXTTVPXTVPXTTVPSE BPVPSE BTXSE BPVVE BPVVE BTSXSE BPTVPXTTTVVE BPVVE BTXSE BTXXVPSE BPTTTTTTTTVVE BTXSE BPVPSE BTXSE BPTVPSE BTXXTVPSE BPVVE BPVVE BPVVE BPTTVVE BPVVE BPVVE BTXXVVE BTXXVVE BTXXVPXVVE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_string(default_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MMjDaDs3tv8i"
   },
   "source": [
    "Looks good. Now let's generate a few strings based on the embedded Reber grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vh1GBGCEtv8k",
    "outputId": "56cae26b-7c7c-406a-e389-e72223c2f095"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTBPTTTVPXTVPXTTVPSETE BPBPTVPSEPE BPBPVVEPE BPBPVPXVVEPE BPBTXXTTTTVVEPE BPBPVPSEPE BPBTXXVPSEPE BPBTSSSSSSSXSEPE BTBPVVETE BPBTXXVVEPE BPBTXXVPSEPE BTBTXXVVETE BPBPVVEPE BPBPVVEPE BPBTSXSEPE BPBPVVEPE BPBPTVPSEPE BPBTXXVVEPE BTBPTVPXVVETE BTBPVVETE BTBTSSSSSSSXXVVETE BPBTSSSXXTTTTVPSEPE BTBPTTVVETE BPBTXXTVVEPE BTBTXSETE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_string(embedded_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3AUMi5Rxtv8k"
   },
   "source": [
    "Okay, now we need a function to generate strings that do not respect the grammar. We could generate a random string, but the task would be a bit too easy, so instead we will generate a string that respects the grammar, and we will corrupt it by changing just one character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "id": "lB3U7RTTtv8k"
   },
   "outputs": [],
   "source": [
    "POSSIBLE_CHARS = \"BEPSTVX\"\n",
    "\n",
    "def generate_corrupted_string(grammar, chars=POSSIBLE_CHARS):\n",
    "    good_string = generate_string(grammar)\n",
    "    index = np.random.randint(len(good_string))\n",
    "    good_char = good_string[index]\n",
    "    bad_char = np.random.choice(sorted(set(chars) - set(good_char)))\n",
    "    return good_string[:index] + bad_char + good_string[index + 1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "owPPvYwttv8k"
   },
   "source": [
    "Let's look at a few corrupted strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "v82ibd9_tv8k",
    "outputId": "040a400e-dc02-4f1a-92d9-3763ecd1459c"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTBPTTTPPXTVPXTTVPSETE BPBTXEEPE BPBPTVVVEPE BPBTSSSSXSETE BPTTXSEPE BTBPVPXTTTTTTEVETE BPBTXXSVEPE BSBPTTVPSETE BPBXVVEPE BEBTXSETE BPBPVPSXPE BTBPVVVETE BPBTSXSETE BPBPTTTPTTTTTVPSEPE BTBTXXTTSTVPSETE BBBTXSETE BPBTPXSEPE BPBPVPXTTTTVPXTVPXVPXTTTVVEVE BTBXXXTVPSETE BEBTSSSSSXXVPXTVVETE BTBXTTVVETE BPBTXSTPE BTBTXXTTTVPSBTE BTBTXSETX BTBTSXSSTE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_corrupted_string(embedded_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xw7T7asjtv8k"
   },
   "source": [
    "We cannot feed strings directly to an RNN, so we need to encode them somehow. One option would be to one-hot encode each character. Another option is to use embeddings. Let's go for the second option (but since there are just a handful of characters, one-hot encoding would probably be a good option as well). For embeddings to work, we need to convert each string into a sequence of character IDs. Let's write a function for that, using each character's index in the string of possible characters \"BEPSTVX\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "DjetSTB3tv8k"
   },
   "outputs": [],
   "source": [
    "def string_to_ids(s, chars=POSSIBLE_CHARS):\n",
    "    return [chars.index(c) for c in s]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yrs-YnM8tv8k",
    "outputId": "698c4f79-5f61-4fb6-bf10-c7924fbc64d1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 4, 4, 4, 6, 6, 5, 5, 1, 4, 1]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string_to_ids(\"BTTTXXVVETE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "orZ26XY9tv8k"
   },
   "source": [
    "We can now generate the dataset, with 50% good strings, and 50% bad strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "id": "Vibpk-Qftv8k"
   },
   "outputs": [],
   "source": [
    "def generate_dataset(size):\n",
    "    good_strings = [string_to_ids(generate_string(embedded_reber_grammar))\n",
    "                    for _ in range(size // 2)]\n",
    "    bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))\n",
    "                   for _ in range(size - size // 2)]\n",
    "    all_strings = good_strings + bad_strings\n",
    "    X = tf.ragged.constant(all_strings, ragged_rank=1)\n",
    "    y = np.array([[1.] for _ in range(len(good_strings))] +\n",
    "                 [[0.] for _ in range(len(bad_strings))])\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "QO4l6qD1tv8l"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "X_train, y_train = generate_dataset(10000)\n",
    "X_valid, y_valid = generate_dataset(2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OlKDQ72atv8l"
   },
   "source": [
    "Let's take a look at the first training sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tAhssGmetv8l",
    "outputId": "8d1d9201-1149-46b0-8693-d85abecb0f16"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(22,), dtype=int32, numpy=\n",
       "array([0, 4, 0, 2, 4, 4, 4, 5, 2, 6, 4, 5, 2, 6, 4, 4, 5, 2, 3, 1, 4, 1],\n",
       "      dtype=int32)>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ChQN2o-htv8l"
   },
   "source": [
    "What class does it belong to?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uFcQIlNXtv8l",
    "outputId": "4e7df68b-22e3-4048-aa0b-ff954376d61c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vZ4Rs1Wctv8l"
   },
   "source": [
    "Perfect! We are ready to create the RNN to identify good strings. We build a simple sequence binary classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lFWo2np6tv8l",
    "outputId": "5c67e34f-c41a-4e2f-dd49-43eef4c1c697"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/sequential_6/gru_12/RaggedToTensor/boolean_mask_1/GatherV2:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/sequential_6/gru_12/RaggedToTensor/boolean_mask/GatherV2:0\", shape=(None, 5), dtype=float32), dense_shape=Tensor(\"gradient_tape/sequential_6/gru_12/RaggedToTensor/Shape:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 18s 53ms/step - loss: 0.6910 - accuracy: 0.5095 - val_loss: 0.6825 - val_accuracy: 0.5645\n",
      "Epoch 2/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.6678 - accuracy: 0.5659 - val_loss: 0.6635 - val_accuracy: 0.6105\n",
      "Epoch 3/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.6504 - accuracy: 0.5766 - val_loss: 0.6521 - val_accuracy: 0.6110\n",
      "Epoch 4/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.6347 - accuracy: 0.5980 - val_loss: 0.6224 - val_accuracy: 0.6445\n",
      "Epoch 5/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.6054 - accuracy: 0.6361 - val_loss: 0.5779 - val_accuracy: 0.6980\n",
      "Epoch 6/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.5414 - accuracy: 0.7093 - val_loss: 0.4695 - val_accuracy: 0.7795\n",
      "Epoch 7/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.3913 - accuracy: 0.8320 - val_loss: 0.2796 - val_accuracy: 0.8955\n",
      "Epoch 8/20\n",
      "313/313 [==============================] - 16s 53ms/step - loss: 0.4481 - accuracy: 0.7648 - val_loss: 0.5198 - val_accuracy: 0.6870\n",
      "Epoch 9/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.4590 - accuracy: 0.7721 - val_loss: 0.3302 - val_accuracy: 0.8660\n",
      "Epoch 10/20\n",
      "313/313 [==============================] - 16s 53ms/step - loss: 0.2588 - accuracy: 0.9078 - val_loss: 0.1560 - val_accuracy: 0.9715\n",
      "Epoch 11/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.1452 - accuracy: 0.9580 - val_loss: 0.1371 - val_accuracy: 0.9605\n",
      "Epoch 12/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.0698 - accuracy: 0.9834 - val_loss: 0.0417 - val_accuracy: 0.9885\n",
      "Epoch 13/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.0835 - accuracy: 0.9776 - val_loss: 0.0347 - val_accuracy: 0.9895\n",
      "Epoch 14/20\n",
      "313/313 [==============================] - 16s 53ms/step - loss: 0.0402 - accuracy: 0.9913 - val_loss: 0.0168 - val_accuracy: 0.9980\n",
      "Epoch 15/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.0275 - accuracy: 0.9953 - val_loss: 0.0082 - val_accuracy: 0.9990\n",
      "Epoch 16/20\n",
      "313/313 [==============================] - 16s 52ms/step - loss: 0.0108 - accuracy: 0.9979 - val_loss: 0.0102 - val_accuracy: 0.9960\n",
      "Epoch 17/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.0136 - accuracy: 0.9972 - val_loss: 0.0084 - val_accuracy: 0.9990\n",
      "Epoch 18/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.0070 - accuracy: 0.9988 - val_loss: 0.0080 - val_accuracy: 0.9990\n",
      "Epoch 19/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.0057 - accuracy: 0.9986 - val_loss: 0.0029 - val_accuracy: 0.9995\n",
      "Epoch 20/20\n",
      "313/313 [==============================] - 17s 53ms/step - loss: 0.0338 - accuracy: 0.9924 - val_loss: 0.0059 - val_accuracy: 0.9975\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "embedding_size = 5\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),\n",
    "    keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS), output_dim=embedding_size),\n",
    "    keras.layers.GRU(30),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(learning_rate=0.02, momentum = 0.95, nesterov=True)\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train, y_train, epochs=20, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0oUkSIN4tv8l"
   },
   "source": [
    "Now let's test our RNN on two tricky strings: the first one is bad while the second one is good. They only differ by the second to last character. If the RNN gets this right, it shows that it managed to notice the pattern that the second letter should always be equal to the second to last letter. That requires a fairly long short-term memory (which is the reason why we used a GRU cell)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "woeAPArztv8l",
    "outputId": "7513a4d5-351a-4088-aecf-00bdabecc8a3"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Estimated probability that these are Reber strings:\n",
      "BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE: 0.06%\n",
      "BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE: 91.51%\n"
     ]
    }
   ],
   "source": [
    "test_strings = [\"BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE\",\n",
    "                \"BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE\"]\n",
    "X_test = tf.ragged.constant([string_to_ids(s) for s in test_strings], ragged_rank=1)\n",
    "\n",
    "y_proba = model.predict(X_test)\n",
    "print()\n",
    "print(\"Estimated probability that these are Reber strings:\")\n",
    "for index, string in enumerate(test_strings):\n",
    "    print(\"{}: {:.2f}%\".format(string, 100 * y_proba[index][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jJ5RpKJQtv8l"
   },
   "source": [
    "Ta-da! It worked fine. The RNN found the correct answers with very high confidence. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0gdLLJPDtv8l"
   },
   "source": [
    "## 9.\n",
    "_Exercise: Train an Encoder–Decoder model that can convert a date string from one format to another (e.g., from \"April 22, 2019\" to \"2019-04-22\")._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7MQ_bXh7tv8m"
   },
   "source": [
    "Let's start by creating the dataset. We will use random days between 1000-01-01 and 9999-12-31:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "id": "3vJWvpzYtv8m"
   },
   "outputs": [],
   "source": [
    "from datetime import date\n",
    "\n",
    "# cannot use strftime()'s %B format since it depends on the locale\n",
    "MONTHS = [\"January\", \"February\", \"March\", \"April\", \"May\", \"June\",\n",
    "          \"July\", \"August\", \"September\", \"October\", \"November\", \"December\"]\n",
    "\n",
    "def random_dates(n_dates):\n",
    "    min_date = date(1000, 1, 1).toordinal()\n",
    "    max_date = date(9999, 12, 31).toordinal()\n",
    "\n",
    "    ordinals = np.random.randint(max_date - min_date, size=n_dates) + min_date\n",
    "    dates = [date.fromordinal(ordinal) for ordinal in ordinals]\n",
    "\n",
    "    x = [MONTHS[dt.month - 1] + \" \" + dt.strftime(\"%d, %Y\") for dt in dates]\n",
    "    y = [dt.isoformat() for dt in dates]\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fweZNTfWtv8m"
   },
   "source": [
    "Here are a few random dates, displayed in both the input format and the target format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VmBpLCECtv8m",
    "outputId": "336a8226-1969-44ff-d3eb-88f29003039b"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input                    Target                   \n",
      "--------------------------------------------------\n",
      "September 20, 7075       7075-09-20               \n",
      "May 15, 8579             8579-05-15               \n",
      "January 11, 7103         7103-01-11               \n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "n_dates = 3\n",
    "x_example, y_example = random_dates(n_dates)\n",
    "print(\"{:25s}{:25s}\".format(\"Input\", \"Target\"))\n",
    "print(\"-\" * 50)\n",
    "for idx in range(n_dates):\n",
    "    print(\"{:25s}{:25s}\".format(x_example[idx], y_example[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Md_T4D2Otv8m"
   },
   "source": [
    "Let's get the list of all possible characters in the inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "6Q5kdGjNtv8m",
    "outputId": "7a56dd61-6a62-4425-a449-1d04dde3b782"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "' ,0123456789ADFJMNOSabceghilmnoprstuvy'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "INPUT_CHARS = \"\".join(sorted(set(\"\".join(MONTHS) + \"0123456789, \")))\n",
    "INPUT_CHARS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hVUgoO8ptv8m"
   },
   "source": [
    "And here's the list of possible characters in the outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "id": "-7cQ8Rkrtv8m"
   },
   "outputs": [],
   "source": [
    "OUTPUT_CHARS = \"0123456789-\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zIiXQMawtv8m"
   },
   "source": [
    "Let's write a function to convert a string to a list of character IDs, as we did in the previous exercise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "id": "28gfYt5Ttv8m"
   },
   "outputs": [],
   "source": [
    "def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
    "    return [chars.index(c) for c in date_str]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iyEmwC9Ltv8m",
    "outputId": "75b750b1-7c8e-48ab-eac5-5b7d1a817387"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[19, 23, 31, 34, 23, 28, 21, 23, 32, 0, 4, 2, 1, 0, 9, 2, 9, 7]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_str_to_ids(x_example[0], INPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xj7gwXTXtv8m",
    "outputId": "609f2479-2293-478f-c777-dfe55b6d53ed"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7, 0, 7, 5, 10, 0, 9, 10, 2, 0]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_str_to_ids(y_example[0], OUTPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "id": "0q2sw_8ptv8n"
   },
   "outputs": [],
   "source": [
    "def prepare_date_strs(date_strs, chars=INPUT_CHARS):\n",
    "    X_ids = [date_str_to_ids(dt, chars) for dt in date_strs]\n",
    "    X = tf.ragged.constant(X_ids, ragged_rank=1)\n",
    "    return (X + 1).to_tensor() # using 0 as the padding token ID\n",
    "\n",
    "def create_dataset(n_dates):\n",
    "    x, y = random_dates(n_dates)\n",
    "    return prepare_date_strs(x, INPUT_CHARS), prepare_date_strs(y, OUTPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "id": "z90RMDxVtv8n"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "X_train, Y_train = create_dataset(10000)\n",
    "X_valid, Y_valid = create_dataset(2000)\n",
    "X_test, Y_test = create_dataset(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RSCJ4ZKAtv8n",
    "outputId": "713f8866-0007-47d5-e0cf-d82a5113a2e6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10,), dtype=int32, numpy=array([ 8,  1,  8,  6, 11,  1, 10, 11,  3,  1], dtype=int32)>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M5twt7zVtv8n"
   },
   "source": [
    "### First version: a very basic seq2seq model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6niu6Al_tv8n"
   },
   "source": [
    "Let's first try the simplest possible model: we feed in the input sequence, which first goes through the encoder (an embedding layer followed by a single LSTM layer), which outputs a vector, then it goes through a decoder (a single LSTM layer, followed by a dense output layer), which outputs a sequence of vectors, each representing the estimated probabilities for all possible output character.\n",
    "\n",
    "Since the decoder expects a sequence as input, we repeat the vector (which is output by the decoder) as many times as the longest possible output sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tERJfXk7tv8n",
    "outputId": "323ee34d-00cc-4f13-d98d-e61e357b8bd7"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "313/313 [==============================] - 6s 9ms/step - loss: 1.8255 - accuracy: 0.3456 - val_loss: 1.3841 - val_accuracy: 0.4841\n",
      "Epoch 2/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 1.2676 - accuracy: 0.5435 - val_loss: 1.1041 - val_accuracy: 0.6076\n",
      "Epoch 3/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 1.0743 - accuracy: 0.6210 - val_loss: 1.1233 - val_accuracy: 0.5800\n",
      "Epoch 4/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 1.1518 - accuracy: 0.5975 - val_loss: 0.9246 - val_accuracy: 0.6608\n",
      "Epoch 5/20\n",
      "313/313 [==============================] - 2s 8ms/step - loss: 0.7419 - accuracy: 0.7272 - val_loss: 0.6349 - val_accuracy: 0.7602\n",
      "Epoch 6/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.6495 - accuracy: 0.7567 - val_loss: 0.5411 - val_accuracy: 0.7875\n",
      "Epoch 7/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.4445 - accuracy: 0.8246 - val_loss: 0.3653 - val_accuracy: 0.8564\n",
      "Epoch 8/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.4815 - accuracy: 0.8322 - val_loss: 0.3781 - val_accuracy: 0.8661\n",
      "Epoch 9/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.2758 - accuracy: 0.9068 - val_loss: 0.2180 - val_accuracy: 0.9336\n",
      "Epoch 10/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.1578 - accuracy: 0.9588 - val_loss: 0.1138 - val_accuracy: 0.9747\n",
      "Epoch 11/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0805 - accuracy: 0.9851 - val_loss: 0.0638 - val_accuracy: 0.9887\n",
      "Epoch 12/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0441 - accuracy: 0.9948 - val_loss: 0.0357 - val_accuracy: 0.9965\n",
      "Epoch 13/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.1565 - accuracy: 0.9641 - val_loss: 0.0730 - val_accuracy: 0.9875\n",
      "Epoch 14/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0364 - accuracy: 0.9965 - val_loss: 0.0254 - val_accuracy: 0.9981\n",
      "Epoch 15/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0165 - accuracy: 0.9994 - val_loss: 0.0150 - val_accuracy: 0.9994\n",
      "Epoch 16/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0104 - accuracy: 0.9998 - val_loss: 0.0101 - val_accuracy: 0.9997\n",
      "Epoch 17/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0071 - accuracy: 0.9999 - val_loss: 0.0072 - val_accuracy: 0.9998\n",
      "Epoch 18/20\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0051 - accuracy: 1.0000 - val_loss: 0.0054 - val_accuracy: 0.9999\n",
      "Epoch 19/20\n",
      "313/313 [==============================] - 2s 8ms/step - loss: 0.0038 - accuracy: 1.0000 - val_loss: 0.0042 - val_accuracy: 0.9999\n",
      "Epoch 20/20\n",
      "313/313 [==============================] - 2s 8ms/step - loss: 0.0029 - accuracy: 1.0000 - val_loss: 0.0033 - val_accuracy: 0.9999\n"
     ]
    }
   ],
   "source": [
    "embedding_size = 32\n",
    "max_output_length = Y_train.shape[1]\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder = keras.models.Sequential([\n",
    "    keras.layers.Embedding(input_dim=len(INPUT_CHARS) + 1,\n",
    "                           output_dim=embedding_size,\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.LSTM(128)\n",
    "])\n",
    "\n",
    "decoder = keras.models.Sequential([\n",
    "    keras.layers.LSTM(128, return_sequences=True),\n",
    "    keras.layers.Dense(len(OUTPUT_CHARS) + 1, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    encoder,\n",
    "    keras.layers.RepeatVector(max_output_length),\n",
    "    decoder\n",
    "])\n",
    "\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_KWcyDdDtv8n"
   },
   "source": [
    "Looks great, we reach 100% validation accuracy! Let's use the model to make some predictions. We will need to be able to convert a sequence of character IDs to a readable string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "id": "bMZqmhuftv8n"
   },
   "outputs": [],
   "source": [
    "def ids_to_date_strs(ids, chars=OUTPUT_CHARS):\n",
    "    return [\"\".join([(\"?\" + chars)[index] for index in sequence])\n",
    "            for sequence in ids]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RpWqdrzNtv8n"
   },
   "source": [
    "Now we can use the model to convert some dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "id": "iTP5hLb3tv8o"
   },
   "outputs": [],
   "source": [
    "X_new = prepare_date_strs([\"September 17, 2009\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lI8O8rH2tv8o",
    "outputId": "ee400b10-485f-40d7-c313-0c1cfbd5d140"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2009-09-17\n",
      "1789-07-14\n"
     ]
    }
   ],
   "source": [
    "#ids = model.predict_classes(X_new)\n",
    "ids = np.argmax(model.predict(X_new), axis=-1)\n",
    "for date_str in ids_to_date_strs(ids):\n",
    "    print(date_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UzAyIB_Mtv8o"
   },
   "source": [
    "Perfect! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vIsVXff1tv8o"
   },
   "source": [
    "However, since the model was only trained on input strings of length 18 (which is the length of the longest date), it does not perform well if we try to use it to make predictions on shorter sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "id": "wbAsRuu8tv8o"
   },
   "outputs": [],
   "source": [
    "X_new = prepare_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aMAB5YD1tv8o",
    "outputId": "2b4da3a3-51ad-45f0-8e93-ee9bfceab141"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-02\n",
      "1789-02-14\n"
     ]
    }
   ],
   "source": [
    "#ids = model.predict_classes(X_new)\n",
    "ids = np.argmax(model.predict(X_new), axis=-1)\n",
    "for date_str in ids_to_date_strs(ids):\n",
    "    print(date_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eBunlIHZtv8o"
   },
   "source": [
    "Oops! We need to ensure that we always pass sequences of the same length as during training, using padding if necessary. Let's write a little helper function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "id": "vgF492Sjtv8o"
   },
   "outputs": [],
   "source": [
    "max_input_length = X_train.shape[1]\n",
    "\n",
    "def prepare_date_strs_padded(date_strs):\n",
    "    X = prepare_date_strs(date_strs)\n",
    "    if X.shape[1] < max_input_length:\n",
    "        X = tf.pad(X, [[0, 0], [0, max_input_length - X.shape[1]]])\n",
    "    return X\n",
    "\n",
    "def convert_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    #ids = model.predict_classes(X)\n",
    "    ids = np.argmax(model.predict(X), axis=-1)\n",
    "    return ids_to_date_strs(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ew6vaPfRtv8o",
    "outputId": "aa9b3599-48a4-423a-eedb-ba31bda48d04"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2020-05-02', '1789-07-14']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "convert_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sWqhRcN0tv8o"
   },
   "source": [
    "Cool! Granted, there are certainly much easier ways to write a date conversion tool (e.g., using regular expressions or even basic string manipulation), but you have to admit that using neural networks is way cooler. ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vFdkDNqEtv8p"
   },
   "source": [
    "However, real-life sequence-to-sequence problems will usually be harder, so for the sake of completeness, let's build a more powerful model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zk3HMCeJtv8p"
   },
   "source": [
    "### Second version: feeding the shifted targets to the decoder (teacher forcing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PK45DCaQtv8p"
   },
   "source": [
    "Instead of feeding the decoder a simple repetition of the encoder's output vector, we can feed it the target sequence, shifted by one time step to the right. This way, at each time step the decoder will know what the previous target character was. This should help is tackle more complex sequence-to-sequence problems.\n",
    "\n",
    "Since the first output character of each target sequence has no previous character, we will need a new token to represent the start-of-sequence (sos).\n",
    "\n",
    "During inference, we won't know the target, so what will we feed the decoder? We can just predict one character at a time, starting with an sos token, then feeding the decoder all the characters that were predicted so far (we will look at this in more details later in this notebook).\n",
    "\n",
    "But if the decoder's LSTM expects to get the previous target as input at each step, how shall we pass it it the vector output by the encoder? Well, one option is to ignore the output vector, and instead use the encoder's LSTM state as the initial state of the decoder's LSTM (which requires that encoder's LSTM must have the same number of units as the decoder's LSTM).\n",
    "\n",
    "Now let's create the decoder's inputs (for training, validation and testing). The sos token will be represented using the last possible output character's ID + 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "id": "bXwbEjfbtv8p"
   },
   "outputs": [],
   "source": [
    "sos_id = len(OUTPUT_CHARS) + 1\n",
    "\n",
    "def shifted_output_sequences(Y):\n",
    "    sos_tokens = tf.fill(dims=(len(Y), 1), value=sos_id)\n",
    "    return tf.concat([sos_tokens, Y[:, :-1]], axis=1)\n",
    "\n",
    "X_train_decoder = shifted_output_sequences(Y_train)\n",
    "X_valid_decoder = shifted_output_sequences(Y_valid)\n",
    "X_test_decoder = shifted_output_sequences(Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LUjoI0_Gtv8p"
   },
   "source": [
    "Let's take a look at the decoder's training inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2wMwFdnKtv8p",
    "outputId": "9c714a7b-ce1e-40e0-e35f-12342e8c3f8f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10000, 10), dtype=int32, numpy=\n",
       "array([[12,  8,  1, ..., 10, 11,  3],\n",
       "       [12,  9,  6, ...,  6, 11,  2],\n",
       "       [12,  8,  2, ...,  2, 11,  2],\n",
       "       ...,\n",
       "       [12, 10,  8, ...,  2, 11,  4],\n",
       "       [12,  2,  2, ...,  3, 11,  3],\n",
       "       [12,  8,  9, ...,  8, 11,  3]], dtype=int32)>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0I2fke1qtv8p"
   },
   "source": [
    "Now let's build the model. It's not a simple sequential model anymore, so let's use the functional API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BxM0r7A-tv8p",
    "outputId": "6418a95d-ec9c-4f4f-a463-5b3f91c380b1"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "313/313 [==============================] - 6s 9ms/step - loss: 1.6803 - accuracy: 0.3743 - val_loss: 1.4168 - val_accuracy: 0.4505\n",
      "Epoch 2/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 1.1884 - accuracy: 0.5587 - val_loss: 0.8931 - val_accuracy: 0.6714\n",
      "Epoch 3/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.6520 - accuracy: 0.7671 - val_loss: 0.3952 - val_accuracy: 0.8698\n",
      "Epoch 4/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.2255 - accuracy: 0.9431 - val_loss: 0.1285 - val_accuracy: 0.9754\n",
      "Epoch 5/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0803 - accuracy: 0.9895 - val_loss: 0.0490 - val_accuracy: 0.9964\n",
      "Epoch 6/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0714 - accuracy: 0.9882 - val_loss: 0.0286 - val_accuracy: 0.9991\n",
      "Epoch 7/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0188 - accuracy: 0.9998 - val_loss: 0.0150 - val_accuracy: 0.9998\n",
      "Epoch 8/10\n",
      "313/313 [==============================] - 2s 8ms/step - loss: 0.0110 - accuracy: 1.0000 - val_loss: 0.0100 - val_accuracy: 0.9999\n",
      "Epoch 9/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0417 - accuracy: 0.9935 - val_loss: 0.0104 - val_accuracy: 0.9998\n",
      "Epoch 10/10\n",
      "313/313 [==============================] - 2s 7ms/step - loss: 0.0068 - accuracy: 1.0000 - val_loss: 0.0058 - val_accuracy: 0.9999\n"
     ]
    }
   ],
   "source": [
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "lstm_units = 128\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder_input = keras.layers.Input(shape=[None], dtype=tf.int32)\n",
    "encoder_embedding = keras.layers.Embedding(\n",
    "    input_dim=len(INPUT_CHARS) + 1,\n",
    "    output_dim=encoder_embedding_size)(encoder_input)\n",
    "_, encoder_state_h, encoder_state_c = keras.layers.LSTM(\n",
    "    lstm_units, return_state=True)(encoder_embedding)\n",
    "encoder_state = [encoder_state_h, encoder_state_c]\n",
    "\n",
    "decoder_input = keras.layers.Input(shape=[None], dtype=tf.int32)\n",
    "decoder_embedding = keras.layers.Embedding(\n",
    "    input_dim=len(OUTPUT_CHARS) + 2,\n",
    "    output_dim=decoder_embedding_size)(decoder_input)\n",
    "decoder_lstm_output = keras.layers.LSTM(lstm_units, return_sequences=True)(\n",
    "    decoder_embedding, initial_state=encoder_state)\n",
    "decoder_output = keras.layers.Dense(len(OUTPUT_CHARS) + 1,\n",
    "                                    activation=\"softmax\")(decoder_lstm_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_input, decoder_input],\n",
    "                           outputs=[decoder_output])\n",
    "\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=10,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oOomKTV2tv8p"
   },
   "source": [
    "This model also reaches 100% validation accuracy, but it does so even faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-YB76znWtv8p"
   },
   "source": [
    "Let's once again use the model to make some predictions. This time we need to predict characters one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "id": "QmNs8WCRtv8p"
   },
   "outputs": [],
   "source": [
    "sos_id = len(OUTPUT_CHARS) + 1\n",
    "\n",
    "def predict_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = tf.fill(dims=(len(X), 1), value=sos_id)\n",
    "    for index in range(max_output_length):\n",
    "        pad_size = max_output_length - Y_pred.shape[1]\n",
    "        X_decoder = tf.pad(Y_pred, [[0, 0], [0, pad_size]])\n",
    "        Y_probas_next = model.predict([X, X_decoder])[:, index:index+1]\n",
    "        Y_pred_next = tf.argmax(Y_probas_next, axis=-1, output_type=tf.int32)\n",
    "        Y_pred = tf.concat([Y_pred, Y_pred_next], axis=1)\n",
    "    return ids_to_date_strs(Y_pred[:, 1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "M8f7h5tAtv8p",
    "outputId": "e6a68212-85ad-430d-cc4b-d6d2f5274f18"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JMJudMw_tv8q"
   },
   "source": [
    "Works fine! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HAu_rpr-tv8q"
   },
   "source": [
    "### Third version: using TF-Addons's seq2seq implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-1rOp9eXtv8q"
   },
   "source": [
    "Let's build exactly the same model, but using TF-Addon's seq2seq API. The implementation below is almost very similar to the TFA example higher in this notebook, except without the model input to specify the output sequence length, for simplicity (but you can easily add it back in if you need it for your projects, when the output sequences have very different lengths)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G5LxOJoKtv8q",
    "outputId": "b5e95063-90f3-4fc0-d985-a4ec57f5ccc5"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "313/313 [==============================] - 13s 30ms/step - loss: 1.6778 - accuracy: 0.3657 - val_loss: 1.4651 - val_accuracy: 0.4271\n",
      "Epoch 2/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 1.3794 - accuracy: 0.4618 - val_loss: 1.1807 - val_accuracy: 0.5543\n",
      "Epoch 3/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.9574 - accuracy: 0.6449 - val_loss: 0.6447 - val_accuracy: 0.7751\n",
      "Epoch 4/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.3974 - accuracy: 0.8755 - val_loss: 0.6728 - val_accuracy: 0.7954\n",
      "Epoch 5/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.1182 - accuracy: 0.9829 - val_loss: 0.0621 - val_accuracy: 0.9962\n",
      "Epoch 6/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0608 - accuracy: 0.9930 - val_loss: 0.0320 - val_accuracy: 0.9988\n",
      "Epoch 7/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0219 - accuracy: 0.9997 - val_loss: 0.0177 - val_accuracy: 0.9997\n",
      "Epoch 8/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0129 - accuracy: 0.9999 - val_loss: 0.0116 - val_accuracy: 0.9999\n",
      "Epoch 9/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0083 - accuracy: 1.0000 - val_loss: 0.0077 - val_accuracy: 0.9999\n",
      "Epoch 10/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0544 - accuracy: 0.9894 - val_loss: 0.0106 - val_accuracy: 0.9999\n",
      "Epoch 11/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0064 - accuracy: 1.0000 - val_loss: 0.0052 - val_accuracy: 1.0000\n",
      "Epoch 12/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0040 - accuracy: 1.0000 - val_loss: 0.0037 - val_accuracy: 1.0000\n",
      "Epoch 13/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0029 - accuracy: 1.0000 - val_loss: 0.0029 - val_accuracy: 1.0000\n",
      "Epoch 14/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0023 - accuracy: 1.0000 - val_loss: 0.0023 - val_accuracy: 1.0000\n",
      "Epoch 15/15\n",
      "313/313 [==============================] - 9s 28ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 0.0018 - val_accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "units = 128\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "encoder_embeddings = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 1, encoder_embedding_size)(encoder_inputs)\n",
    "\n",
    "decoder_embedding_layer = keras.layers.Embedding(\n",
    "    len(OUTPUT_CHARS) + 2, decoder_embedding_size)\n",
    "decoder_embeddings = decoder_embedding_layer(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(units, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.TrainingSampler()\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(units)\n",
    "output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell,\n",
    "                                                 sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings,\n",
    "    initial_state=encoder_state)\n",
    "Y_proba = keras.layers.Activation(\"softmax\")(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs],\n",
    "                           outputs=[Y_proba])\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=15,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vU6otu8itv8q"
   },
   "source": [
    "And once again, 100% validation accuracy! To use the model, we can just reuse the `predict_date_strs()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EFs8mzoRtv8q",
    "outputId": "7a150fd5-a38f-491e-fd34-33964ad787df",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XDblL58mtv8q"
   },
   "source": [
    "However, there's a much more efficient way to perform inference. Until now, during inference, we've run the model once for each new character. Instead, we can create a new decoder, based on the previously trained layers, but using a `GreedyEmbeddingSampler` instead of a `TrainingSampler`.\n",
    "\n",
    "At each time step, the `GreedyEmbeddingSampler` will compute the argmax of the decoder's outputs, and run the resulting token IDs through the decoder's embedding layer. Then it will feed the resulting embeddings to the decoder's LSTM cell at the next time step. This way, we only need to run the decoder once to get the full prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "id": "plvfzfXntv8q"
   },
   "outputs": [],
   "source": [
    "inference_sampler = tfa.seq2seq.sampler.GreedyEmbeddingSampler(\n",
    "    embedding_fn=decoder_embedding_layer)\n",
    "inference_decoder = tfa.seq2seq.basic_decoder.BasicDecoder(\n",
    "    decoder_cell, inference_sampler, output_layer=output_layer,\n",
    "    maximum_iterations=max_output_length)\n",
    "batch_size = tf.shape(encoder_inputs)[:1]\n",
    "start_tokens = tf.fill(dims=batch_size, value=sos_id)\n",
    "final_outputs, final_state, final_sequence_lengths = inference_decoder(\n",
    "    start_tokens,\n",
    "    initial_state=encoder_state,\n",
    "    start_tokens=start_tokens,\n",
    "    end_token=0)\n",
    "\n",
    "inference_model = keras.models.Model(inputs=[encoder_inputs],\n",
    "                                     outputs=[final_outputs.sample_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyhHT6Cxtv8q"
   },
   "source": [
    "A few notes:\n",
    "* The `GreedyEmbeddingSampler` needs the `start_tokens` (a vector containing the start-of-sequence ID for each decoder sequence), and the `end_token` (the decoder will stop decoding a sequence once the model outputs this token).\n",
    "* We must set `maximum_iterations` when creating the `BasicDecoder`, or else it may run into an infinite loop (if the model never outputs the end token for at least one of the sequences). This would force you would to restart the Jupyter kernel.\n",
    "* The decoder inputs are not needed anymore, since all the decoder inputs are generated dynamically based on the outputs from the previous time step.\n",
    "* The model's outputs are `final_outputs.sample_id` instead of the softmax of `final_outputs.rnn_outputs`. This allows us to directly get the argmax of the model's outputs. If you prefer to have access to the logits, you can replace `final_outputs.sample_id` with `final_outputs.rnn_outputs`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6bef1fH5tv8q"
   },
   "source": [
    "Now we can write a simple function that uses the model to perform the date format conversion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "id": "WLWoDQ8Otv8q"
   },
   "outputs": [],
   "source": [
    "def fast_predict_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = inference_model.predict(X)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-0I01yt3tv8r",
    "outputId": "bba1ca75-5f84-458b-b61c-d18ca5a453ea",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fast_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KoBa7tCYtv8r"
   },
   "source": [
    "Let's check that it really is faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0Nlchxkstv8r",
    "outputId": "2664a6ed-6f31-467e-9a18-bc8f7ade1e84"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 5: 383 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lbT1GWBgtv8r",
    "outputId": "15281fcb-57a3-41b5-a734-a6a8e21db6fc"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 5: 38.4 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit fast_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6f7gwa7Qtv8r"
   },
   "source": [
    "That's more than a 10x speedup! And it would be even more if we were handling longer sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_tzy8KJQtv8r"
   },
   "source": [
    "### Fourth version: using TF-Addons's seq2seq implementation with a scheduled sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JTC3AHDLtv8r"
   },
   "source": [
    "**Warning**: due to a TF bug, this version only works using TensorFlow 2.2 or above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Evt0PT6Otv8r"
   },
   "source": [
    "When we trained the previous model, at each time step _t_ we gave the model the target token for time step _t_ - 1. However, at inference time, the model did not get the previous target at each time step. Instead, it got the previous prediction. So there is a discrepancy between training and inference, which may lead to disappointing performance. To alleviate this, we can gradually replace the targets with the predictions, during training. For this, we just need to replace the `TrainingSampler` with a `ScheduledEmbeddingTrainingSampler`, and use a Keras callback to gradually increase the `sampling_probability` (i.e., the probability that the decoder will use the prediction from the previous time step rather than the target for the previous time step)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gK8OEbpFtv8r",
    "outputId": "f0a8fbdc-024b-42e0-f0d8-b9a0cca2c91d"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_1_grad/Identity_4:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_1_grad/Identity_3:0\", shape=(None, 32), dtype=float32), dense_shape=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_1_grad/Identity_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/gradients/grad_ys_0_indices:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/gradients/grad_ys_0_values:0\", shape=(None, 32), dtype=float32), dense_shape=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/gradients/grad_ys_0_shape:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/Identity_1:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/Identity:0\", shape=(None, 32), dtype=float32), dense_shape=Tensor(\"gradient_tape/model_5/basic_decoder_3/decoder/while/gradients/model_5/basic_decoder_3/decoder/while/cond_grad/Identity_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 17s 42ms/step - loss: 1.6779 - accuracy: 0.3658 - val_loss: 1.4628 - val_accuracy: 0.4332\n",
      "Epoch 2/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 1.4142 - accuracy: 0.4476 - val_loss: 1.3384 - val_accuracy: 0.4717\n",
      "Epoch 3/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 1.1022 - accuracy: 0.5820 - val_loss: 0.8807 - val_accuracy: 0.6834\n",
      "Epoch 4/20\n",
      "313/313 [==============================] - 13s 42ms/step - loss: 0.6877 - accuracy: 0.7457 - val_loss: 0.4740 - val_accuracy: 0.8270\n",
      "Epoch 5/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.3614 - accuracy: 0.8761 - val_loss: 0.2645 - val_accuracy: 0.9165\n",
      "Epoch 6/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.2485 - accuracy: 0.9265 - val_loss: 0.1711 - val_accuracy: 0.9531\n",
      "Epoch 7/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.1698 - accuracy: 0.9549 - val_loss: 0.1240 - val_accuracy: 0.9680\n",
      "Epoch 8/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0876 - accuracy: 0.9795 - val_loss: 0.0648 - val_accuracy: 0.9867\n",
      "Epoch 9/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0582 - accuracy: 0.9883 - val_loss: 0.0427 - val_accuracy: 0.9918\n",
      "Epoch 10/20\n",
      "313/313 [==============================] - 13s 42ms/step - loss: 0.0326 - accuracy: 0.9941 - val_loss: 0.0275 - val_accuracy: 0.9958\n",
      "Epoch 11/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0207 - accuracy: 0.9970 - val_loss: 0.0204 - val_accuracy: 0.9967\n",
      "Epoch 12/20\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.0140 - accuracy: 0.9982 - val_loss: 0.0138 - val_accuracy: 0.9981\n",
      "Epoch 13/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.1035 - accuracy: 0.9782 - val_loss: 0.0370 - val_accuracy: 0.9951\n",
      "Epoch 14/20\n",
      "313/313 [==============================] - 12s 40ms/step - loss: 0.0167 - accuracy: 0.9981 - val_loss: 0.0115 - val_accuracy: 0.9987\n",
      "Epoch 15/20\n",
      "313/313 [==============================] - 12s 40ms/step - loss: 0.0083 - accuracy: 0.9992 - val_loss: 0.0076 - val_accuracy: 0.9992\n",
      "Epoch 16/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0056 - accuracy: 0.9995 - val_loss: 0.0054 - val_accuracy: 0.9996\n",
      "Epoch 17/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0043 - accuracy: 0.9996 - val_loss: 0.0041 - val_accuracy: 0.9997\n",
      "Epoch 18/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0029 - accuracy: 0.9998 - val_loss: 0.0034 - val_accuracy: 0.9995\n",
      "Epoch 19/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0024 - accuracy: 0.9998 - val_loss: 0.0023 - val_accuracy: 0.9999\n",
      "Epoch 20/20\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.0033 - accuracy: 0.9995 - val_loss: 0.0026 - val_accuracy: 0.9998\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_epochs = 20\n",
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "units = 128\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "encoder_embeddings = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 1, encoder_embedding_size)(encoder_inputs)\n",
    "\n",
    "decoder_embedding_layer = keras.layers.Embedding(\n",
    "    len(OUTPUT_CHARS) + 2, decoder_embedding_size)\n",
    "decoder_embeddings = decoder_embedding_layer(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(units, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.ScheduledEmbeddingTrainingSampler(\n",
    "    sampling_probability=0.,\n",
    "    embedding_fn=decoder_embedding_layer)\n",
    "# we must set the sampling_probability after creating the sampler\n",
    "# (see https://github.com/tensorflow/addons/pull/1714)\n",
    "sampler.sampling_probability = tf.Variable(0.)\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(units)\n",
    "output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell,\n",
    "                                                 sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings,\n",
    "    initial_state=encoder_state)\n",
    "Y_proba = keras.layers.Activation(\"softmax\")(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs],\n",
    "                           outputs=[Y_proba])\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "def update_sampling_probability(epoch, logs):\n",
    "    proba = min(1.0, epoch / (n_epochs - 10))\n",
    "    sampler.sampling_probability.assign(proba)\n",
    "\n",
    "sampling_probability_cb = keras.callbacks.LambdaCallback(\n",
    "    on_epoch_begin=update_sampling_probability)\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=n_epochs,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid),\n",
    "                    callbacks=[sampling_probability_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HbTw5wmGtv8s"
   },
   "source": [
    "Not quite 100% validation accuracy, but close enough!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Of8iaZmctv8s"
   },
   "source": [
    "For inference, we could do the exact same thing as earlier, using a `GreedyEmbeddingSampler`. However, just for the sake of completeness, let's use a `SampleEmbeddingSampler` instead. It's almost the same thing, except that instead of using the argmax of the model's output to find the token ID, it treats the outputs as logits and uses them to sample a token ID randomly. This can be useful when you want to generate text. The `softmax_temperature` argument serves the \n",
    "same purpose as when we generated Shakespeare-like text (the higher this argument, the more random the generated text will be)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "id": "DHerrHF_tv8s"
   },
   "outputs": [],
   "source": [
    "softmax_temperature = tf.Variable(1.)\n",
    "\n",
    "inference_sampler = tfa.seq2seq.sampler.SampleEmbeddingSampler(\n",
    "    embedding_fn=decoder_embedding_layer,\n",
    "    softmax_temperature=softmax_temperature)\n",
    "inference_decoder = tfa.seq2seq.basic_decoder.BasicDecoder(\n",
    "    decoder_cell, inference_sampler, output_layer=output_layer,\n",
    "    maximum_iterations=max_output_length)\n",
    "batch_size = tf.shape(encoder_inputs)[:1]\n",
    "start_tokens = tf.fill(dims=batch_size, value=sos_id)\n",
    "final_outputs, final_state, final_sequence_lengths = inference_decoder(\n",
    "    start_tokens,\n",
    "    initial_state=encoder_state,\n",
    "    start_tokens=start_tokens,\n",
    "    end_token=0)\n",
    "\n",
    "inference_model = keras.models.Model(inputs=[encoder_inputs],\n",
    "                                     outputs=[final_outputs.sample_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "id": "f-eod_S7tv8s"
   },
   "outputs": [],
   "source": [
    "def creative_predict_date_strs(date_strs, temperature=1.0):\n",
    "    softmax_temperature.assign(temperature)\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = inference_model.predict(X)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KLpi4Peztv8s",
    "outputId": "3cc56601-c381-4e48-ade7-efe7cc7019b5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-00']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "creative_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pkFmwSV3tv8s"
   },
   "source": [
    "Dates look good at room temperature. Now let's heat things up a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lVk5rV6Wtv8s",
    "outputId": "e9c31ffd-e8ac-4457-f5ec-f0b45deb610e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['7479307-19', '200040?400']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "creative_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"],\n",
    "                           temperature=5.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xJWXvaOBtv8s"
   },
   "source": [
    "Oops, the dates are overcooked, now. Let's call them \"creative\" dates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6ZkGjCy-tv8s"
   },
   "source": [
    "### Fifth version: using TFA seq2seq, the Keras subclassing API and attention mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ycp3xsdWtv8s"
   },
   "source": [
    "The sequences in this problem are pretty short, but if we wanted to tackle longer sequences, we would probably have to use attention mechanisms. While it's possible to code our own implementation, it's simpler and more efficient to use TF-Addons's implementation instead. Let's do that now, this time using Keras' subclassing API.\n",
    "\n",
    "**Warning**: due to a TensorFlow bug (see [this issue](https://github.com/tensorflow/addons/issues/1153) for details), the `get_initial_state()` method fails in eager mode, so for now we have to use the subclassing API, as Keras automatically calls `tf.function()` on the `call()` method (so it runs in graph mode)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2NbiNcPxtv8s"
   },
   "source": [
    "In this implementation, we've reverted back to using the `TrainingSampler`, for simplicity (but you can easily tweak it to use a `ScheduledEmbeddingTrainingSampler` instead). We also use a `GreedyEmbeddingSampler` during inference, so this class is pretty easy to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "id": "rGxaGZQStv8t"
   },
   "outputs": [],
   "source": [
    "class DateTranslation(keras.models.Model):\n",
    "    def __init__(self, units=128, encoder_embedding_size=32,\n",
    "                 decoder_embedding_size=32, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.encoder_embedding = keras.layers.Embedding(\n",
    "            input_dim=len(INPUT_CHARS) + 1,\n",
    "            output_dim=encoder_embedding_size)\n",
    "        self.encoder = keras.layers.LSTM(units,\n",
    "                                         return_sequences=True,\n",
    "                                         return_state=True)\n",
    "        self.decoder_embedding = keras.layers.Embedding(\n",
    "            input_dim=len(OUTPUT_CHARS) + 2,\n",
    "            output_dim=decoder_embedding_size)\n",
    "        self.attention = tfa.seq2seq.LuongAttention(units)\n",
    "        decoder_inner_cell = keras.layers.LSTMCell(units)\n",
    "        self.decoder_cell = tfa.seq2seq.AttentionWrapper(\n",
    "            cell=decoder_inner_cell,\n",
    "            attention_mechanism=self.attention)\n",
    "        output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "        self.decoder = tfa.seq2seq.BasicDecoder(\n",
    "            cell=self.decoder_cell,\n",
    "            sampler=tfa.seq2seq.sampler.TrainingSampler(),\n",
    "            output_layer=output_layer)\n",
    "        self.inference_decoder = tfa.seq2seq.BasicDecoder(\n",
    "            cell=self.decoder_cell,\n",
    "            sampler=tfa.seq2seq.sampler.GreedyEmbeddingSampler(\n",
    "                embedding_fn=self.decoder_embedding),\n",
    "            output_layer=output_layer,\n",
    "            maximum_iterations=max_output_length)\n",
    "\n",
    "    def call(self, inputs, training=None):\n",
    "        encoder_input, decoder_input = inputs\n",
    "        encoder_embeddings = self.encoder_embedding(encoder_input)\n",
    "        encoder_outputs, encoder_state_h, encoder_state_c = self.encoder(\n",
    "            encoder_embeddings,\n",
    "            training=training)\n",
    "        encoder_state = [encoder_state_h, encoder_state_c]\n",
    "\n",
    "        self.attention(encoder_outputs,\n",
    "                       setup_memory=True)\n",
    "        \n",
    "        decoder_embeddings = self.decoder_embedding(decoder_input)\n",
    "\n",
    "        decoder_initial_state = self.decoder_cell.get_initial_state(\n",
    "            decoder_embeddings)\n",
    "        decoder_initial_state = decoder_initial_state.clone(\n",
    "            cell_state=encoder_state)\n",
    "        \n",
    "        if training:\n",
    "            decoder_outputs, _, _ = self.decoder(\n",
    "                decoder_embeddings,\n",
    "                initial_state=decoder_initial_state,\n",
    "                training=training)\n",
    "        else:\n",
    "            start_tokens = tf.zeros_like(encoder_input[:, 0]) + sos_id\n",
    "            decoder_outputs, _, _ = self.inference_decoder(\n",
    "                decoder_embeddings,\n",
    "                initial_state=decoder_initial_state,\n",
    "                start_tokens=start_tokens,\n",
    "                end_token=0)\n",
    "\n",
    "        return tf.nn.softmax(decoder_outputs.rnn_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hJenC7DRtv8t",
    "outputId": "6ddb9904-e4c2-4ed4-826a-92534c2355b7"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "313/313 [==============================] - 19s 42ms/step - loss: 2.1368 - accuracy: 0.2335 - val_loss: 2.0080 - val_accuracy: 0.2648\n",
      "Epoch 2/25\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 1.8487 - accuracy: 0.3307 - val_loss: 1.5100 - val_accuracy: 0.4396\n",
      "Epoch 3/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 2.1037 - accuracy: 0.2437 - val_loss: 1.6046 - val_accuracy: 0.3954\n",
      "Epoch 4/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 1.7171 - accuracy: 0.3651 - val_loss: 2.5416 - val_accuracy: 0.2658\n",
      "Epoch 5/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 1.4480 - accuracy: 0.4810 - val_loss: 1.3507 - val_accuracy: 0.5063\n",
      "Epoch 6/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 1.3200 - accuracy: 0.5156 - val_loss: 1.2034 - val_accuracy: 0.5402\n",
      "Epoch 7/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 1.1148 - accuracy: 0.5612 - val_loss: 1.1936 - val_accuracy: 0.5586\n",
      "Epoch 8/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.9277 - accuracy: 0.5986 - val_loss: 0.9126 - val_accuracy: 0.6054\n",
      "Epoch 9/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.8577 - accuracy: 0.6169 - val_loss: 0.8899 - val_accuracy: 0.6176\n",
      "Epoch 10/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.7644 - accuracy: 0.6612 - val_loss: 0.7695 - val_accuracy: 0.6752\n",
      "Epoch 11/25\n",
      "313/313 [==============================] - 12s 40ms/step - loss: 0.7101 - accuracy: 0.6922 - val_loss: 0.7124 - val_accuracy: 0.6978\n",
      "Epoch 12/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.6503 - accuracy: 0.7088 - val_loss: 0.6945 - val_accuracy: 0.7086\n",
      "Epoch 13/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.6199 - accuracy: 0.7190 - val_loss: 0.6227 - val_accuracy: 0.7230\n",
      "Epoch 14/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.6372 - accuracy: 0.7171 - val_loss: 0.6330 - val_accuracy: 0.7210\n",
      "Epoch 15/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.5939 - accuracy: 0.7314 - val_loss: 0.6056 - val_accuracy: 0.7382\n",
      "Epoch 16/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.6529 - accuracy: 0.7228 - val_loss: 0.5973 - val_accuracy: 0.7352\n",
      "Epoch 17/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.5760 - accuracy: 0.7416 - val_loss: 0.5807 - val_accuracy: 0.7454\n",
      "Epoch 18/25\n",
      "313/313 [==============================] - 12s 40ms/step - loss: 0.5532 - accuracy: 0.7517 - val_loss: 0.5717 - val_accuracy: 0.7523\n",
      "Epoch 19/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.5168 - accuracy: 0.7727 - val_loss: 0.8258 - val_accuracy: 0.7143\n",
      "Epoch 20/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.3717 - accuracy: 0.8395 - val_loss: 0.7097 - val_accuracy: 0.7858\n",
      "Epoch 21/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.2551 - accuracy: 0.9105 - val_loss: 0.3303 - val_accuracy: 0.9349\n",
      "Epoch 22/25\n",
      "313/313 [==============================] - 13s 41ms/step - loss: 0.1716 - accuracy: 0.9569 - val_loss: 0.2559 - val_accuracy: 0.9586\n",
      "Epoch 23/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.0999 - accuracy: 0.9795 - val_loss: 0.1478 - val_accuracy: 0.9835\n",
      "Epoch 24/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.0724 - accuracy: 0.9858 - val_loss: 0.1377 - val_accuracy: 0.9775\n",
      "Epoch 25/25\n",
      "313/313 [==============================] - 13s 40ms/step - loss: 0.0576 - accuracy: 0.9853 - val_loss: 0.0527 - val_accuracy: 0.9877\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = DateTranslation()\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=25,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lGMYbeSCtv8t"
   },
   "source": [
    "Not quite 100% validation accuracy, but close. It took a bit longer to converge this time, but there were also more parameters and more computations per iteration. And we did not use a scheduled sampler.\n",
    "\n",
    "To use the model, we can write yet another little function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "id": "xrzcaX6mtv8t"
   },
   "outputs": [],
   "source": [
    "def fast_predict_date_strs_v2(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    X_decoder = tf.zeros(shape=(len(X), max_output_length), dtype=tf.int32)\n",
    "    Y_probas = model.predict([X, X_decoder])\n",
    "    Y_pred = tf.argmax(Y_probas, axis=-1)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pOygdiuDtv8t",
    "outputId": "03507f10-a73a-4991-821f-1120c07e94ea"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-06-14', '181805-015']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fast_predict_date_strs_v2([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fSlnKiXptv8t"
   },
   "source": [
    "There are still a few interesting features from TF-Addons that you may want to look at:\n",
    "* Using a `BeamSearchDecoder` rather than a `BasicDecoder` for inference. Instead of outputing the character with the highest probability, this decoder keeps track of the several candidates, and keeps only the most likely sequences of candidates (see chapter 16 in the book for more details).\n",
    "* Setting masks or specifying `sequence_length` if the input or target sequences may have very different lengths.\n",
    "* Using a `ScheduledOutputTrainingSampler`, which gives you more flexibility than the `ScheduledEmbeddingTrainingSampler` to decide how to feed the output at time _t_ to the cell at time _t_+1. By default it feeds the outputs directly to cell, without computing the argmax ID and passing it through an embedding layer. Alternatively, you specify a `next_inputs_fn` function that will be used to convert the cell outputs to inputs at the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DyIUU7iwtv8u"
   },
   "source": [
    "## 10.\n",
    "_Exercise: Go through TensorFlow's [Neural Machine Translation with Attention tutorial](https://homl.info/nmttuto)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lLketeOjtv8u"
   },
   "source": [
    "Simply open the Colab and follow its instructions. Alternatively, if you want a simpler example of using TF-Addons's seq2seq implementation for Neural Machine Translation (NMT), look at the solution to the previous question. The last model implementation will give you a simpler example of using TF-Addons to build an NMT model using attention mechanisms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G5TgfNSAtv8u"
   },
   "source": [
    "## 11.\n",
    "_Exercise: Use one of the recent language models (e.g., GPT) to generate more convincing Shakespearean text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k6K-yFNhtv8u"
   },
   "source": [
    "The simplest way to use recent language models is to use the excellent [transformers library](https://huggingface.co/transformers/), open sourced by Hugging Face. It provides many modern neural net architectures (including BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet and more) for Natural Language Processing (NLP), including many pretrained models. It relies on either TensorFlow or PyTorch. Best of all: it's amazingly simple to use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G2XnpwzVtv8u"
   },
   "source": [
    "First, let's load a pretrained model. In this example, we will use OpenAI's GPT model, with an additional Language Model on top (just a linear layer with weights tied to the input embeddings). Let's import it and load the pretrained weights (this will download about 445MB of data to `~/.cache/torch/transformers`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 141,
     "referenced_widgets": [
      "9cfa61c805664e2bbb15cd8e86136c20",
      "30fdc919e05d48d78442166df9d87475"
     ]
    },
    "id": "T0M1gzwhtv8u",
    "outputId": "a0487be1-6bd0-436b-fd95-f62cb068abd3"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9cfa61c805664e2bbb15cd8e86136c20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/656 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30fdc919e05d48d78442166df9d87475",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/466M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint layers were used when initializing TFOpenAIGPTLMHeadModel.\n",
      "\n",
      "All the layers of TFOpenAIGPTLMHeadModel were initialized from the model checkpoint at openai-gpt.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFOpenAIGPTLMHeadModel for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "from transformers import TFOpenAIGPTLMHeadModel\n",
    "\n",
    "model = TFOpenAIGPTLMHeadModel.from_pretrained(\"openai-gpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uyt828D4tv8u"
   },
   "source": [
    "Next we will need a specialized tokenizer for this model. This one will try to use the [spaCy](https://spacy.io/) and [ftfy](https://pypi.org/project/ftfy/) libraries if they are installed, or else it will fall back to BERT's `BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most use cases)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 86,
     "referenced_widgets": [
      "c4ab9ac17b8b4524b774fb53ca3b0a6a",
      "7e6e90bef3bb457b9583c8a0d9603115",
      "276d7047b5ce4b3cb2c509e6cec1e7bc"
     ]
    },
    "id": "0QBeH-N7tv8u",
    "outputId": "3977e2c1-162c-4598-85aa-00a1b842a2e6"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4ab9ac17b8b4524b774fb53ca3b0a6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/816k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e6e90bef3bb457b9583c8a0d9603115",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/458k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "276d7047b5ce4b3cb2c509e6cec1e7bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.27M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "metadata": {
      "tags": null
     },
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.\n"
     ]
    }
   ],
   "source": [
    "from transformers import OpenAIGPTTokenizer\n",
    "\n",
    "tokenizer = OpenAIGPTTokenizer.from_pretrained(\"openai-gpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WbVADEHdtv8u"
   },
   "source": [
    "Now let's use the tokenizer to tokenize and encode the prompt text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DF2R5ZRUtv8u",
    "outputId": "1a840ed5-38d6-42ef-e438-23d28d1f437e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 10), dtype=int32, numpy=\n",
       "array([[  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187]], dtype=int32)>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt_text = \"This royal throne of kings, this sceptred isle\"\n",
    "encoded_prompt = tokenizer.encode(prompt_text,\n",
    "                                  add_special_tokens=False,\n",
    "                                  return_tensors=\"tf\")\n",
    "encoded_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_pLNPN5-tv8u"
   },
   "source": [
    "Easy! Next, let's use the model to generate text after the prompt. We will generate 5 different sentences, each starting with the prompt text, followed by 40 additional tokens. For an explanation of what all the hyperparameters do, make sure to check out this great [blog post](https://huggingface.co/blog/how-to-generate) by Patrick von Platen (from Hugging Face). You can play around with the hyperparameters to try to obtain better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ElEOscvytv8u",
    "outputId": "2d01a00d-dda0-452f-a9e0-271ac010f1e8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5, 50), dtype=int32, numpy=\n",
       "array([[  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   240,   509,   481,  9313,  6640,   498,  1389, 11031,\n",
       "          239,   481,  2204,   544,   525,   481,  4906,  3659,   498,\n",
       "          481,   653,   911,   498,     8, 38648,   641,  1236,   481,\n",
       "         5018,   498,   481, 21368,   488,   481,  6404,   948, 35174,\n",
       "          715,  1076,   763,   641,  2520],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   980,  1981,   557,   481,  2827,   498,   481,  1028,\n",
       "          498,  9606,   239,   244, 40477,   244,   862,  1256,   240,\n",
       "          547,  2185,   239,   244,  7395, 21800,   513,  2185,   239,\n",
       "          244,   599,   636,   512,   649,   485,   788,   257,   244,\n",
       "        40477,   481,  2228,   535,   741],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   239,   481,  3549,   498,   589,   547, 16375,   240,\n",
       "         4258,  1076,   498,   547,  5080,   260,  2228,   240,  1485,\n",
       "         6945,  2034,   547, 13509,   239,   256, 40477,   256,  1504,\n",
       "          240,   256,   603,   481,   618,   240,   256,   249,  1259,\n",
       "        20024,   547, 11244,   240,   488],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   487,   509,  3105,   500,   240,   616,   908,   487,\n",
       "          558,  2160,   781,   575,   240,   507,   544,  6322,   500,\n",
       "          481,  1279,   498,  2857, 24711,   504,   481,  7361,  2075,\n",
       "          498,   481, 16187,   240,   524,  1584,   759,   580,  1132,\n",
       "          822,   481,  4644,   498,  2857],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   240, 40477,   488,   481,  1922,  4198,   535,  8908,\n",
       "          240,   834,   240,   481,  2185,   498,  1092,   239, 40477,\n",
       "          244,   500,   481,  1385,   498,   481,  2878, 13658,   240,\n",
       "         2122,  2821,  1085,   589,  8162,   240,   244, 40477,   655,\n",
       "          544,   597,  1203,  3126,   500]], dtype=int32)>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_sequences = 5\n",
    "length = 40\n",
    "\n",
    "generated_sequences = model.generate(\n",
    "    input_ids=encoded_prompt,\n",
    "    do_sample=True,\n",
    "    max_length=length + len(encoded_prompt[0]),\n",
    "    temperature=1.0,\n",
    "    top_k=0,\n",
    "    top_p=0.9,\n",
    "    repetition_penalty=1.0,\n",
    "    num_return_sequences=num_sequences,\n",
    ")\n",
    "\n",
    "generated_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pgOScr7ktv8v"
   },
   "source": [
    "Now let's decode the generated sequences and print them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cMTt9J29tv8v",
    "outputId": "ec0e9284-e3f4-4a4c-c81e-ba2cc134a89e"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this royal throne of kings, this sceptred isle, was the largest collection of such affairs. the problem is that the descendents of the earls of astoria were under the rule of the sages and the throne took precedence over those who were forced\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle has passed as the beginning of the age of kings. \" \n",
      " \" well done, my lord. \" velvet complimented her lord. \" what would you like to see? \" \n",
      " the lady's eyes\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle. the bones of all my comrades, including those of my ex - lady, lie crushed upon my battlefield.'\n",
      "'ah,'said the king,'i must consult my contacts, and\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle he was born in, this door he had placed before him, it is located in the heart of galdir on the outer edge of the isle, his line can be found through the houses of gal\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle, \n",
      " and the pendragon's portal, too, the lord of light. \n",
      " \" in the course of the seven pillars, ye shall find all treasure, \" \n",
      " there is now three bodies in\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for sequence in generated_sequences:\n",
    "    text = tokenizer.decode(sequence, clean_up_tokenization_spaces=True)\n",
    "    print(text)\n",
    "    print(\"-\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ez2vcMLYtv8v"
   },
   "source": [
    "You can try more recent (and larger) models, such as GPT-2, CTRL, Transformer-XL or XLNet, which are all available as pretrained models in the transformers library, including variants with Language Models on top. The preprocessing steps vary slightly between models, so make sure to check out this [generation example](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) from the transformers documentation (this example uses PyTorch, but it will work with very little tweaks, such as adding `TF` at the beginning of the model class name, removing the `.to()` method calls, and using `return_tensors=\"tf\"` instead of `\"pt\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TbduRAdStv8v"
   },
   "source": [
    "Hope you enjoyed this chapter! :)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "16_nlp_with_rnns_and_attention.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}