5.1 Recurrent Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Recurrent Neural Networks I\n",
    "\n",
    "Classical neural networks, including convolutional ones, suffer from two severe limitations:\n",
    "\n",
    "+ They only accept a fixed-sized vector as input and produce a fixed-sized vector as output.\n",
    "+ They do not consider the sequential nature of some data (language, video frames, time series, etc.) \n",
    "\n",
    "Recurrent neural networks overcome these limitations by allowing to operate over sequences of vectors (in the input, in the output, or both)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vanilla Recurrent Neural Network\n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/vanilla.png?raw=true\" alt=\"\" style=\"width: 400px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Unrolling in time of a RNN\n",
    "\n",
    "By unrolling we mean that we write out the network for the complete sequence.\n",
    "\n",
    "$$ s_t = \\mbox{tanh }(Ux_t + W s_{t-1}) $$\n",
    "$$ y_t = \\mbox{softmax }(V s_t) $$\n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/unrolling.png?raw=true\" alt=\"\" style=\"width: 600px;\"/>\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/TanhReal.gif?raw=true\" alt=\"\" style=\"width: 200px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Vanilla Recurrent Neural Network (minibatch version)\n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/minibatch.png?raw=true\" alt=\"\" style=\"width: 400px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "+ We can think of the **hidden state** $s_t$ as a memory of the network that captures information about the previous steps.\n",
    "+ The RNN **shares the parameters** $U,V,W$ across all time steps.\n",
    "+ It is not necessary to have outputs $y_t$ at each time step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/kar.png?raw=true\" alt=\"\" style=\"width: 600px;\"/>\n",
    "\n",
    "<center>\n",
    "Source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "RNN have shown success in:\n",
    "\n",
    "+ Language modeling and generation.\n",
    "+ Machine Translation.\n",
    "+ Speech Recognition.\n",
    "+ Image Description.\n",
    "+ Question Answering.\n",
    "+ Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## RNN Computation\n",
    "\n",
    "```python\n",
    "class RNN:\n",
    "    #...\n",
    "    def step(self,x):\n",
    "        self.h = np.tanh(np.dot(self.W_hh, self.h) + \n",
    "                 np.dot(self.W_xh, self.x))\n",
    "        y = np.dot(self.W_hy, self.h)\n",
    "        return y\n",
    "    #...\n",
    "```\n",
    "\n",
    "We can go deep by stacking RNNs:\n",
    "```python\n",
    "y1 = rnn1.step(x)\n",
    "y2 = rnn2.step(y1)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Training a RNN is similar to training a traditional NN, but some modifications. The main reason is that parameters are shared by all time steps: in order to compute the gradient at $t=4$, we need to propagate 3 steps and sum up the gradients. This is called **Backpropagation through time (BPTT)**.\n",
    "\n",
    "The inputs of a recurrent network are always vectors, but we can process sequences of symbols/words by representing these symbols by numerical vectors.\n",
    "\n",
    "Let's suppose we are classifying a series of words: $x_1, ..., x_{t-1}, x_t, x_{t+1}, ... x_{T}$ are the word vectors corresponding to a corpus with T symbols. Then, the relationship to compute the hidden layer output features at each time-step $t$ is $h_t = \\sigma(W^{(hh)} h_{t-1} + W^{(hx)} x_{t})$, where:\n",
    "\n",
    "   + $x_{t} \\in \\mathbb{R}^{d}$ is input word vector at time $t$. \n",
    "   + $W^{hx} \\in \\mathbb{R}^{D_h \\times d}$ is the weights matrix used to condition the input word vector, $x_t$.\n",
    "   + $W^{hh} \\in \\mathbb{R}^{D_h \\times D_h}$ is the weights matrix used to condition the output of the previous time-step, $h_{t-1}$.\n",
    "   + $h_{t-1}  \\in \\mathbb{R}^{D_h}$ is the output of the non-linear function at the previous time-step, $t-1$. \n",
    "   + $h_0 \\in \\mathbb{R}^{D_h}$ is an initialization vector for the hidden layer at time-step $t = 0$.\n",
    "   + $\\sigma ()$ is the non-linearity function (normally, ``tanh``).\n",
    "   + $\\hat{y}_t = softmax (W^{(S)}h_t)$ is the output probability distribution over the vocabulary at each time-step $t$.  Essentially, $\\hat{y}_t$ is the next predicted word given the document context score so far (i.e. $h_{t-1}$) and the last observed word vector $x^{(t)}$. Here, $W^{(S)} \\in \\mathbb{R}^{|V| \\times D_h}$ and $\\hat{y} \\in \\mathbb{R}^{|V|}$ where $|V|$ is the vocabulary.\n",
    "\n",
    "The loss function used in RNNs is often the cross entropy error:\n",
    "\n",
    "$$\n",
    "\tL^{(t)}(W) = - \\sum_{j=1}^{|V|} y_{t,j} \\times log (\\hat{y}_{t,j})\n",
    "$$\n",
    "\n",
    "The cross entropy error over a corpus of size $T$ is:\n",
    "\n",
    "$$\n",
    "\tL = \\dfrac{1}{T} \\sum_{t=1}^{T} L^{(t)}(W) = - \\dfrac{1}{T} \\sum_{t=1}^{T} \\sum_{j=1}^{|V|} y_{t,j} \\times log (\\hat{y}_{t,j})\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of classifying a series of symbols/words, the *perplexity* measure can be used to assess the goodness of our model. It is basically 2 to the power of the negative log probability of the cross entropy error function:\n",
    "\n",
    "$$\n",
    "\tPerplexity = 2^{L}\n",
    "$$\n",
    "\n",
    "Perplexity is a measure of confusion where lower values imply more confidence in predicting the next word in the sequence (compared to the ground truth outcome)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## RNN Training\n",
    "\n",
    "Recurrent neural networks propagate weight matrices from one time-step to the next. Recall the goal of a RNN implementation is to enable propagating context information through faraway time-steps. When these propagation results in a long series of matrix multiplications, weights can vanish or explode. \n",
    "\n",
    "Once the gradient value grows extremely large, it causes an overflow (i.e. ``NaN``) which is easily detectable at runtime; this issue is called the *Gradient Explosion Problem*. \n",
    "\n",
    "When the gradient value goes to zero, however, it can go undetected while drastically reducing the learning quality of the model for far-away words in the corpus; this issue is called the *Vanishing Gradient Problem*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Clipping\n",
    "\n",
    "To solve the problem of exploding gradients, Thomas Mikolov first introduced a simple heuristic solution that *clips* gradients to a small number whenever they explode. That is, whenever they reach a certain threshold, they are set back to a small number. \n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/exploding.png?raw=true\" alt=\"\" style=\"width: 400px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Better initialization\n",
    "\n",
    "To solve the problem of vanishing gradients, instead of initializing $W^{hh}$ randomly, starting off from **random orthogonal matrices** works better,  i.e., a square matrix $W$ for which $W^T W=I$.\n",
    "\n",
    "There are two properties of orthogonal matrices that are useful for training deep neural networks:\n",
    "+ they are norm-preserving, i.e., $ ||Wx||^2=||x||^2$, and\n",
    "+ their columns (and rows) are all orthonormal to one another.\n",
    "\n",
    "At least at the start of training, the first of these should help to keep the norm of the input constant throughout the network, which can help with the problem of exploding/vanishing gradients. \n",
    "\n",
    "Similarly, an intuitive understanding of the second is that having orthonormal weight vectors encourages the weights to learn different input features.\n",
    "\n",
    "You can obtain a random $n \\times n$ orthogonal matrix $W$, (uniformly distributed) by performing a QR factorization of an $n \\times n$ matrix with elements i.i.d. Gaussian random variables of mean $0$ and variance $1$. Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.00471241  0.20250343  0.67828716]\n",
      " [-0.40961315  0.36636658  0.21412201]\n",
      " [ 0.18052579 -1.68407099  1.62059982]]\n",
      "\n",
      "\n",
      "[[ 1.00000000e+00  4.40432839e-16 -1.46140146e-16]\n",
      " [ 4.40432839e-16  1.00000000e+00  1.76849036e-16]\n",
      " [-1.46140146e-16  1.76849036e-16  1.00000000e+00]]\n",
      "[[-0.01052689  0.1512298  -0.98844258]\n",
      " [ 0.9150204  -0.39719676 -0.07051528]\n",
      " [-0.4032702  -0.90518743 -0.1341971 ]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from scipy.linalg import qr\n",
    "\n",
    "n = 3\n",
    "H = np.random.randn(n, n)\n",
    "print(H)\n",
    "print('\\n')\n",
    "\n",
    "Q, R = qr(H)\n",
    "\n",
    "print (Q.dot(Q.T))\n",
    "print(Q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Steeper Gates\n",
    "\n",
    "We can make the \"gates steeper\" so they change more repidly from 0 to 1 and the model is learnt quicker.\n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/steeper.png?raw=true\" alt=\"\" style=\"width: 600px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Gated Units\n",
    "\n",
    "The most important types of gated RNNs are:\n",
    "\n",
    "+ Long Short Term Memories (LSTM). It was introduced by S.Hochreiter and J.Schmidhuber in 1997 and is widely used. LSTM is very good in the long run due to its high complexity.\n",
    "+ Gated Recurrent Units (GRU). It was recently introduced by K.Cho. It is simpler than LSTM, fasters and optimizes quicker. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LSTM\n",
    "\n",
    "The key idea of LSTMs is the cell state $C$, the horizontal line running through the top of the diagram.\n",
    "\n",
    "The cell state is kind of like a conveyor belt. It runs straight down the entire chain, with only some minor linear interactions. It’s very easy for information to just flow along it unchanged.\n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/lstm.png?raw=true\" alt=\"Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\" style=\"width: 600px;\"/> \n",
    "\n",
    "LSTM has the ability to remove or add information to the cell state, carefully regulated by structures called gates.\n",
    "\n",
    "Gates are a way to optionally let information through. They are composed out of a *sigmoid* neural net layer and a pointwise multiplication operation.\n",
    "\n",
    "Let us see how a LSTM uses $h_{t-1}, C_{t-1}$ and $x_{t}$ to generate the next hidden states $C_t, h_{t}$:\n",
    "\n",
    "$$ f_t = \\sigma(W_f \\cdot [h_{t-1}, x_t]) \\mbox{ (Forget gate)} $$\n",
    "$$ i_t = \\sigma(W_i \\cdot [h_{t-1}, x_t]) \\mbox{ (Input gate)} $$\n",
    "$$ \\tilde C_t = \\operatorname{tanh}(W_C \\cdot [h_{t-1}, x_t]) $$\n",
    "$$ C_t = f_t * C_{t-1} + i_t * \\tilde C_t \\mbox{ (Update gate)} $$\n",
    "$$ o_t = \\sigma(W_o \\cdot [h_{t-1}, x_t]) $$\n",
    "$$ h_t = o_t * \\operatorname{tanh}(C_t) \\mbox{ (Output gate)} $$\n",
    "\n",
    "There are other variants of LSTM (f.e. LSTM with peephole connections of Gers & Schmidhuber (2000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### GRU\n",
    "\n",
    "The transition from hidden state $h_{t-1}$ to $h_{t}$ in vanilla RNN is defined by using an affine transformation and a point-wise nonlinearity. \n",
    "\n",
    "What motivates the use of gated units? Although RNNs can theoretically capture long-term dependencies, they are very hard to actually train to do this. Gated recurrent units are designed in a manner to have more persistent memory thereby making it easier for RNNs to capture long-term dependencies. \n",
    "\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/gru.png?raw=true\"alt=\"\" style=\"width: 300px;\"/>\n",
    "<center>\n",
    "Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "</center>\n",
    "\n",
    "Let us see how a GRU uses $h_{t-1}$ and $x_{t}$ to generate the next hidden state $h_{t}$.\n",
    "\n",
    "$$ z_{t} = \\sigma(W_z \\cdot [x_{t}, h_{t-1}]) \\mbox{ (Update gate)}$$\n",
    "$$ r_{t} = \\sigma(W_r \\cdot [x_{t}, h_{t-1}]) \\mbox{ (Reset gate)}$$\n",
    "$$ \\tilde{h}_{t} = \\operatorname{tanh}(r_{t} \\cdot  [x_{t}, r_t \\circ h_{t-1}]   ) \\mbox{ (New memory)}$$\n",
    "$$ h_{t} = (1 - z_{t}) \\circ \\tilde{h}_{t-1} + z_{t} \\circ h_{t} \\mbox{ (Hidden state)}$$\n",
    "\n",
    "It combines the forget and input gates into a single “update gate.” It also merges the cell state and hidden state, and makes some other changes. The resulting model is simpler than standard LSTM models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `keras`\n",
    "\n",
    "> Keras is a high-level neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation.\n",
    "\n",
    "The core data structure of Keras is a model, a way to organize layers. The main type of model is the ``Sequential model``, a linear stack of layers. \n",
    "\n",
    "```Python\n",
    "from keras.models import Sequential\n",
    "model = Sequential()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stacking layers is as easy as ``.add()``:\n",
    "\n",
    "```Python\n",
    "from keras.layers import Dense, Activation\n",
    "\n",
    "model.add(Dense(output_dim=64, input_dim=100))\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(Dense(output_dim=10))\n",
    "model.add(Activation(\"softmax\"))\n",
    "```\n",
    "\n",
    "Once your model looks good, configure its learning process with\n",
    "``.compile()``:\n",
    "\n",
    "```Python\n",
    "model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to, you can further configure your optimizer.\n",
    "\n",
    "```Python\n",
    "from keras.optimizers import SGD\n",
    "model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now iterate on your training data in batches:\n",
    "\n",
    "```Python\n",
    "model.fit(X_train, Y_train, nb_epoch=5, batch_size=32)\n",
    "```\n",
    "\n",
    "Evaluate your performance in one line:\n",
    "```Python\n",
    "loss_and_metrics = model.evaluate(X_test, Y_test, batch_size=32)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or generate predictions on new data:\n",
    "\n",
    "```Python\n",
    "classes = model.predict_classes(X_test, batch_size=32)\n",
    "proba = model.predict_proba(X_test, batch_size=32)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN in Keras\n",
    "\n",
    "Whenever you train or test your LSTM/GRU, you first have to build your input matrix $X$ of shape ``nb_samples``, ``timesteps``, ``input_dim`` where your batch size divides ``nb_samples``. \n",
    "\n",
    "For instance, if ``nb_samples=1024`` and ``batch_size=64``, it means that your model will receive blocks of 64 samples, compute each output (whatever the number of timesteps is for every sample), average the gradients and propagate it to update the parameters vector. \n",
    "\n",
    "> By default, **Keras shuffles (permutes) the samples in $X$** and the dependencies between $X_i$ and $X_{i+1}$ are lost. \n",
    "\n",
    "With the stateful model, all the states are propagated to the next batch. It means that the state of the sample located at index $i$, $X_i$, will be used in the computation of the sample $X_{i+bs}$ in the next batch, where $bs$ is the batch size (no shuffling).\n",
    "\n",
    "> Keras requires the batch size in ``stateful`` mode and ``shuffle=False``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Data\n",
      "Input shape: (50000, 1, 1)\n",
      "Output shape\n",
      "(50000, 1)\n",
      "Sample:  [[100.]] [99.98797429]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAACSCAYAAABIW82mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXmYXFWZ/z+n9qre93T2hSQQ9iRsArIYlcVRVGRxdNSfyjg6royKoozzKDPquKKOIy6IjiAgRBCQfZMtG9lDQjprdyfdnd6ruvaq8/vj3nPr3KpuAiadDl3n8zx5+tZbt6vurVS/3/Mu5xwhpcRgMBgM5Ytnoi/AYDAYDBOLEQKDwWAoc4wQGAwGQ5ljhMBgMBjKHCMEBoPBUOYYITAYDIYyxwiBwXCUI4T4sBDi2Ym+DsPkxQiBYVIihNgthEgIIWLav59O0LU8JYT42ES8t8HwWvBN9AUYDOPIP0gpH5voizAYjnZMRGAoK4QQPxdC3K09/o4Q4nFhcb4QokMI8VUhRK8dVfyjdm5QCPE9IcReIUS3EOJ/hRBh7fl3CSHWCSGGhRA7hBAXCSFuBM4FfqpHJUKIY4UQjwoh+oUQ24QQV2iv0yCEuM9+nZXAvCPy4RjKFhMRGMqNa4F1QogPAzuAjwKnSCmlEAJgCtAITAPOBB4UQqyWUm4Dvo3llE8BMsBtwA3AV4QQpwO/Ay4HHgdagSop5UNCiLOB/5NS/gpACFEBPGr/7sXAicCjQohNUsotwM+ApP0ac4CHgV3j+qkYyhoTERgmM38WQgxq/z4upYwDHwR+APwf8GkpZUfR731dSpmSUj4NPABcISyVuAb4vJSyX0oZBf4TuMr+nY8Cv5FSPiqlzEspO6WUW8e4rncAu6WUt0gps1LKtcDdwPuEEF7gvcANUsoRKeUm4NbD95EYDKWYiMAwmblstBqBlHKFEGIn0AzcWfT0gJRyRHu8B5gKNAERYI0dOQAIwGsfzwAefI3XNQs4QwgxqNl8wO/t9/EB7UXXYDCMGyYiMJQdQohPAUFgH/Cloqfr7NSNYqZ9Xi+QAI6XUtba/2qklJX2ee2MncsvXuK3HXhae51aKWWllPJfgANAFktY9GswGMYNIwSGskIIsQD4FvABrBTRl4QQpxSd9h9CiIAQ4lysNM5dUso88Evgh0KIZvu1pgkh3m7/zq+Bjwgh3iKE8NjPHWs/1w3M1V7/fmCBEOKDQgi//e80IcRxUsoccA/wDSFERAixCPjQ4f8kDIYCRggMk5m/FM0jWI5VF/iOlHK9lHI78FXg90KIoP07XcAAVhTwB+ATWq7/y0Ab8KIQYhh4DFgIIKVcCXwE+CEwBDyNlQIC+DFwuRBiQAhxk11feBtWfWGf/Z7fwYpSAP4VqLTtvwVuOcyfi8HgQpiNaQwGCyHE+VjdPdMn+loMhiOJiQgMBoOhzDFCYDAYDGWOSQ0ZDAZDmWMiAoPBYChzjBAYDAZDmfOGmFnc2NgoZ8+ePdGXYTAYDG8o1qxZ0yulbDrYeYdFCIQQv8GaeNMjpTzBttUDdwCzgd3AFVLKAXvNlh8DlwBx4MNSypde7fVnz57N6tWrD8elGgwGQ9kghHhNy5McrtTQb4GLimzXAY9LKedjrcZ4nW2/GJhv/7sG+PlhugaDwWAw/B0cFiGQUj4D9BeZ30Vh1cRbgcs0+++kxYtArRCi9XBcRzHpbJ6ntvWwbzAxHi9vMBgMk4LxLBa3SCn328ddQIt9PA33yoodts2FEOIaIcRqIcTqAwcO/F0X0DeS4sO3rOKBDfsPfrLBYDCUKUeka0hakxVe14QFKeXNUsqlUsqlTU0HrXWMSmtNmLqIn919Iwc/2WAwGMqU8RSCbpXysX/22PZO3EvsTrdt40JTVZDeWGq8Xt5gMBje8IynENxHYfncDwH3avZ/sveIPRMY0lJIh53KoI+RVG68Xt5gMBje8BwWIRBC3A68ACy0N//+KNb+rm8VQmwHltmPwdrFaSfWcr6/BD55OK5hLCqCPqKprMs2nMzw9h8+w6NbusfzrQ0Gg+ENwWGZRyClvHqMp94yyrkS+NTheN/XQlXIx/6hpMu2oX2Ibd1Rvv/INt66qGWM3zQYDIbyYNIvMVER8BFLuiOCwUQagHQuPxGXZDAYDEcVk14IIgEviYy7RjAYzwDWzuPFmNVYDQZDuTHphSDo95LKuoVgKGEJgdfjloKheIbF33yUu9d0HLHrMxgMholm8guBz0Mqm3eN9JUQZHPu0f+mfUMMxDP8+PHtR/QaDQaDYSIpCyGQEjKa04/aNYPibiJVVA75J/3HYjAYDA6T3uMFfV4AV3oonbWKxMVF5Fgy43peIaXk7jUd9ETd3UcGg8EwGZj0QqBG9ynNuatuoVQ250oZRYuEQbHjQIxr71rPV+/ZNI5XajAYDBPDpBeCQkRQEIKMfZwvShnF7FRRcUSwrSsGQFtPdFyv1WAwGCaCyS8EKiLQWkj1+QN6ykjVDEbSRe2m9ryD0RpLdxyIlXQlGQwGwxuJyS8EvlFSQ9pxMlM4VjWDkVTWlTIaTlj24i6j/UMJ3vL9p/nm/VsO/4UbDAbDEaIMhKA0NTRWRKBSQ9m8dJ0TtYvIubxbCDZ2DAHw5Na/b78Eg8FgOBooAyGwbjGZKe0aArdAjGjtpHFtxdJhWwhGitpN+0bSY77vS3sHHAExGAyGo5nJLwT+VxcC3e4ShXTB6atuopF0dvSJaXl3cbm9P857/ud5vnLPxsNxCwaDwTCuTHohCHit1JCe38/k8kQCo6SMxogUhm2Hn5e41i1SaxaVpIw6rZTR6t0Dh+UeDAaDYTyZ9ELg91nrCWVy7hpBVchagTtZ1E2klh/SRSGeLq0jQCEiiI2RMsoXLWCXzeX55v1bWNc++Hffj8FgMBxuJr0Q+DzWLerF30w2T1XID7hH/qlsbgy7XkfQage2ECQzeZfQDNpCUBwprO8Y4tfP7uLf7lp/aDdlMBgMh5FJLwQBr3WL+sQxPSJIFdUOqsOl9tQYNYVhrRisL1cxaAtEqmhi2p6+EaAQSehs2TdcUow2GAyGI8GkFwKVGsrm3CN8NfLX5xGks3mqgqNHCpVBX4ldFwV9eYqBuBURxFJZV1TQb0cKxTOXOwbiXHLT3/jqclNcNhgMR57JLwRORKClhnJ5qmzHrqeM0lktUtCFIJOnJqyEo+D8dRGJZ7SIID56pNAbGyNl1G4Vl59+pXQ+wnAyQz5vNssxGAzjx+QXAqdGoKWGsnkqglbXUGkRWUUEemooN6pAJDM5Z1G7hFZQ1lM8evqofyQFlEYKB+xVTYtnLvfFUpxx4+N89+Ftr/2GDQaD4XUy+YWgqGsom8uTl1BhRwTOAnR5SSYnqXZqBwePCFLZPLXhgG0/eMqof+TVI4XiLqPVewZIZHL87oXdJffVF0u50l0Gg8Hw9zL5hcBODSmnqYrGFQF3akj9HDU1lC0IQXFEUBspFYhEJufMU0hoKaN4evRIoTeWsp/PuSKU7mErUijeRvlANMVZ//UE33rg5YPev8FgMByMSS8EPntigEoNqUJtpe3wlTAoB1+cGsrb6w5Vj1ojyI1ZO6iLWJFCIj36fAS9c0hFBFBoSYWCEGTz7q02V+/uJ53Lc9uKvSX3+1xbryMsBoPB8FqY9EIghMDvFc5IO5WznHGFPWJXwpDOjh4RqEjBiQiKUkOOwy+KCOoq/CX2ZCbnpJ70lJFeUxjUhKBn2HLomZx0vY7aUlMWLYzd1hPjH3+1gs/+ce2rfSQGg8HgYtILAVjpIVULUBFAwOdxCUQhNaQcvvVYjfSLU0NSSqtGECltQ02mc9RXBAF3OiieztFQadldKaNRlq0A96J2A5q9y4kUpKvovLHTmrH84s5+1/1ncnk+c/taHn+5e9TPx2AwlDdlIwTZvDs1ZAmBpyQiiAS8+DzCSQ0px1+cAnLsY9QI6kexx9M5GioCzrFjT2VpqbYEQo8O9ILygCYKTkQgYTBesHcOJACrPVVPJa3Y2c996/eNOqN5U+cQQ3GzSqrBUM6UjRA4RWElBF4vAZ+nEBFoAhH0eRxHryKDiqAPjygIgHLwqmtIpW4yuTzZvKTWqRG4U0P1owlBOkdT1ShCkMo6qSo9UtCdf78mEJ2DSdfvKnb1WlttDhQ5/F29I7zjJ8/y+TvXlX5oBoOhbCgTIRBaaijv2CyBUMViyzEHvB6Cfq8jDMoe8nsI+b2lEUHYjxCF2oF6Xjn8RKaQSoqns4XUkEsIsjTZdn2bzJF0ltaaEACxVMGJDyezThFcTx/1DBeEoE8rQO/tjzvH+vuu3WutjvrE1h7X5xVNZjj720/wy2d2YjAYJj9lIgSFkX9KG/kHvGNHBMUOP+jzuiIF9XzI7yHk8zoRgfpZF3EXi1NZa/5CY2VpRDCiRQR6TWEklaWl2hICvbgcS2aYUR8B3NGBuxOp0DnUYaeMiu27ekecYz2F9VxbL52DCW580N2emszk+OQf1vCX9fswGAyThzIRAkHGrhEoxx/wWsXi4hpBwOch4CukklREEPS5IwJVHA75vYT8HudxSrOH/V4StmN3UkmRAELg2DO5POlsXksNuZe8VkKgp4yiyUKkoAvEcDLD9Low4G5JHdDEoidaEILdfYVI4YBm7xoqRBb6ukgrdvXz4MYuvnz3BnQyuTw33LuJJ7aaYrTB8EakTISg0DVU7PALbaVq5G/XCIoce9CJFNwRQdDnsRx+UUQQDngJBwp2FQFEAl4ifq/zWP2siwTwiEJEkM3lSWbyhSJy0Z4IrTVh51gxnMgyqyFiH+s1hQxTbeHQHb5a8gKgJ1pw/rqI6KKwbu+gc826QLy4s4/fvbCHz9zurjVkcnm+cs9GHtrUhcFgOHopHyEYJQU0lj3o8zoRQVJFBH4vIb+3pJsoZNsdIbAddtiJCKzz1PORgJdwwOe0jKrzK4I+KgI+JyIY0QTC5xGOw8/m8sTTOabWjhER1EacY8eeyDCvuRKAvhFdCDJMq7UERc1ZAHf6qHOwkFbSaw17+wtpJSUQsVTWldp6tq2X21fu5TO3u+c1JNI5PnLLSv6wYg/FFC/IZzAYxp8yEQLhzB8oFIs9rmJxOlsY+Qd8noLDL4oIRqsRBP1ep1jsRAROyqhUICIBr/NY7Y0cCXiJBL2OI1WpoMqgj8qQz3mshKI2EiDk9zgCkbEForU2hBBWQVkxmMgw064p6HWEgZE0x06pAtwpo95YmqDP+mrsHyoIQbdWjN6vRQo7DsSc450HCgKxdo9VjE7n8q7upqe29fDktgNcv3yTq821L5biTd9+nH+/dxPFbNk37HoNg8Fw+CgTIfCUTBxTxeJ00Qg/4LWKwoWuocLIP+iqEajUkJewf5SUkd9LJOArSQ2FA5YQKIcfT6lIwY4IlEDYDl5FCsrhR+3uoaqgj6qQ34kI1M/asJ/KoM9JDaWyOeLpHFOqQwS8HkcIpJT0x9PMaawA3HWE3ljKEQjd+XYNJzl+ajXgFoLu4RQBWzj0COKVbl0gCsertL2c2/sL5z/2cjfdwylufWGPa/XX7d1RLv3J3/jYravQyeUl339kG08WdT2p52TxIk0Gg2FUykIIXLUAx+GrGkHpRLOAPo9AKxa7IgJHIIqLyO7UkHL47tSQXiOwHX7AS0WwMPKP6RGBZlcOvzLkoyroI2qngJTjrw77qQ75ndSQcvy1ET/VYb9znsrzN1UFqQ75XPMUemMp5jRW4PcKV3tq91CSk6bXOseOPZrk1BmWfZ8mBPuHEk7qSU8r7ewtiELbgahzrAvEln3DzvFDm7qQEl7aO+iqcTy1rYefPNHGR29d5VqJ1VqU73G+cKd7Ap2UkrvXdLCho3TPaCMahnKmLITA5ylNDZUsMVHs8DNu4QjatYNCEbkQEYT8XqeWoBeLQwGvM49AdQmF/T47IigqIgcte3EKqMJODcWKBKIq5LYrx18d8lMV8jmCoWYN10QC1IR9jjCokX5dRYDaSMCJCKSU9MZSNFUFqYsE6LPrBSOpLNFUlpn1EeorAuzX0kQ9wymOa60m5Pc4s5sB9g0lOX1OPUK4haC9P87ps+sB2NVbsO/uHXGK4209BbHY2DnkHKu5DwCP25FAXrrPWb62g55oiuVrO11zK55t6+Xau9Zz9c0vuiKOA9EU53znyZI1mvJ5yR9W7OH5tl6KGUllXa9hMLyRKQshGLUo7C1aYiI3erHYqRGonL/9x590FYs9Ts5fpYisiMBDMl2aGgr7fc7jkeKIIO12+BVByx6zhUFFAJVBn8vhDyesn9Vh98jfiQjCfmojAeexcvz1kQB1Eb8TEYykcyQzeRorg9RXBBzBUOsbtVQHaa4KOsXlWCpLLJVlSk2IqTVhJ2WUzubpjaWY1RBhSnXIEQIpJR0DCU6aXkNV0MdubS7Dnv44Z89rxOcR7NTsu3pHOGtuAwDbNYF4pSvq1D50IVi5qyAWz+/oc47vX7/fucdV2jm3r9xL52CCe9fto62nEKH8dVMX1y/fxD/+eoVrvkbHQJzTbnyMd//sedey4fF0ls/fsY6bHt+OjpSSRzZ3sa69NBIZTmZcczgMholgwoRACHGREGKbEKJNCHHdeL6Xf4ylJPT5AiWpoZJawOgRQSE1pEb+BftobaWRgCoWu2sEhdqBu0ZgpYa8JamhqpBVC1DrETkRQdhnp4Ysu3LwtRE/NWH/qBFBTSTgODoVATRUBmmsDDqpITVTuakqSHN1yGk37dEEYkpNiH12cbl7OImU0FoTYnpd2IkUDsRSpLJ5ZtRHmNUYcQQins5yIJpiblMFMxsiTk0hl5fs6Ytz0owaWmtC7LCFQErJ9p4Y58xvpDrkY2tXwYG/vH+YS09qJejzsHlfQSDWdwxy2uw6PAJW7i4szPdcW68TieizrNXEOSnhkS2FORIPbeoins6xZf8wT28rbC96//r9LF/byQ8efcWVfnpiaw/X/H4N7/mf51yps02dQ5xx4+O8/UfPuNqAe6JJrvzFC3zhjnWuLqp0Ns8PHn2lpNtKSsmT23pYP4rQdA0lXSJmMIzGhAiBEMIL/Ay4GFgEXC2EWDRe72fNIC7tGgoUzTgWwkojBV0TyjS7v7RrSKWGUkWpoZDfPY8gUVIsLq4RqPZROzWULhSLdYdfEAJVLHbXCGrCfqrDhWLxoGavCRdG/k5EUGFHBPZ5qnW0sTLgigiUQNRXBGjRIoJu+2dLVYjWmrAz70BFBq01YabVhp3ZzernjPowM+sjtNtCoARhZkMFcxsr2G2njDoHEqRzeeY2VnBMc6UTERyIpRhKZFjQXMmxrdVs3W/VFIYSGToHE5wwtYZjp1Sx2a41JDM5tvfEOGNOA/Obq9hoO+pMLs+GjiEuObGVWQ0R1tidTvm85MVdfVy+ZDrNVUGe3V5IDz2xtYe5jRVUh3w8sqUwR+KhzV3URfwEfR7ueanTsavjvMS1h8Rvn99NIpNjT1+c/3ux4Nx/+kQbK3b1c8/aTv60pr1gf7KNmx7fzvXLN/Hw5sL73rZyLx+5ZRWX/c9zrNYE7vGXuznnO09w7nefZHt3QSjbemKcfuNjLPvB0y5h6okmef8vX+Tjv1vtnrEez/DV5Ru58YEtriVKEukcP39qB7c+v9tVo8nm8vxl/T4e2dzlqr1IKVm1u39UweocTDjfBZ3hZMbV6aa/h9mh7/Dhm6D3PR1ok1LuBBBC/BF4F7BlPN7MqhG4R/5qraFMtlAsDng9CCFcI/90Lk/QZ9lDvkKbaCqbxyOs1wn5vK6uISG0mcjFqSHVTeSkhlSNwEoBqQhBLxa/WhE5WhwRhNzFYvUHXRsOuCKCgZFCyqguEnBWNz0QtX46qSE7ElCRQWNlkObqIAdiKfJ56UQGzdUhWmtC9EStLTRV2+nU2hDT6yL8ZcN+srm888c+vS7CjPoIj23pcUb9ALPqI8xuqOBv23vJ56VTWJ7bVMm8pkruXN1OPi/ZbnckLWipYmfvCMtf6kRK6QjCca1V7O0f4a+bLGe0Zf8wubzkhGk1dA0neWpbj31+lEQmx+KZdQzGMzzb1ouUkm3dUQbjGc6a20Aik2PV7n6klERTWVbu6udj585lV2+M59r6kFISS2V5dnsv/3TWLHb2jvDE1h7+/R8WMZLO8fjWbj545ix29sZ4cON+rn3bAuLpHA9u3M9Vp81ge0+Mu9d08M9vnksik2P5S51cdspUtnXHuOW53VyxdAbJTJ7fv7CbCxY2sbsvzs+ebONti1pIZfP86LHtHDulioF4mm//dSt3feIsUtk8N9y7mSk1IaLJLF+/dxO3f/xMpISv3LOBwYTlYK+7ZyO3fuQ0AL541wae39GHR8AX/7SBmz+4BIBr71rHYy9bkdL+oSQ/ufpUpITP37GOh2xB2tAxxPfedxJSwr/dtZ4/r7OiqQ+cOZNvvusEpITr/7yR21dawnbNm+fylYuPRQjB71/cw3/ct5mclHz6wvl8ftl8AO5c3c4N925GSvjssvl88vx5APxxVTs32rvzfW7ZfP7f2XMQAu5a3cH3H7X29/7UBcfwgTNmAXDP2k5+8fQOAD5y9hyuPG0GArh3fSf/9+Je8lJy9Wkzee+S6XiE1bRw24o99I2kueyUabzzlKn4PIJVuwe4e00HA/E0l57UysUntOLzCF7c1cdf1u8nns7yluNaeNuiFnwewYpd/Ty8uYt0Ns858xu5YGEzfq+HFbv6eHrbAbJ5yelz6jlnfiMBr4eVu/pZsauPTE6yeGYdZ81tYHeflSI92W7GGC8mSgimAe3a4w7gDP0EIcQ1wDUAM2fOPKQ301NDqVyegO3Y/T7highU77yrayiTI+izNrEpjghCfq8lEH6PM/JPZnKEfJY9EnCnhpT4RAJe4pmcsxCdzyMIeD1UBL2MpLNIKRlJZfHa0UlV0EfMtseSWTzCSjFVhyx7Pi8ZThTZU8qeQQgrgqgOW+2mubxk0LZX25HCcDJLNpd3IoKmqiCNlQGiqSzJTM5JDdVFArRUh8jlJX0jaWduQUt1kNZay34glnJHBHVhcnlJ13DSaS+dWhtmVn0F6Vye7uEke5UQNESY3VhBKpunazjpzEuYY0cE8XSOruGkM8I9pqWSBb1VRFNZOgcTvOwIQTXtAwluX9nOvqEkGzusFNFJ02voiSb505oO9g0leckuPi+eVcdAPM3ytZ10DCRYsdOqLZwxt55YKssDG/bTMZBgQ8cQ2bzkwmOb2doV4uHN3eztj7O+Y4h0Ls9FJ0zh5a4oT2ztoa0nxuZ9wyQzed55ylS2dUX52p838fL+KFv2DxNP57h8yXS2dUe5fvkm1ncMsa1rmGgqywfOnMX2nhhfuWcjq/cM8Ep3lIF4hk+cN4/tPTG+9udNvLizn21dwxyIprjpqlPZ3hPlhns387ftvbzSHaVzMMFtHzuDV7qjfOMvW3hmey+dAwlW7R7gu5efRDyV5Rt/2cJ96/cxGM/w9CsH+Oa7jieRyfGfD27lT2s66B9J89jLPXzDFrX/fngbp8+ppzeW5qHNXXzt0uMYTma56fHtzKyPMJzM8Od1+/jsW+Yzksryq2d3EfRZ3XO3r2znY+fMIZ7JcfMzO4kmre/oHavbefOCJuoifm56fDs7DsQQwP0b9nPm3Hpqwn7+++FtrLKjnae2HeCMOfWEA16+9cDLPLy5CyEEK3f1s2RWHX6v4IZ7N3PHqna8HsGGjiFOnFaD1yP46vKN/PrZnfg8HrZ1R1nQUonX4+FLd2/gpie2E/B62Nk7QlXIR31FgGvvWs9//XUrAa9g31CSioCXmrCfR7Z08/XQJvxeD30jaSoCXiJBH/eu20fYby1lH01lCfu9BP0e/riqHb9X4PUIkhnLB/k9gt8+vxuPAI8QZPMSn0fg8Qhu1hZ8XDqrjj/9y5sOwQMenIkSgoMipbwZuBlg6dKlh9Tb50oNZSUBex/jgNfr2rM4oBy+nRqSUtp2j8uez0uSmTwhv3V+yO8ll5dkcnkSmRwhv3V+2O8lq+zpnHN+OGCdn8pak8DCASUcPvLSKjiPpHJU2PaKoA8pragimsxQEfQhhKAq5EdKiKWzRJMZqkJ+hBBUhy17NJllKJGhKujD4xHOngrRZIaheJrqkB+vRzgL5A0ns44Q1FcEaLRXRO0fSdM/kqI65CPg89Bsr4vUE03SPZwi7PdSGfQ56x/tH0rSNZSkKmRFM2r9o86BBPsHk1SHrGhGFXr39sfZ2x+nKuSjNhJgrj23YXfvCLvsP8qGigDzmqzZ0W09Mbb3xKgJ+2mqDDpzHl7pjrK1K0p9RYDmqqAz52Fz5xCbOodoqAjQWhPiZLsFdt3eQda1D9JcFWRqTYils6xOptV7+lm9Z8Cub0Q4Y65lf2FnHy/u7KMm7GfxzFpnfainth1g5a5+mqqCLJ5Zx/S6CF//8yYe2tTF2vZBptaEWDKzjrmNFfz7fZu5d30n6/YOMrshwpJZdSyYUsU379/CHava2dAxyMKWKpbMqmPR1Gr+68GXueW5XWztinLitBpOn1PPyTNq+dFj2/nuw1vpHkqydFYdZ81rYPGsWn7x9E6+fu8mhhMZ3jSvgTcd08iS2XX89vndXHvnOuLpHGfNbeB9S6aTl7B8bSfX3b2RTC7PBQub+MCZs8hLeOzlHr589wbyEi49sZUPvWm21cK7Z4Ab7t0MwPuWTOej58wBYE/fCD987BUA/umsWXzOHtXHMzl+/ewuAD5x3jy+fNFCpLSaI375t10IAZ88fx7Xvm0hHgHHNFXyw8dewesRXPvWBXzygmPwCPjNc7v53sPb8Aj42qXHOVHAnavb+d4jr+AR8K3LTuD9p89ECLh33T5+9mQbAN+9/CQuXzwdIeCBjfu55bndSCn50ZWn8M6TpyKE1Rhwx6p2JPDP583lnSdPI+T38NS2A/zppQ7yecm1b2vh4hOnEPJ5eWb7AR7YsJ+clFywsJllx7UQ9Hl4YWcfj2zuIpuXnDu/kfMWNBPweVi1u58nt/WQyUrOmtfAOcc04vcK1uwZ4LkdfWRzeU6bXc8Zc+vxeTys3TvA6j0DZHJ5lh3X8ho93d/PRAlBJzBDezzdto0L7p0vomVLAAAXz0lEQVTIco5j9/vci87pEQFYUYIeKShHbtlzjj1s25MZq+MmrAkEWA48kS5saO9sbJ/OEU/lqAhY/w2VQcs+krY6cSqCtt3ek2AklSWWylFt76JWHS5sexlNFvYuUA5/OJlhOJl19ltWDn8gnmEwkXHOU3snDMTT9MZS1Eb8+L0eZ8ns3liKvpG087ipynL4PcMpeqIpmquDCCGYUm05/K6hJPuHEkyxF8xTcwk6BhLsG0ww1X6sC0H7QNx5PNsWgl19lhDMbaxACMEx9jIZOw7EaOuJMb+5EiEEC2wh2NplCcHCliqEEBw3pRqPgM37htm0b5gTptVY9tZqAj4P69oHWN8+yCkzahFCsHBKFVUhHyt3DbBmzwBLZtUBsLCliqaqIE9u7WHFrn7OW9CEz+thTmMF85oquG/9PrbsG+Y9i6fh8Qim1IRYPLOWW1/Yw0A8zcfOnYPHI2ioDPLm+Y384mlrtPelixZawh3yc+mJU7l9pVU/+OZlJzgDg6tPn8kv7NHhz96/2I5AvXzivLl8y06PfO+KkwGrXvW1S4/jk7e9RMjn5evvWOTYv3/FyXz4llXURQL89/tOQgiBV8APrjyFL9yxjuqwn+9fcYpj/+n7T+XbD26lIujjK5dYKRwh4MdXn8qv/7aLSMDLh8+ejRDWcujfe9/JnDu/ibDfy8UnTHHsN152Au+wC/dLbKEVAq6/dBFXnT4Tn0cwq8H6/wb49Fvm84EzZ+ERwtn0CeCj58zhg2fOQgirvqe48rSZXHlaacbgslOncdmp00rs7zhpKu84aWqJ/ZITW7nkxNYS+wXHNnPBsc0l9vMXNnP+wlL72cc0cvYxjSX2M+c2cKbd+aZzxtwGzngd9vFiooRgFTBfCDEHSwCuAt4/Xm+mt4lmshK/1/qSFi9DrY/8wYoSUqPYU9lcUURg2ZMZFREURv6WPUc8kyNiO3wlBPFMjpF0VhMI6/l4Kkc8rQmB/TOayhJLZZzHalvNqO3w1WPl4IcSGYY1h6/2Vx6MpxlKZJxtNtXPgZE0fbG0EwmoJbP7YpZd7a6mOmysiCDprJCq1j/aN5igazjFlBpltyOCwQT7hpLO49baEF6PoL0/Tnt/nAUtlkOfUh0i6PM4EcFps+uc66kO+WjrsYTgrYuskVJ1yM/UmhBb90d5pTvKFUtnOJ//nMYK1rYPsr07yoXHNln/7z4Px0+t5sltB9jZO8J7l0wHwOsRLJ1V5zhk9YcohODN85u4+6UOAC46YQqKtx0/hZ8/ZeWf33VKwfFcvmQGX12+EYArlxbGPP983jyefuUA9RUBrtYc2OeWzeelvQNMrwu7zv/ssvlEU1mm1oS45MTC+370nDnUVwRoqgrypnkFx3Pxia08+vk3UxH0OQsTAiyZVc+ar70VjwCf5kjnNVVy77+eQzHNVSF+cOUpJfbKoI/P2qN9Hb/Xw+X256gjhHBdn46K8Iqps79nxai/Q8PhZ0KEQEqZFUL8K/Aw4AV+I6XcPF7vp7aqzOfdqR6/10NeWi2KqlgM1pwBsOYQuOw+5djzJDOFiCCoRQQpXQj8hZF/Ip3VBMJn27PE0zkidiRQ4YoIrNQQFIQglsxaKSP7PBUBWBGBlQICtxAMJTJOBKFGWIPxDIPxgkAox983YkUESgCU/UAsRf9I2lnZVKVEuodT9AwnOdFOtdSE/YT8HrqGknQPJZnfbDmAkN9LU1WQjoE4+4cSLJlV63z+U2tD7O6L0z6Q4C12COzxCGY3VLC1K8q+oQSzGy0Ho6KC1bsH6BtJOxECwMIpVTxkF+aOa61y7MdPreE+uw30lBl1jv202fVOHvbc+QVHtWxRC0/aLaFvW1QIyd9/xgzuWdtBS1WIC7UR4sfPncvzbb0saKlyBAvgytNmEE9nmdVQwVzN4Z05t4HHvnAe1WG/y+HNqI/w5L+dTzGRgI//fPeJJXYhBO9ZXOp4AY5prhrVbhypYSwmrEYgpXwQePBIvJf6A8jk3Y7dsefy7lqAtzDyT2XzjqNXI/9Suzs1pNcIwCoUJzJaakhLGcXTWS1SKKSA4qnSiEDN7lUOXI8I9D0KqvWIIJlhbqPliOq0FNBQorC5TYM28u+NpZ3cum7vG0mx2E6VBH1e6iJ+p0awzBYGIQRTa6xW0QOxlHM9ANPrwmzviTEYz7hGqjPrI6zZ3U86m2dGXcE+uzHCw5ut3n3dkc5rquSuNdbIfJF9nepYOfDT5xRC6guPbXaE4Kx5BfuVp83gt8/tZuGUKk6cVuPY37t4Olv3R1k4pcqJdMAaUT/4mXNpqAg4/99g1VJGG1F7PYKPnTu3xF58PwbD0cBRWyw+nCjHn8lZEYHKMfq9hVqAKzVkO/J0Nk86m3OEQUUEqkYQUrUDXyE1lNQiglBAE4K0lhoK6kJQ2MdYOf6RdI5YKsv0SMRlj6ayxJIZptkpGD0i0Pc3dmoEKiKwawmuGkE8Ta19nnr/vliK3mjKiQQiAWvZi55okoF4xkkNgZU62NEzQiKTcznMKTUhNnYOkctLl316XcSZoKUiC7CE4Lk2q0NnfkthJKvqBAAnaY568aw6RwhO1Ub47zx5Gj9/agfHTqlmtvb6l5zYyopdfSyeWecIKliC8tQXz7e3GhWOPeT38s3LTmA0jmutHtVuMLzRKQshUDWBtO3wnaKwbVe7hBW6idwCoRxx0HH41shfjbydWkA2RzKbK9hVpGA7/AbNwYI1mSyezjGjznqsUj7xlGWvLEoBqdRQoUZg/RxOFrqGoLhGkC0Ul0N+PMKqBeg1gqDPS1XIR+dggmgq66R+wEoPbe+OkctLl725OujsQ9BcXbC31oSdZR1U8RcKBWOA+VrqYmZ9weHrjvZN86yiatjvdQnHpSe18ocVezh/QbPzuYOVGrr/0+fSYheuFQGfh/96z0mMxlTtmgyGcqY8hEBPAWkjfz01lMrlqQ3YjlGLCFLZPHWR0bqGSruJkvbIX6WGIlpEkMzkHGFwisXpHPFU1nFoqnsolsoyksoSCSqBUJGCNfKvDBYcO1gjf71rKBKw+pj7RtIkMjlHGFQL6d7+OHlZEAywHL7qwdcdfkNlwFnHR0/1NFeFiNqT2/RUj37O3KaCk59elPZRnLegie88tJVFrdWu6zn3mEY+t2w+S2bVuRx7dcjP/Z8+l9HQU0UGg+G1UxZCoEb46WyeTFGxGKxOInfXUMHhj5YySmXyrqJwyFcoCiczeedx2O9OASkBcNm12oFy+PG01U1UWdw1lMza9sJ8B79XcCCaIpuXTkSg5hKoWbzVmoOtiwSc2bqqZgDQUBFggz3pqrkoIlhrj/z1EXSLFgWM5fCnagJx3oImPMLK2avPFyznfdvHzmCmNuoHS7Q+t2wBBoNh/CkPIdDaQdO5vNOXr4QgncuRzuZKisipbM5ZYgLcqSF3RGDbVe0g4C4iq2JxqCgiUPMIittKhxMZkpm881g5/B57ITd1/WpSmVovRtkBZ+SvjhW1ET+b7PV39JROQ2XAmVyn5/b19I4+2tfteu3g/IXNVAZ9LDuuGY+nMJKfUR/hiWvPZ1pdaTrmTaP0XRsMhiNHeQiBtyg1VFQsTmdLZxBb9jypTMGup4aS2gzikq4hFREEdHuuZL5ANJkhnXM7fK9HcMCe3asiATW7WC0FrVJDgJPbB6jWhKA67OcVe0VO1f0D1uQxNadCTwHpzl8/XtBS6HCp1xz+eQubCPg8XKJNHgKrWPz0F893iZJCLwAbDIajh7IQAr+WGkrn8k7NIFhcOxilO8iKCLyu8532UZ975J+yHX5x+2g0mSWTk9qMYw9CWHsDQyESEEJQEfA6K3sqwQBLFLpsuyoqgyUEasE2PdVTE/Y76xypLiBwO/nmqsLxfK0nv06b0almg541t8Hl8Ftrwjz9xfNd76lo0N7PYDAc/ZSHEBQ5/OBB2kcDWkQwWu2g0CbqTg1FU1myeelagwgKa/+HNYcf8Xudkb/u8CuCPseuO/zKoM9JAakiMUBVsLBvsT5ib9SiAH3kP0cr1NZqKaNj7Y6dmfURl8M/prmSu//lLGY3lI7m9SKxwWB441IWQhDQUkB6sdg9X2D0pST0NYWUw4+nLYdfHBGobSHVeV6PIODzOEs86+2O4YDPWeNfd/iqbx8KXURgCYHafEWfkVqrjd71FND0uoLD10ft+roueg5/6aw6vvj2hZwzSr5eRQUGg2FyUh5C4LPnEdgRgd9b6vD1GoH6mUjnyORkSUSgNn1RDt/vtXL7atOXsDbzNOz3Omv56/ZIwOukhnR7ZdDnFHP1Rbf0nHu95tj1VI8eEajuHY+wBElx1rwGpteF+dBZs12fkRCCT11wDAaDofwoDyHwWo42M0aqJ5XNWw6/SCDUJjDFkYLa3EVvgwz7vQwmLMceLHL4KjUUCbjtqvirp4YiAZ+zPaGeutGdf21Fwa4manmE+3rU0swLp7h766tDfv72pQtc6R+DwVDelIUQ+PWIYJSRv9oGslgg1IQp9dhjbyCj9gNWwgBWdDDgpIbcAqGEwGUPeJ0IQi0BAe40kR4RtNitmx6Bs7gcFHr7PUWO/cRpNdz47hNGXfrWiIDBYNApDyHQFpEbbeSvtnUstJVajjJaJBDqd4YSpQ4/6PM620KGXALhdTZ0d4/8C7+rF3/19I7e/6/W9vfb22kq1Jr57y5ae10IwT/aW/UZDAbDq1EWQqAc/Ii9H3BxqqfY4Vv7FnucWoA+8g/6PVpqaPSIQHf44UBhP+OqkDsFpNAdvmrpDPo8rlSPWttfFu3VNqUmxLNfvsDVGWQwGAyvh/IQAp8SAndKJ6j1+evnqeNYqjQFFPR5R40IQn4vu3qt/XX19I5eCNaFQJ+Nqy8BoWb76rswAZw+u57qkI+PnD2n5P70DiGDwWB4vZSFECinOlbxN5Zyp4as57wFgfAePCII+73YNV7XcschlxC4F3kDa+9WvatHRQSyaOhfVxFg5fXLXK9nMBgMh4Oy2LIoUNwFZDt2n8fah3WsWoCTGvK7BULVAvR5AbqDrgiOnvbRBUL1/BcXeZfOrsPvFVx9euk+rEYEDAbDeFAmEYHlbIu7g1QtYCwhGEyoeQH6CN9DJmeN1iuCbruiYhSHXxn0uUb+KiLw+9xa3FgZZNX1y1wFZIPBYBhPyiMiUMXi9GgO30tUdQ0V1QhGG/nr6aDIWBGBZle1AJ/XPfJX++oeP8oa+rWRgGvWr8FgMIwnZRERCCHwewUx1TXkdY/8VUQQdNUCvM7IPxJwt4kqKkbp/An5Pfi011HtoMXF32Oaq/jVPy3l5Bm1h3ZzBoPBcIiUhRCA5YhjY4z8h0dLDXndhWDFWCkg5fD1OgAUloDI54v6PoFli1pe/40YDAbDYaZshCDg85TMIwArIui2l3pwpYA0h188UxisuoP+OkoIfB73yH/JrDoWtVbzz+fNPVy3YjAYDIeVshECv3fseQEqBaSP/PX0kZ4aqrXX/NEnhEFBCIpT+xVBHw9+dvQ9dg0Gg+FooCyKxWA59kL76MFH/mPZ1bLPeooICou/VZluH4PB8AajbCKCgM/DyPDobaKKsL+0KBywt49UqLX9i1NAp8yo5b2Lp/OBM0v7/w0Gg+FopmyEwO8VZO2CrbtY7F4RVKH2/9XTQlCICIoX8PR5PXz/ipMP6zUbDAbDkaB8UkNFnUKK4BjHNfbIv6poE3a1CmjAVzYfncFgmOSUUUSgCYG3NP8f9ntdyzurTWGKl4A4ZWYty45r4ePnli7+ZjAYDG9EykYI3M6/cKxG/OExUkDFyz4HfV5+9aGl43SVBoPBcOQpm/yGPvlLb/1Ua/qE/a+tFmAwGAyTjbIRAlX0Le4CUttEBoty/idMrcHnEbzn1OlH7iINBoNhAiib1JBaF6i4C8hZ5bNo5N9cHeL56y6kodLs/GUwGCY3ZSMEEXvXsEhRCqh6jKIwWGJgMBgMk52ySQ2piCBUFBE0OmsEmWKAwWAoT8omIlDF4uKR/8kzalnUWs11Fx87EZdlMBgME04ZCYEVCXiLhMAsCmcwGMqdskkNqZZRv8+kgAwGg0HnkIRACPE+IcRmIUReCLG06LmvCCHahBDbhBBv1+wX2bY2IcR1h/L+r4eptVbhN57OHam3NBgMhjcEhxoRbALeAzyjG4UQi4CrgOOBi4D/EUJ4hRBe4GfAxcAi4Gr73HFnUWs1tRE/n7lw/pF4O4PBYHjDcEg1Ainly4BrjR6bdwF/lFKmgF1CiDbgdPu5NinlTvv3/mifu+VQruO1UBsJsPbrbx3tWg0Gg6GsGa8awTSgXXvcYdvGsh8RjAgYDAZDKQeNCIQQjwFTRnnqeinlvYf/kpz3vQa4BmDmTLPZi8FgMIwXBxUCKeWyv+N1O4EZ2uPpto1XsRe/783AzQBLly6Vo51jMBgMhkNnvOYR3AfcJoT4ATAVmA+sxFrRZ74QYg6WAFwFvP9gL7ZmzZpeIcSeQ7ieRqD3EH7/jUi53XO53S+Yey4XDuWeZ72Wkw5JCIQQ7wZ+AjQBDwgh1kkp3y6l3CyEuBOrCJwFPiWlzNm/86/Aw4AX+I2UcvPB3kdK2XSI17laSllWmwiU2z2X2/2Cuedy4Ujc86F2DS0Hlo/x3I3AjaPYHwQePJT3NRgMBsPho2xmFhsMBoNhdMpFCG6e6AuYAMrtnsvtfsHcc7kw7vcsZPGmvAaDwWAoK8olIjAYDAbDGExqIZioBe4OF0KI3wgheoQQmzRbvRDiUSHEdvtnnW0XQoib7HvdIIRYrP3Oh+zztwshPqTZlwghNtq/c5OY4KnXQogZQognhRBb7MUMP2vbJ/M9h4QQK4UQ6+17/g/bPkcIscK+zjuEEAHbHrQft9nPz9Ze66ha6PFg2OuPrRVC3G8/ntT3LITYbX/31gkhVtu2o+O7LaWclP+w2lN3AHOBALAeWDTR1/U67+HNwGJgk2b7LnCdfXwd8B37+BLgr1hzNc4EVtj2emCn/bPOPq6zn1tpnyvs3714gu+3FVhsH1cBr2AtTjiZ71kAlfaxH1hhX9+dwFW2/X+Bf7GPPwn8r318FXCHfbzI/o4HgTn2d997NP8dAF8AbgPutx9P6nsGdgONRbaj4rs9mSOC07EXuJNSpgG1wN0bBinlM0B/kfldwK328a3AZZr9d9LiRaBWCNEKvB14VErZL6UcAB4FLrKfq5ZSviitb9HvtNeaEKSU+6WUL9nHUeBlrLWoJvM9SyllzH7ot/9J4ELgT7a9+J7VZ/En4C32yM9Z6FFKuQtQCz0elX8HQojpwKXAr+zHgkl+z2NwVHy3J7MQTOgCd+NIi5Ryv33cBbTYx693ob9p9nGx/ajADv9PxRohT+p7tlMk64AerD/sHcCglDJrn6Jfp3Nv9vNDQANH6UKPr8KPgC8BeftxA5P/niXwiBBijbDWUoOj5LtdNltVTkaklFIIMenavoQQlcDdwOeklMN6qnMy3rO0Zt2fIoSoxZqgOak30BZCvAPokVKuEUKcP9HXcwQ5R0rZKYRoBh4VQmzVn5zI7/ZkjghebeG7NzLddhiI/bPHto91v69mnz6KfUIRQvixROAPUsp7bPOkvmeFlHIQeBI4CysVoAZq+nU692Y/XwP08fo/i4nkbOCdQojdWGmbC4EfM7nvGSllp/2zB0vwT+do+W5PdAFlvP5hRTs7sYpIqmB0/ERf199xH7NxF4v/G3dx6bv28aW4i0srZaG4tAursFRnH9fL0YtLl0zwvQqs3OaPiuyT+Z6bgFr7OAz8DXgHcBfuwukn7eNP4S6c3mkfH4+7cLoTq2h6VP8dAOdTKBZP2nsGKoAq7fh5rN0bj4rv9oR/Ecb5w78Eq/NkB9b+CRN+Ta/z+m8H9gMZrJzfR7Fyo48D24HHtC+BwNoGdAewEViqvc7/wyqktQEf0exLsbYb3QH8FHuC4QTe7zlYedQNwDr73yWT/J5PAtba97wJuMG2z7X/sNtsBxm07SH7cZv9/Fztta6372sbWsfI0fx3gFsIJu092/e23v63WV3T0fLdNjOLDQaDocyZzDUCg8FgMLwGjBAYDAZDmWOEwGAwGMocIwQGg8FQ5hghMBgMhjLHCIHBYDCUOUYIDAaDocwxQmAwGAxlzv8HQC6fTBIkDF8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd2b8436b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "'''Example script showing how to use stateful RNNs\n",
    "to model long sequences efficiently.\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense\n",
    "from keras.layers.recurrent import LSTM, GRU\n",
    "%matplotlib inline\n",
    "\n",
    "# since we are using stateful rnn tsteps can be set to 1\n",
    "tsteps = 1\n",
    "batch_size = 25\n",
    "\n",
    "# number of elements ahead that are used to make the prediction\n",
    "lahead = 1\n",
    "\n",
    "\n",
    "def gen_cosine_amp(amp=100, period=25, x0=0, xn=50000, step=1, k=0.0001):\n",
    "    \"\"\"Generates an absolute cosine time series with the amplitude\n",
    "    exponentially decreasing\n",
    "    Arguments:\n",
    "        amp: amplitude of the cosine function\n",
    "        period: period of the cosine function\n",
    "        x0: initial x of the time series\n",
    "        xn: final x of the time series\n",
    "        step: step of the time series discretization\n",
    "        k: exponential rate\n",
    "    \"\"\"\n",
    "    cos = np.zeros(((xn - x0) * step, 1, 1))\n",
    "    for i in range(len(cos)):\n",
    "        idx = x0 + i * step\n",
    "        cos[i, 0, 0] = amp * np.cos(idx / (2 * np.pi * period))\n",
    "        cos[i, 0, 0] = cos[i, 0, 0] * np.exp(-k * idx)\n",
    "    return cos\n",
    "\n",
    "\n",
    "print('Generating Data')\n",
    "cos = gen_cosine_amp()\n",
    "print('Input shape:', cos.shape)\n",
    "\n",
    "expected_output = np.zeros((len(cos), 1))\n",
    "for i in range(len(cos) - lahead):\n",
    "    expected_output[i, 0] = np.mean(cos[i + 1:i + lahead + 1])\n",
    "\n",
    "print('Output shape')\n",
    "print(expected_output.shape)\n",
    "print(\"Sample: \",cos[0], expected_output[0])\n",
    "\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.plot(expected_output)\n",
    "plt.title('Expected')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Model\n",
      "Training\n",
      "Epoch 0 / 25\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/keras/models.py:944: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  warnings.warn('The `nb_epoch` argument in `fit` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "50000/50000 [==============================] - 11s 215us/step - loss: 365.9475\n",
      "Predicting\n",
      "Ploting Results\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAACSCAYAAABG3nGMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztfXmcHVWV//dUvbVf70s6+56QhUUw7IKAKJuI44giI/JjRnGdUWccx3UcHZ0Z0ZlxXEZFwQXcEAQdBWSRHZKQEALZl87S3Vl6X1/3W6ru74+6t+rcevUgmDRJXt/v55NPvz7v9q2ql+7vOfd7zj2XhBAwMDAwMKh8WEf7BgwMDAwMXh0YwjcwMDCYJDCEb2BgYDBJYAjfwMDAYJLAEL6BgYHBJIEhfAMDA4NJAkP4BgYhENFcIhJEFJPf30dE178K1/0XIrp9oq9jMHlhCN/guAUR7SaiMSIaIaKDRPRjIqo+0tcRQlwmhPjJId7PxUf6+gYGRwqG8A2Od1wphKgGcBqAFQA+x98kD+b33MAAhvANKgRCiE4A9wE4kYgeJaKvENFTALIA5hNRHRHdQkT7iaiTiL5MRDYAEJFNRF8noh4iagNwBZ9bzvde9v37iGgzEQ0T0SYiOo2IbgMwG8D/yRXHJ+XYs4joaSIaIKL1RHQBm2ceET0m53kQQPMEf0wGkxyG8A0qAkQ0C8DlANZJ03UAbgRQA2APgB8DKAJYCOBUAG8CoEj8fQDeLO0rALz9Ja5zNYB/AfAeALUA3gKgVwhxHYC9kCsOIcRNRDQDwB8AfBlAI4BPALiLiFrkdD8HsBYe0f8rgAnPExhMbhjCNzjecQ8RDQB4EsBjAP5N2n8shNgohCjCI9vLAXxMCDEqhOgC8N8ArpFj3wHgG0KIdiFEH4B/f4nrvRfATUKIZ4WHHUKIPWXGvhvAvUKIe4UQrhDiQQBrAFxORLMBnA7g80KInBDicQD/92d/CgYGh4DY0b4BA4PDxFuFEA9xAxEBQDszzQEQB7Bfvgd4wY4aMz00vhyBA8AsADsP8d7mALiaiK5ktjiAR+Q1+4UQo6HrzjrEuQ0MXjEM4RtUKngb2HYAOQDNMuIPYz90op39EvO2A1hwCNdUY28TQrwvPJCI5gBoIKIMI/3ZEXMYGBwxGEnHoOIhhNgP4AEA/0lEtURkEdECInq9HHIHgL8joplE1ADgUy8x3Q8BfIKIXisrgBZK8gaAgwDms7G3A7iSiC6RieEUEV1ARDOlDLQGwBeJKEFErwNwJQwMJhCG8A0mC94DIAFgE4B+AHcCmCbf+wGAPwJYD+A5AL8pN4kQ4tcAvgIv4ToM4B54OQLA0/4/JytyPiGEaAdwFYDPAOiGF/H/I4K/u2sBnAmgD8AXAPz0SDyogUE5kDkAxcDAwGBywET4BgYGBpMEhvANDAwMJgkM4RsYGBhMEhjCNzAwMJgkMIRvYGBgMElwTG28am5uFnPnzj3at2FgYGBwXGHt2rU9QoiWlxt3TBH+3LlzsWbNmqN9GwYGBgbHFYjopdqB+DCSjoGBgcEkwaQk/P7RPA4Mjh/t2zAwMDB4VTEpCf+Kbz6Bs/794RL7bSv34JN3ri+x7+3N4sntPa/GrRkYGBhMGCqa8LccGML9Gw6U2PeVie4/f88G3LGmA66rt5s4/2uP4N23rIITst/34n786+83HbkbNjAwMJhAVDThX/qNJ/CB29eWfX+84ETa+7P5SHvXsO4oPvzz53DLk7vQNaTbx/IOntrRU+IgDAwMDI4mKprwFUZzUS3QgYFsIdLeNZyLtO8bGNO+zyS9IqftXSOa/X8e3o6/+uEq/HFj6eri8W3d6BmJnt/AwMBgIjEpCL93JDpiL0e8B0MReyZhAwAODOrj5zRVAQDaunXCV45h+0Hd/ovVe/GeW1fjQ7c/p9kdV+DWJ3fh3hf3v9RjGBgYGBwWKprw47Z3nF13GWIPE75teeO7hnR7U3USALB/UI/wG6oSAIA9vVnNrk7R2907qtkf29rtzR+Shn6xei++9PtN+NDPdEcwNF7Aii8/hL/436ci79/AwMDglaCiCb8u7RFyb1nC1yP/hqo4gFJCrpIR/v5QsrfguACAvX064eeLnr09ZJ/RkJY/p2v7Ww8Ml8wJeEnhnpEc1u0dwFg+yDfkiy7ef9saPLTpYORzGRgYGEShwgnf09jDEX4y5j12OMK3ZGh+MBThF2XydWdIulHEXY7ww3ZF5vsGx/wxQOAIAGj7Awj+gdtaInlv3yj+uPEg3vvTNSX5ie88ssOUkBoYGESiogm/MeNF+HtDkkvC9h57fygJq4i9o18fX5REHdbkFYHv6c2CnxyWk2TeNZwricwBQAj9GrlCQP6j+YDAXTbn0HiQYM6yOXl+wnUFvvbHrXj3LasQxrq9/Vizu6/EbmBgMHlQ0YSvqiK3HRzW7HlJ1JsP6PaCJOTn2we0WnwVyXcOjGGERdTKPlZwcIAlenn0zomd23klUK4YEDiP2Dmxa04hF9i5I+DOIoy/+N+n8fbvPVP2fQMDg8pHRRO+iswf2dqt1dyryHz1rj70jwYRcsF1UZeOoz9bwLMsGi44LprkamHTviHNrnT/tu4gQZtzXNTIks0OtorIM32ekz9/zcl8jN1zjo3JMmLncw6PFyPHvBQe3nwQP3yi7ZDGGhgYHN+oaMLnydH/fXQnAK8E0hXA9LoUAOBHT+/2xxQdgVNn1wMA2npG2TwulkyrAaBX6hQc19ffe5njyBdd1Gc8R5ArlEo6gL7pi5M5J3ke7fPxoyzyL7Cf5dF+OH+gEN4M9jc/WYMv/2Fz5FgDA4PKQsUQ/hPbu0u0+qLr4pwFTQCCWnoV3V9+0jQAQKOM0IUQKLoCMyWBc3IuOgL1sgSTR9GFootMIlYyPld0UJ2UhM8jecf1K35yofH8WgrjBT6GRfgRslL43njugM8/UmYTGh8DANsPDuPhzaYKyMCgknBECJ+IbiWiLiLawGyNRPQgEW2XXxuOxLWiIITAdbesxvlfe0SzFx2BmpRHyIro1ddqaR8vKrtHnIrAeXlkwXVRFbflnJzABaqTpYRfcFz/ulx7LziuP14nfOZc3GhHUDbC1ySdIMLnc44wR8DHcHAHAQBXfecp/M1PzNkEBgaVhCMV4f8YwKUh26cAPCyEWATgYfn9hGBoLCA0HuV7EbUkZEnoStLwiVcSsiLalCR2J5S0TcvIvMjsRdf12ytwci4UBSN2XdLxHUHIrkpFy0X4/H54hF9Ow+eEP6wRfnSEPxoifJUwDjsCAwOD4xdHhPCFEI8DCNf8XQXgJ/L1TwC89UhcKwoFFhU/tTOoQS86AjGLYFvkV90o4ozL0kxHlj6qCD8Zl8Qrxwkh4LjCdwSc8AvFgPDzoSg9ExHJ54uBA+IEnisGkX84wle7hfl1y0X4POHLVwSc5Mv1FcqWsYdXBIPZAlbvMuWdBgbHIyZSw28VQqjmMAcAtEYNIqIbiWgNEa3p7u7+sy7ESyi5fFF0XcRjFmwinzAVwVsWwSKUOIJUTI/wlSMoF/lnIiL/giOQkhF7mNjTfi6B2x3fQeh27iBYYjfPNfzo6h1N6skFpH0oiWOOsVBH0X/9wya84/vPlJWGDAwMjl28Kklb4e1KiuwVLIS4WQixQgixoqXlZc/gjYTDNihxiaPgCMRVhC/HKN60ybOrn1WRdTymR/iKONO+hh9E/nnHDSJ/RtRFbtccQZC05QSeKwR2niMYLzgs8g/m4SRcKLLIP1dO22dSD7PzbqHlCJ/LSgBwz7rOkp81MDA4PjCRhH+QiKYBgPzaNVEX4lE0j/aLjouYbcG2yCdkRewxi2BRIPUo/lUSkCLkoi8BEYgColbXTPiRvO5o0onSFQGv0imGIv9MBLF7EX7pPOMF13dAXM7SI/xg/GA2OsLvHc2x8dH5gnCEr+4vXNVjYGBw7GMiCf93AK6Xr68H8NuJuhDjPC3aL7gCMTs6wrd8YtcdgYr8ixGaf4zZ1deYTZrduy5LwnLCL7pIx6VE4+j2SMIvBFKPnsx1guojvmmrjKQzMKbvEVDgbRl0e+AIyiVtx/J65P/AxgN4x/ee0a5rYGBwbOFIlWX+AsAzAE4gog4i+hsA/wHgjUS0HcDF8vsJASd5JxThxy0Z4avIXI61LWiSji/1WB6Bu2FHUOIgpCOQ8zvMQQgBxCzPQZRG/qWOwKvbL5V0vMg/IsIvun5ZKZ8nm3P8Hb7cEXD5hRMybx7H7bxNBNf5NbkpFOHfeNtarN7dZw6HNzA4hhE7EpMIId5V5q03HIn5Xw6cDDnxugJ+hO+EInaLykT4VnSEH7MIMcsKInwnGB9j8yjiVNcNR/gJ29YcEKAnZwuhSF7tC+DzjBcYsbPxo/ki6qriGM4VNTvfbMUj+f4yUg/vFsolnU7WJoI7At44rm80j1mNVf73z+3tR+9IHm9cFpmzNzAweBVRETtteVfJoMxSJmFtSyNk14/wCTaVOgJF4L7m77DxFvlErwg1HiJ2rvnHLCqRbhIxq8QR5IouUnF9paDsStLRk7wOalJx7TkBr3a+Lu3Z86FyTZVr0KQe1nKZjx8ce/kVAU/m8rYSYc3/2h+sxPt+uqbk/OBN+4awsq0XBgYGrx4qgvD5blY3gngtKo3YbSJYTNtXjsJL2lqsekfX6sNVPTHbQsy2gpWCivAtndhVVU/CJsQjHEEy5kX+hZLIvzTJO14IpJ5iGcLX6/OLqPftwTwDZSL8IUb43AFxR8CJvY8RfpjYlWMIHxt5zc3P4JqbV8LAwODVQ0UQ/mrW2VLxHCfemF268coKRfgqkrcsPTJ3XJ3Aw+NjIQmIR/7cESh7dITvIBmzEGcrC8Ajz3TchkVhDd9BKm4jbhMKXMPPO6hKxDw7J/x80XcEXEoaGCsgFVeRfzAPb8LG75PvaOYrjnKRf7nVgXcNby7uRADgsW3d+O8Ht8HAwODIoyII//WLW/DFtyxHwrb8iF2TXCIi/KD80pvDFdEEXmTjNaknHPm7eq+euK0Tu5JMEjF9ReC4AgVHlNgBL8JPxi0tdwB4jiAVsxG3LS05O5YvIpO0EbMsjcCzeQf1VaURfjZXRKNsCpdnSVid2KMjfD4Pj/B5uSY/pWskF5385T8LAJ/5zYv4n4e3l6wUDAwMDh8VQfgLp1Tj+nPmojoV8yNYLrnYEdKNZREsKyD6Io/8eR0+1/Btelnpxo/8bb1KR0kmibAjkPZkzEbMCiLzguN6LR2k1KPVxucdpOJWSTnoaN5BVcKL/PUe+8VIqSebd1AbIfUMjRf8U8GKZQif3085SYeXffI20TxZPBLq7aMSw2EJyMDA4PBREYSvYBExSScUsTuq/DLQ8Hnk7IYj+fCKwNbHh5O2vnSjduzaYakn2MkbY8nfgPA96UndpyLOVNzWVhaAV5bpSTqWTuC5IjKJGBIxq4TYM8kYLNLr+bMFRvhMohkeL/grAh6Nc22f28tt7NIcQRl7uRYN4SZvt6/cgxt+tFqrCDIwMHhlOCJlmccKbCsgbi6hbDkwjC3yOENejcObqhWZI+ArAt8eUW8PvESEHxofjvCVXUkgSrpRxKuSoumELVcWnt11hZfklYTPHdlo3kFVMlbiCIbHC6hJldrH8kVMqckA0Fs0jOSKqK+Ko2s4VyLppOIWxgtu2cift2jgun1Oi/wD+zArGeVkHm7y9rl7vM7b3SM5TKlJ+XbHFcgVHb+s1cDAoDwqKsK3iVXRMOLl8CUdIk1CcUOOoOiEiV1F/lJyUXX7tr5Rq1wdPifwuB2sFHLMEcSYZKR2uKZVhB8an47biLHkrJq/Oqki/6A6aCRXRE0qrtkBL/KvlpE/dwQjOYcleXWppymT1D7fwJ7Q7g8oH+GHTweLGl/ufN6eYV3z/9w9L+L1X3tUa6lhYGAQjYoi/H2D43ihYwCATrwcfEftlgPDeGCTd6pTkRE+J96yO22VpGNZsPmGrDJ1+JzA+Tyq73xVIqZJN1qEb0VJPZZH4HIeRZCqSketcHJFFwV5UEvM1jd8jeUd3wGFpSFF+DzCHxorolESe7h6R9lzh6Dh8wg/V2bDF0/ycjLniWAA+MXqdnQP57TdwQDw6NYufPX+LTAwMAhQUYQPANsOjgDQifc1s+r99x228YpDl3qsUg0/XL3DHErkTtuQI+AEzlcKgd3SiFeP8IN5sgXlIGwtF6A6ZVYrSUcSqdLCoySdbN5BVdxGIhT5j+aKvrYfjuQbFOFzbX+8gLp0HMmYpRF472getanScwH6ykT4PFHLW0D3jHJHEL1rOEz4H7h9Lb776M6Sss/xgoP2Muf9GhhUOiqO8BV4Fc2Z8xr9ZmYOi9g5uCbfN5rD9oOe5l8MST1BclZP2pYkZ0NVOj6xl0T4RWmPaXZdAgqkG6VtZ0JavbJXJWwtaasIsiYVQ9wiTeoZK8gIP5TkHckVUZuKg0iv2x8aL/hnAIelntoIwu8bzWFqXQpEOuF3j+RB8uPn5aCctLOsaVtnf3RLh67hYDxPKAsh/P0A/NB5wCv7PO+mR7TOogYGkwUVS/gFvzxST8IqXrOJcMnyoL9LUJYJtPeNYZ9sAsZzATySDxyHTuzjYSlGjh/P6xF+ISTRpBM2Ykyi4RG+p/mHCN/fYCU0ux/hS7uqgqlOxhGzLd8RKkIM7sf7Xggv+evV81OJdFOXjpf0AhoaK6I2FUMybusR/kgeTZmkdAQBUe8bGMO8Ji9ZzMfzxmuc8DsY4fPunVwC4mcB8J4/3cPBGMcV+I3s59/WPQqOPb2jeHa3OcnLoLJRUYT/pmWtbOdo9AYohxH73OaM32Mm2HgVSvKyFcFYwfFJqVCm7DPQ5G3NQZSP8IPxcSbRhJO8+WLE/MwR+PakruGPaJIOlTiCcDJ3rODAcYXnIKzAQQghMCwj+fBO4YFsnkX4AfH2jOTQVJ1AMmZr7S/a+7NYMKUaQEiWGRxHc3USRDqxb5OrLf65AHr0zqt6XugY9F/zTWT8aEaeOO4fzePKbz2Jq7+nn+TlugK/fb4Tm/cPIQyTJDY4HlFRhN9ckyw5G1YRvhDeH6myJ2zv6MOSskwLeP/5830JiNfbb+gcQluPFxnyOvkoAlfEXoywc6IOJ3OLURF+hESTkcSuyilH/AhfT8KqFgbVIQlIEV5TJqFJQF0yap5Sk9QcRO9oHq4AGjMJrQVENl/E0HgRrbUpTdJxXYF9A+OY0ZAusXf0j2FecwZEetO2Xb2jmNtUhaq4rUX4m/cPYbbswMkJf0NnQMRc21/fPuC/5o7gQZmgB/TGcT96erf/OakcEAD84tm9+Ogvn8e7frBSc2SPbevG0n++H//82w3gaO/L4h/uWI+n2bnKgBd8PLKlK3LPgWMch8GriIoifN4bh0fgManXO0KURP5BP/ygXJM3VcuxnbBLp9X61xpjkXYUUXsReHA/voMoF/mrck2nVBpKMA1fac++hu/q9qpETCZhdQdRm4rLKh3vuv2S8BszCU3SUYnT1toUEjHLJ+T9A559en1ac3BKX5/ZkNYi+YPD48g7LmY1VCEZDyL/7pEc8kUXs0KOAPBklnnNGaQTMYwVAqLevH8Yp8yqB1EgjQFe62X1f8KJfX3HAOY1Z7TnB4C1e/uxUK4s+lmE/9jWLr/ddBfLIzyw0XMQA9kCth0IHMFPnt6NXNHFbSv3aPP8+32bcddzHfjYL5/Xkto33b8FN/z4WXzoZ8+B44417Vj42Xvx3Ud3avZnd/fhrd95CqtC3UQ7+rP40v9twr4BPS8xXnDw+xf2RbajODg0bjarGfioLMKPKpu0LViK8F3hR8Qx2UVTCNnJUtXDy0PPVeDFN0adu6DJP7ScSzQx5iB4maVWpXMIkTwn5Gy4SsfX6j17JqHvwFVljBlVfhnW8ENVOj0swueSzkGpeU+pTWrVPkoXn16X1nIKvr0+jQSTdNr7PPvsxirNEagKmZmNVUjYll+u2T2cQ89IDkum1aIqEUT4w+MFdA6MYcnUGqRZ5J8rOljfPoBzFzQhEbMwIh2e4wq82DGIsxc0yc/LswshsOPgMM6a3wgAGBwL7G09ozhvcTMAoEs+/3jBwapdvTh7vjdPW09A+Nu7htFSk4QQwAudg/48T27vQXUyhq7hnL/RDwCe2uER9xPbezQZ6qfP7IYQwDcf3q4lkW9fuQfPtw/gk3e9oElH//XgNtz61C78010vgONbf9qOj/x8Hb78h02a/Y417Tjz3x7GLU/u0uxr9/Tj0m88jke26KeO7h8cwyfvXK9JaID3O3r3ug5tR7X/mXaNRMpbRvI6NlFRhG8xolbElpCtDABJ+EzSsWSpiCuCE5xSoe6UiqhSMU9ayfuRNk/OBoSsyglVrxuXRfIxi0r684cdR7gOPxUPX1dG8iGJJusnc6XmH9Lwq5MxxK1gvEpmttQkNcmoi0X4fH5FVNPqU1rOghN+Mmb5jlMR+6zGKk3bb++X9oYqpFiSd6skyKVTazTC393jjV/QkkFVIuaXpe7qGUWu6OLkWfWoTsZ8Yt83MIbRvIOTZtRpjuDgUA6jeQeLW2tQnYz55Zp9o3kMjxdx6qwGWBTsDn6hYxDjBRfXnDELQJA4Hi846Ogfw8VLvYT/3t5Rf56h8SLe/tqZALx+/4BHirt6RrFiTgOAQG4SQqCtexTzWzIYKzh4fm8gQ6nPYk9vVnMcG6WE9cT2Hm0l8uzufgDAr9d0aCuaBzYeAAB877E2rQrru4/uwJYDw/jKvZu16P/mx9twx5oOfP4eXar6xsPb8PFfrcen79YdzS1P7sLF//UYvvh/GzX7vS/ux4n/8kfcGnI0a+TKhUtrgPd/+am7XsDGfYOafWi8gO8/tlN7VsBz9g9uOlhy/Kb6rKOcDV9xTWZMOOET0aVEtJWIdhDRpybyWrYVELUiHq9sMmgEVihyScf7OccVmiavVgRCCN8RJOOWVtY4XnBAJHvghDT8dNwGUelOW3XwuM26WeaLrneguqV69QRJW9VKmUs66izZdKiXzki+KPvxWCWSTjJmyW6cAVH3jOQQtwl16bg2f9dwDsmYhdqU6snjjd8/OI5kzCpZEewfGIdtEVprklqVzt6+LIiAGfW6dKMi/5kNaaTitu/YlCOY25xBOmH7/x+7JKHObc4gk7R9x8ZXENXJmO/Y9vTKeZoyqGF2FaHPb65GXTrun/O7S+ZkFk6pRmMm4RP+Vhnlnj63EY2ZhE/4bd2jEAI4e0ETUnELu+X11DznLWpGzCLs6fO+PziUw1jBwZuWt4IIPoF3DeeQzTu46pQZAID1MtFcdFy0dY/iipOnAYBfOVR0XLT1jOD1i1sAeHkEwPsd3bx/CPNbMsgVXTy1I8gfbOgcQjpuo2ckhzXSKQgh8Hy7d60dXSOaQ3lujzdm1a4+7O0N9iqs3OmtUO7fcECrerpvg+dQfvFsu5afuH3lHmTzDr7x0DZNZvreYzvxfPsA/vm3GzQC/voDW/HLZ9vx8V89rzmg257Zg3+/bws+cPtazf7Tp/fgfT9dg4/9ah04bnlyFy78+qP4l5ADum3lHiz5/P343mO6dHb/hgO44GuP4O51HZr96Z09uP7W1dpnCXj/x5+9+8WSJH7nwBhufnxnSQnw4FgB92/YrzlhwAtGN+4bPCpOaEIJn4hsAN8BcBmAZQDeRUTLJup6FtPk/V46tgW12Va1IgakpGOpCF/4kXwyFkT+jiu0HjhEgSNQm5aIvC6afrVMwfEPLQmXaybjpfaiK/zdwFrztHzgIHjPnILj+nsCNAkoFxx47kkxnn1ovOifjqVJOsM5NGWSINVETs7fPZxDS41n5yuF7uEcptR6dh6x7xscw5SaJGK2pRF758AYWmu8PACXdAayBVQnY0jFbaTjAbHvHxyHRd6Kg0f4aqUwu7EK6biNUWnvkA5iZkMamWTMl7QU0c5pqkJtOu5vPFNlmPNbMqhLx/26fUXUc5szaK5Oolu2btjZNYJMwsa0uhRmNaT96+3s9hzHoinVmN1Yhb3y/pRDmNucwYyGtO941PzLptVhTmOVH70r+6mz6zGzIY0NMrrd3ZtF3nFx0QlT0JRJ+OSyuzeLgiNw5SnTkY7b2ChXEPsGxzE8XsS7z5yDuE14Xq4geka83cfvO28eiIBVuzzSPjA0jp6RHD50wQIAgeMYLzjYuG8Il580FQDwlEw8j+U9+3mLmuEK+CQ4XnDwQscATp1djzxzNOMFB2v29GNxazWGxov+qWa5ooOnd/aiuTqB/YPjWCOdy3jBwSNbupCIWdh2cATP7Q1WQL95ziPi5/YOYNP+YMX0qzXtAIA/bjyIHV3e/4frCvzkmd0APIJXvzdFx8V3/rQDRVfgPx/Y6ttdV+Cm+7dgd28Wn/7Ni74jyxdd/OOvX8Bj27rxgdvX+psEhRD4pztfwM9W7cV1t6z25S0hBD5w21r8271b8I7vB1VeQgjc+NM1+MDtz+Ht333aX5kLIfDB29fiim8+iWt/sOpVbwM+0RH+GQB2CCHahBB5AL8EcNVEXYxX3XBJx7bVpivhE1hck3S8SN6WkottBVJP0RGwKGibrOy5YpjAg8g/5Ufy+sEoCU7sbCUSl/cX7rGjKoVijKgLjutLVJq2ny8GDiLGVwRF3wHxqptxdpoWl4zG8oHD4pF/ruj14AeAZNz26/izOQc1cjctdwRjecc/aJ0nbXNFxy+dVY3YAODA4Biaq728QToe8wlftWquSsSQScb8P5yO/jGk4zaaMgkvks95f2h7erNIxCxMrU2hJhXzD3Np788iYXv2unTcP+1rd+8obIswsyGN5uqkH+EfGBzHtPo0iAgzG6r85PSOrhEQAfOaM5jdWOUTyD4ledWlNEegiH1eSwYLp1T7jmePWrk0ZbBkaq3vCJR+fsLUGsxtzvg/rzYCLm6txqLWamzv8r7fIonw5Jl1mN9cjW1yHuUQzlrQhBNaa7BOEumW/d77F5wwBQtaMn6p6gsdgyi6Am87dSaaMgl/RbC+YwBFV+C6s+agviruE/vz7QMoOAIfeP0C1CRjeGybZ1+3dwD5oouPX7wYqbiFR7d6DuW5PQPI5h187oplSNiWL+s8t6cfGe1RAAAgAElEQVQf2byDr/7lSbAtwsObPfuGziHs7B7FP15yAmyLcO+L+7352wewo2sEf//GxbAI+N36fQCAlW29aO8bw6cuWwKLCLev3AMAeGRrNw4MjeOLb1kOAH6U/9i2brT1jOLjFy9Gvuji5sc9+30b9qNzYAyfuXwJRnNFfPPh7QCAx7f3YPXuPlxz+iz0jubwrT9tl+MP4MXOQbzrjFno7B/DV/6wGQDwx40HsGpXHy5Z3oqtB4fx+Xu8Vce9Lx7AQ5u7cOEJLVi9uw+fu2eD3Cj46hD/RBP+DADt7PsOaZsQ8KobrRqHRew8mcvtHiF7H4faBeoKT/OPSbvF7I4rfOItIXZF1FoSOZhHs7uc8MknxoIjNLsiZO+YRM+eiDE7cxA8MvfmYQ7C7/Pv+isLTux5x/Xv36v/V/MH95MKEXhSOQIWyeeKwX3yyH+84AbjWYTfN5pHc7XXmK0qYfu5kLG8gyp5nCOP/A8MjWNaXQpEhEzS9pPZnf1jmFmfhmURalNBJN8/mkdjJgHLItRXxX0Nf3dvFjMbvER0c3Ug6fRl835/oJkNaXQMjMF1BXZ2j/j5h9mNGezty0IIgf0D46hLx1GViGmEv+3gMKoSNqbVpjC/pRq7ej2NeVdPFjGLML0+hSVTa2ROwsHGfYOwLcLCKdWY25TBbukYNu8fgkXA4tYaLJpSg+2yfPTFzkFYBCydVovFU2t8iWZ9+wCIgBNn1OGUmfVY3zEAIYQfKZ8wtQZnzGvEs7v74LjCrwg6bU4DTpvTgHV7PcJ/dlcfiIAz5jXi9LmN/ulyyn7WvCacOb/JL0VdtasXRMA5C5px1vwmPC5XEE/t6IFFwEVLp+CsBU14ePNBCCHw6LZuxCzCG5dNxZnzGnH/xgMQQuDOte1I2BbefeYcnLOgCb9bvw9CCPxy9V6k4zZuOHcuzlnQjLvWdsBxBX62ai9qUzH8v3Pm4k3LWvGrNe0YLzj4xeq9aKlJ4tozZ+Ptr52FX6/pwIHBcXz/8Z2YWpvChy5cgLeeOgO3rdyDruFx3PrUbsxpqsJ7Xzcf7zx9Fm5fuQc7u0fw1fu2YGZDGl+66kS847Wz8OOnd+P59gH814PbsHBKNb781pPwvvPn45fPtuOPGw/gpvu3YuGUanzn2tPwtxctwl3PdeDWJ3fh6w9sxeLWavzw+tPxdxctxJ1rO/DV+7fiym89WSI5TQSOetKWiG4kojVEtKa7u/uw5oqqulFJUsAryyy6Log8kvYlHVePnG0W+Rcdgbi0E7MXHMEi7RCx+46ASTFMuuH2IpunOhn3iavockIOIvyiIxD3HUpQxllkDoKPzzuBQ4mxCL/A7ex+8oyo46F5lCNIxQKi5isRL5IPVgTJuCJ8O9Keitt+F80xJoVxYh/NedIZ4O0uzsrPZ2S86K8sqlNxXyftz+b9fj+16ZhfX9+fDXr8exq+R/jdQzm01nrtllWEL4RA32je7wA6syGNfNFF90gOO2WiFQBmN6aRzTvoGcljy4EhLJD2OU1VGMgWMDhWwJYDQ1jcWgPLIsxrziBfdLFvcAxrdvdh6bRaxGwLJ0ytgeN6FS/Ptw9gydQapOI25rdkcHAoh6HxAp7d3Y/FrZ79hKnV6BrOoX80j5VtvVjcWoNMMoal02rQOTCGwWwBT+3oweIpNahNxfGa2fUYyBawpzeLR7d2Yem0WtSl4zhjXiOGx4vYemAYf9rahWXTatGYSeC02Q1o6xlF32geD2w6iOXTa1FflcCZ8xqxpzeLA4Pj+P0L+3HyzHrUVcVx7sIm7OnNYm9vFves68TpcxpRVxXH+Yta0NYzirbuEdz1XAfOXdiM2lQcFy+dgt29WbzYOYg713bgwiVTUJ2M4bKTpqGtexRP7ujBXc914oqTp6GuKo6rXjMD7X1juGNNO+55fh/eeuoM1KTiuOaMWegcGMNN92/B/RsP4JozZiMVt3HdWXMwkC3gxtvW4k9bunDdWXMQty186IIFcITAO29+Bivb+nDj+fMRty387UWLUHAELvvGE1jfPoAPXbAAlkX4+MWLkYxZuOS/H8em/UP45KVLkIhZ+MzlS9FUncBbv/MUdnSN4JNyFfLxixfjhNYavP+2tWjrGcXnrliKmG3ho29YhLPnN+FLv9+E3b2j+MKVy2FbhI9dvBhXnDwN33tsJ9p6RnHyzDpMNCaa8DsBzGLfz5Q2H0KIm4UQK4QQK1paWg7rYkpycVyB21fuBRD0ugEAxxEaAdosYi86ASFb2oqAR/iBg3Bc4dvDEX5kJO+4iFuldn4/1akYRnJFb4MYcwTJWCB9eERdqvlzB8GTqkU2v+4I2AqCVemoA9WVPa8cBHMEXIpRxzCq+fPFCEcQC8ovc2z+dDywq86dgFf5pCowxgpFVMncRFXS9ruCjuaKfs6iOhnzCX8gW0CDJPZwhN8gj3OskxG+EAI9Izm0yJVFc00S4wUXI7ki+kYDxzFTbvra25dFW/cIFrZ4tfxzZHuIXT2jeLFzEKfIJn1qk1hb9whe7BjESTO8P2S1N+CFjkGsax/A+bIUdMnUGgDAqrY+rGrrw7kLPfvSaZ79mZ29WLWrF29c5lUGLW717I9v75aywVQ53tuT8MAmT054s0z8njLTu68713bg2d39uPxEb/zpc70S1R8+0YZ1ewfwttO8xfdps73x33x4O17sHMQ7Vnh/wmfM88Z/7p4N2HpwGH91xmwA8O/3ultXYXdvFu85Zw4A4IITvL/nt333aewfHMd7z5sPALh4aStsi/CWbz+F/mweH5T5hCtOmobqZAzX3bIa+aLr5xkuO3EqptWl8E93vYhkzMLfvWEhAOCS5VOxfHotvv94G1qqk/jwBZ797AVNeOOyVjy+rRtLptbgxvO9685qrMKN58/Hnt4szlnQhPecPcf/f/m3vzgRI7ki3nbqDFz9Wu95p9Sm8IPrV2BRaw0+fOECXCk/z7qqOH723rNw2YlT8cW3LMeb5Oefitv4+fvOxPvPn49vX3sqLjhhCgCPH350w+n416uW42fvPdP/vCyL8K1rTsUt16/A7//2dThngWefSEz0qRHPAlhERPPgEf01AK6dqIvZLJJXjbhUtYyy84jdCkX+PrEzrb7gBpIIl3S0FUGJRCPttt7SIabZeWQeEDvg9drnEXg6EfMJMO+4fvuHREi6ibHI3HGFL1VxBxGsFHgkT0EE7rioT3iEmUnY6JTJyrzj+jmCTDLma8u5YtA7n0f4+aLrd9xMxi2/Goc7gqpEzE+qjhVcNGZYhF9wvL4+uSDyr0nGMJgt+D3+Z2U8Yq1Ne3bHFRjI5n3ia61NoWfEi5D7s3ksmerZmzNJ5Isu+rMFdI/kcF61R+xzJFFv7xrBQDaI8Gc1pAF4rRlyRddvCzFLjn9480GMF1y/K+vsRo/Yf/NcJ0bzjv8HPl8S/k+f2Q3HFThvkUeIc5szSNgWvvHQNhRdgbecMh0A/Pv97N0vwhXAm0/27Or5PvrL52ER8BenekR90ow6WAT8450vIGFbePsKr0R0cWs10nEb335kB9JxG391lkd0MxuqMKM+jd+s60RDVRzvPN0jutfMrseUmiR+/PRuzKhP+4S/fHodFk6pxkObD2LptFrfQSyaUo0z5zVi1a4+nLeoGVec5BHj/JZq/OVpM3HXcx1454pZfoXR9Po0/uFNi/E/D23HRy9ehNNmeyWrjZkEvn71yfjfR3fixvPnY5F0bJlkDD94zwr8+OndeNcZszGtLi1/by386IbT8dt1+3DpiVNRJx09EeHb156KZ3b24rQ5DX5ODQA+eckJ+KszvTl4A8V3nj4b7zx9NsI4Z0Ez7vvoeSX2hVOq8d13v7bE3lSdxKcvX1piT8VtXHf23BK7ZRHesLS1xD5RmFDCF0IUiegjAP4IwAZwqxBi48v82J8NHoGft6jZL2kLIn+3TMSuR9Q+sasI37K0eXwNn2njfMOXLvUwQmb2YoSGz++fV+9UJWzkHe/eeY4gKZOtQjos5cjiMZLX9ByHet6aVFC1wh0KlzJyBQeJGi/inVaXxuPbuiGkg1OtjmfUp/HAxoNwXa+6SRF4UyaBXNFF/2heI/aptSn0ZwsYyRVlUtuzL2jJ4O51nRgeL2C8EET4LdVJOK7wShpZEnlhaw2Gc0V0Dox5h7rICH/xlBrkHRe7ekbQzyL80+c2whVeInGASTonyaXzyrZeDI8X/dyBsj+5vQeugL8imFHvEbvaqLSgRRF+GhbBrxo5WUbSi1q90s/bVu6BRfA3gbXUJNFSk8TKtj40Vyf92vy4bWHx1Gps6BzCCa01WD69Vn7+KZwyqx7r2wdw3qJmnCBXAq21KZy3qBlPbO/BX542E3OlI2muTuKq18zA3es68YELFvjEGLMtvO/8+fjOIzvwxbcs93MTAPCxixfh+4+34bNXLPWruZIxG//9ztfgZ6v24MMXLtSKEL5/3Wvx2+f34dozZvu/V0SEm9+zAs/s7MH5i1t86RMAvvb2k/GRixZiblMVOD50wUJ88PULtLEAcOmJ03DpidMQxokz6vD1q08psU+pSeF9MoLnSMZsP8LmUEn4yYoJPxdOCHEvgHsn+joAgrp6IZCK25jX7P1hKqItukKL2LXIX9PYdQ1f2QMN3yNwmzkC3g8/SurhK4iYZfkHo+SLXBoK7r/AHI0ivGzBkRKNdx+phA0hgkNOuOYPeFF5wXH94/8aMwlk8w7GCw7yjkBVIiDkXNHFQLagafXT61MYzTsYGit62r60z2ysQt7xNG1O7Cry3LR/CLmi449XkdqOrhHkii7qZeS/XEodm/YNeYQvHcHJMlJe3zGA0XwRU6XGrohw474hTdJZPsOzr9s7gLFCsOI4ZVYdiIDn9w542r4k8FNm1iNuk78xqbFaEXsa0+pS+D9Z+aGIMZ2wsWRqDdbs6YdFwBIptSRjNs5d6BHvjPq0T2px28KVp0zD7Sv34pLlU/37ISJ85MKF+MLvNuKDFyzw/98B4DOXLcW3/rQDf/+mxf7vGRHhP68+Gfes24d3y6hc4VvvOhVP7+zFRUt0Urvp7Sfjwxcu9PMJCn//xsX4yIUL/f8ThatXzMLVK2YhjHMXNvsrE44FLdX4+zcuLrHXpeORRK1yF1EIk73BxKOiDgLl2rvjCp9AeWsFLWLXCDzQ2NUvoiOUg9CrdPyI2uYRe1BXryJY/WAUFuHb4Qg/KncQOCZ1/aIjtMhcJTOzec8RKAJUBJyTZ8+qeRSB9WfzKLBy0Gl1HqHuGxzzNHxbEX46sLPrzpQSR0d/VqvSmSs17c7+Ma1qSGnO2w4OI1dwkJQriBOne4S/Yd+QtjFtmXQcWw8MexvZpMNbOrUWFinCD/YdLGipRsK2sLLNW9EpKakmFcf0ujTWtfd7ETsj8GXTav0adOUIiAgXL23FbbKkT60IAOCq18zAlvu34Ix5jahNBfYPX7gQnf1j+OjFizQC+8zlS/GaWQ14Y2i5fv05c3HtmbP9z1LhnIXNOCeCYBdOqcEnLjmhxF5flcDlJ5USbNy2/H5BYYTJ3mDyoaII34/MfcIPiBcIiDSI2OGP53blCITQq25KHUqEhu+4iEki0o4+ZJF2eEUQZysC/37cwDEpR+PtIwhWBGnW14c7FLU/YLzgeDt55XhFbL0jec3RtEgC7hnJa5G8T/gDY1r1ziyf8Me0pG2TjJR7fUnHu4/Zsm9OW/eoHG/7180kbOwbGMNYPti/kIrbqE3F0DuSQzZfREZ+bumEjZaaJPb2jiLvuH6VTty2ML8lg3XtXimhsgPeRqu1cpNPAyPwE2fU+btbObFfduJUn/DV5wUAN5w7F9WpWElEfdb8JvzpExcgjKpEzG+zEEaY7A0MXi1U1G8el2h4nTyP8HkdO6/qKTIHwQmWJ0O1ZG6o6qYYFclb/MCUwEFoh6E7ej2/un8u0fBWD161TFDeCXjVR1xKSjPCL7rB/gLVOnpcOoh42HHkHU3SUZH//sFxFJhdadod/WOahl+ViCEdt9E7kvMi+VjwOTdk4p62X3CQYpFmY3UC3cOeNMSTa83VSfSM5pHNBRE+4DkJlTDOMPuM+rS/qUk9J+AlSlXeooFp1yfOCErg6tOB/ewFTf7npco1AfjlfjOkEzQwOB5RURG+loR1XY1ggdIIX9fwmbTCNXxNcvGuI3yHogjN8vvta9U4kvCF0KtlvBO4gqSwivzD9x9O5gb3War5c+lGEe14QSVtlTTkfc0XXW2Hbzq0IkiEHIGycwcRtwkjuaIm9QBetDww5uUCkozY69MJ9Gfz2ooAABozSb8HCSf8puqEF+EXvNO3FKbUpLBGJuMzjNin1QfkXM0ifK4f84j9JE74LMInIvzh716Hh7d0obU2CQODSkJFEX44wg9LJcUSqSSInPUqHVYtE2WXSdtUXBE4/PkLrr6jFlAriOC6vibvljoCdf/8ujzXwJOtvDUEr7dXBO+XobJ6e8BL5nJHxqWhfNH1x+nJX6FpwAnbwrg8HUuzy46ZOSYNAUAm6TVK41IP4FX2qLYCKeYImjJJbD4wBMcVvkMEvINZ+KEuCqoiBYCmsU9jETmXdFSlDaATPuAlmVWi2cCgklBZkg5PenKNnUXOnGADTV6volEEriL8cGuFUodiBddldf62zRxN1MrC1aUYvdVDQOA81+AlW8OOSe38Da0IXLXRTK/eKTgCeusGZZdJXlXeqexFoUX4gEfsaterZrctZPMOhIAW4cdtr0Z/nEk9gEfaqkFVIiT1qD41VQld6lGoYcTOSw25I+BRPZd0uEyUZisLA4NKRmURPiNSruHHOPG6etmkGh8VyQcRchTBRkhGQt+Zq0f4urbv3Y+raey81UO5MtGoJG9Ykio3XhFqwXG1Kh11P6oPDv98bIuQd3RtX82les2rnwc8YleNzHgkn4hZfk6BSzeZZMzflJVgjqM2FffPNsiwCJ9H4zw5q0o9w/bGDLMnoxe0pjzQYLKgMiUdSeC2FYrAQz1wtEjbFahiG0kAmSR1BTIlEhBkhB+axwnX+QfSDXcEpRG+Lg359f8REXtRKxPl85RWGQWOTI/k80VXi/zV/aj+OGoeIIjYvdeBPRmz/UNHwpG/anPAHUQyZvnJUx7hc3JOlLHzaLyWETvX6uuqou18RRAm9if/6UKY0/8MJhMqivB5spXX4fsEKKtZqv2ySe99FZnHrTBhQtvoZGlSDyv79FcQbkk1jjeP7gj4RrBIYvcj89IkMpdWbLYi4Bu1gpWClHRiEddljkP9nDrsRVUfAV6iN4rY4zb5jiAechCqE2V4RaB6hXPC59F7WOrxx7CkbV2ZSJ5X2vCVBR8fxmTecWkwOVFRhG+HNflQS4RiSLoJtHdXq67hPXPKETJvZaCvLKKu64bswUYqTyrRJSMR2vkb3A9keaS+IghLT3ryN6i6ibEqnaik9rgv6egSjTp0RCdw2+9NH47wVSSf0ByE5Sdbk5qko8s+Cpzw03EWyXPCTwav66qiiZ2I8LGLF/kNygwMJjMqi/AVAboCrgiILKxplyZJoyNkdQZuuLWCquoJR/LF0IascuWgfEXAHYFV5n7Cydzw+HByWV2/4LhwRRDBq6+KqGOhFUcg6ehEnY2SbmzyiT1WIul4kXw8VNWjWlbzahxO7Alb1/aD19ERO5+n/iUi+Y9dXNoKwMBgMqKiCL+EAEOEH0gZEZG5W6qBu6GyRl4e6Y0vrdJxRGmOoOhEJ21L9wUgdN1SSYffT7hKJyxJqYhdSTpqvvFQktQKRfhqHsBzCtERvuW3Ko5r2n7QOjkRcgTBmGhiL6fh8zJLTvhck69K2Pib183zWwUbGBiUoqIIX6vSiYjAy3W/DCc39XJK3lrBu46qey+tuvEifEWgL1cdVAzt/CUtORvdh59r8moe1SI5PH8upMkru6qK4UQds4LTtrSI3bb8FUFYookqy0zGol/zypxyWn05SYcnasM18wpEhM+/ecKOSzYwqAhUFOHr/e1LyxrVRqdwRO2EVgTcQRRYJE/h6pdQBO7nAiJ67/AIvJxWz1cWWsTOIv8CcxBqnlxIe7dCET4/GAVAkGxlBGtbFET43BHY5BN7+Qg/Onrn43ktPR/DK2q4Q+GRP4/2qxIx/PW58/yOlQYGBoeOiiJ8vsEq3NwM8AiZEyZP2kY1T1MRfkl9u6qTt0rHuyIgXK6l8+tZbHxU8lRp3WFHE9bkw5F8uEdQrqBH4OHyS16No0X4lh7JD47l/DHcrurktTLO0GYrBV5aybV3rUqH2TnJh5uN/fOVJpI3MPhzUFmEX7LRKSzdIDKiVknSyK6VTOoJn40b3plbKAptnCL2gMD1iD3nR9SK2BE9XkXyRV2T9+dR40MrAnVerF+loyQdP8IPiNqL8Evr8GNMuglH+P7rMpKOFuHHy0T4yegIv6EqgVTcwlnzm2BgYHBkUFGEz8smHVeURNoqwo/U8N2IZKus0gknTx25Q7aE2B2PGG07bNelEgoRePi6PrG/TCRfKuno9jCBExFiVlA/zyP5mG1FSjoJm5AtRBC+ra8OFHiUzsdUlYnkeQVO2Fk88cmLUJuuqF9RA4OjCuvlh5QHEV1NRBuJyCWiFaH3Pk1EO4hoKxFdcni3eWjQJRe3pOwwfLBISXvkck3MQvPkiwIiQlrJywjfDjkan5BDZZZ+UjVM4KEIX0X+46EVQUlylh25yOfhkohtEUvahjX8CEfAjm/Uqm54ArdMBQ53EOlEmaRtmZ22gNcKmc9nYGBweDjc8GkDgLcB+D43EtEyeAeWLwcwHcBDRLRYCOEc5vVeEjwJ67oBIVosYo+SYoKdtnq9elBOGYqci7okoubP+1q9Lq2oyD+8Yzdf1CNqq4wjCDuIcslZ33EoSSeiGicuu1wC8Ddwec9M/nNpG6/KSDRcDuK5gGSZVQCXblJl5B3uFAwMDI48DivCF0JsFkJsjXjrKgC/FELkhBC7AOwAcMbhXOtQwOvS86xVgtbiQKt+UcQuk6EvQ8jhZGgiRMiFok74VKLhhyWXEFFLDs2FtPfwSqFchM/PzI2a3xsTRPi6pEOBts+j91CiVoFvkuLkX07D5zthk7HoXzsTzRsYTCwOi/BfAjMAtLPvO6RtQlGidcfCxOjq9jLEq4gwLH2EWxCUVNeEI/yQROOvOKRd1bcrTTt8n+EdvuENU+H7SYQcVi5i5yzX8PXeOJZP+OFqnMjX2uogWqvnhF+v7ZDVif39r5+P2Y2mr42BwUTjZSUdInoIQNT2xc8KIX57uDdARDcCuBEAZs+efVhzKSIdCyUZ7ZDdJ+oyO08Vr43l9a6PpRuX9IhdEWlKRqqlkbm+UlDXVZGtn8wt42jGCrrDCktA4RYKQTM0nZzHI5qexS0rUtvn8k45ueZQNHnel57/LAB8+rKl+PRlS2FgYDCxeFnCF0Jc/GfM2wlgFvt+prRFzX8zgJsBYMWKFYfVrNYnxnx0BB6WLOyQJh+UTXrvZ0MtBcI9Z8IEPiK7QaakFq0IWW1QUgdtqCTsmE/40RF+PFSWGe4bX67pmR2q3uE6fKxM98uYTX5dffhAE4Vydq69c62eOxS+W9ZiDsjAwODVw0RJOr8DcA0RJYloHoBFAFZP0LV8KAJU7XnVph4igkW8/jzaEYSJPegDHx3JB5G/d33Vc0YRu+I1RbBKyrBD86gI3y4T4SsH4dtDK5eSjVd2yJFpEX60pMNlH34CVLlNVVoFTrxMb5xQFZCBgcHRxeGWZf4FEXUAOBvAH4jojwAghNgI4A4AmwDcD+DDE12hAwSE/NX7twAAZjYE55nGLAsHhsYBAC3VnrygSEgdpTe11jsIW0Wg3SPeDtN6KUco8usd8Y7kU4281HWHfEejSzq+A5JkaIUcjdLwfWIPJXnDElDQ9KzM+DK5CTUmvIIAdNknlSij27PxVWWOCOQ7ZM1JUgYGxxYOqyxTCHE3gLvLvPcVAF85nPlfKcJRJN+laVuEx7Z1AwBaJbErYtx2cAQAMKfJSxyqiHhvr+cIlONQxH5w2HMc6hxVdd290nFMlwdnK2Lf1TMKAJglD9xQ83QNew6lKTRPOemmJDdxiDkCjfDLJGR5JM8JvFzSlideuURTXeYYQQMDg6OPivrrtFlE+aMbTteISJElACxu9RpvKelj0/4hAMEJSIqQtx4cBgDMkISviHTzviEQAQtaqrXx2w4OI26TT/jqfnZ0jaAqYaO1NinHe/fROTAGAJhal9LmUSsFFS0HyWiP2JPhKqNw3X6I8PVWCdGv42UknfBpVgq8GoeDJ23D+NerlvuHoBgYGLz6qCjCZ9WBOHNeY9lxSlqxQ5KDilpVFNw9nMOSqTV+P3Zlb+sZxfzmTDCPtO/sHsXy6bUlVTS7e7NYPr3WlzjU+H0DY6hJxXzyVHyqJCAVLZfmApSkI3MBOT35G4x3tOfyniF6M5R/0Llt6WWcrA8Pd6DpRLQamCnjCADgurPnln3PwMBg4jFRSdujAi7plItAH/vHCyLHK1kFCCJ/ADh3YbP/mhPeyTPrIufhdu6A5jVnAjsFhK/yBkCgeQ+GInzuIIBSSWpoXM8REBFsi/zDxKvKROw17GAR5cx4J0tvvPd9eBcsP3aQwyRnDQyOXVQW4R9CknBOU0C8nJxWfuYNkfNce2awN4Dr3xctbfVfc45bNj3aEcyX8o833rO7IpBz+HX7szLCD0k6Hf1jqEnGfKJWDkgdNcgrZGwKCJ+TtbqnmlRMu79yxK6SueFdsKYNgoHB8YcKk3ReWXQZRXhh+4IIogaAi5ZMibQvZYdlc8cxvzna0bSyCF/Ns6tnFLMa0z7JqucayRVxQmtNyTx+hB8mdsf7yjdGqeesC50Bq/T88MpIlYCG2yFUvQTh/+1FCzG/JVP2fQMDg6ODiiL8l4rwd/7b5ZH2TMLGaXMaNFusjOPgRM31bymADqYAAAjqSURBVG5fxAiZlyVyAuTTq8odQJeATprBVgpsnmn1pSuC4fEikjFde1f3VBW3tftQzxY+KlBp++G2BzE/wtcJPx0vT/j/8KYTyr5nYGBw9FBRhM+rTsIopy1v/NKlEfN45PaXp808pDl4hM8jZz5+TiMjfGZfMTdwNnyeE2dE5wKm1aUj7c3Vych7DUsvahdxOMKP+xG+Pl7JRFVJI+kYGBzvqCjCr0nF8YUrl+HSE6Na/7wy7P6PK0ps8TIOpZwj4JE5P8iDE3u55C+P8Pn46RGaP6DnAvhctWWIvZTwPUfQXJ3Q7CpxXBVK0tan4zh7fhNuOHcuDAwMjg9UFOEDwA3nzpuwudNxGxcvbcW7z9KbvJWN/FkEzmUVTtS8UsbSJKDqyPGqxj983XKE35jRCTxWRsNXEXtLjb5SUHmEcEQfsy384sazYGBgcPyg4gh/IkFE+OH1K0rsiqgzIVK0yuQUyqUauN/gZaJcAuIaPhGBCBBCj/yBwEk0hQm/TOSvNHre1RIA1JVNEtbA4PiHIfwjgIaqOJZMrSlJVpaVeg7BrrUuYMOnMw0f8Mge0LV9PldTSKIpyJ794Qhf9dcJR/LnLGzC9WfPwQdevyDyng0MDI4fGMI/AojZFu7/2PkldhXJ14T6y5SP/F/eEYSlG4VZoQNE9g96m7SWTK3V7KpTZpjwRyM2aQFemeYXrzox8poGBgbHFwzhTyBqkh6pfuISPfJX2n44CXwoVUDhskmFWY16hK962y+cUq3Zu2Tjt9Ya3XHccO5cbNg3hCtPmR45v4GBwfEPQ/gTiHTCjqz2UZjbpOvi5faNHUq7Al7Pz8FbRANAe58X+Yc1+YVTavDbD5/7stcxMDA4flFRrRWOFzRlkrjh3Ln41rWnavZyUk85O0emTFti3qsHAL5w5TK01ibNGbIGBpMQJsI/CrAtwheuXF5iV7Xw7zpDL/v8c/qRndBag60Hh7XdtwDwttNm4m2hDWUGBgaTA4bwjyHYFmHDFy8pSZwSERozCVwf0V54TlMVFk2pKbH/6v1noeAc1hHBBgYGFQYS4tghhRUrVog1a9Yc7dswMDAwOK5ARGuFEKWbhEI43DNtv0ZEW4joBSK6m4jq2XufJqIdRLSViC45nOsYGBgYGBw+Djdp+yCAE4UQJwPYBuDTAEBEywBcA2A5gEsB/C8RmW5bBgYGBkcRh0X4QogHhBDqkNKVAFQ28CoAvxRC5IQQuwDsAHDG4VzLwMDAwODwcCSTtn8N4Ffy9Qx4DkChQ9pKQEQ3ArhRfjtCRFv/zOs3A+j5M3/2eIV55skB88yTA4fzzHMOZdDLEj4RPQQgqt/wZ4UQv5VjPgugCOBnr+QOAUAIcTOAm1/pz4VBRGsOJWlRSTDPPDlgnnly4NV45pclfCHExS/1PhH9PwBvBvAGEZT8dAKYxYbNlDYDAwMDg6OEw63SuRTAJwG8RQiRZW/9DsA1RJQkonkAFgFYfTjXMjAwMDA4PByuhv9tAEkAD8pOjyuFEB8QQmwkojsAbIIn9XxYCOEc5rVeDoctCx2HMM88OWCeeXJgwp/5mNp4ZWBgYGAwcTDN0wwMDAwmCSqC8InoUrmjdwcRfepo388rBRHdSkRdRLSB2RqJ6EEi2i6/Nkg7EdE35bO+QESnsZ+5Xo7fTkTXM/triehF+TPfpHInrbxKIKJZRPQIEW0ioo1E9FFpr+RnThHRaiJaL5/5i9I+j4hWyfv8FRElpD0pv98h35/L5orcxX6s/h0QkU1E64jo9/L7in5mItotf/eeJ6I10nZs/G4LIY7rfwBsADsBzAeQALAewLKjfV+v8BnOB3AagA3MdhOAT8nXnwLwVfn6cgD3wTtu9iwAq6S9EUCb/NogXzfI91bLsSR/9rKj/LzTAJwmX9fA26W9rMKfmQBUy9dxAKvk/d0B4Bpp/x6AD8rXHwLwPfn6GgC/kq+Xyd/xJIB58nffPpb/DgD8PYCfA/i9/L6inxnAbgDNIdsx8btdCRH+GQB2CCHahBB5AL+Et9P3uIEQ4nEAfSHzVQB+Il//BMBbmf2nwsNKAPVENA3AJQAeFEL0CSH64bW9uFS+VyuEWCm835afsrmOCoQQ+4UQz8nXwwA2w9uYV8nPLIQQI/LbuPwnAFwE4E5pDz+z+izuBPAGGcmV28V+TP4dENFMAFcA+KH8nlDhz1wGx8TvdiUQ/gwA7ez7srt6jzO0CiH2y9cHALTK1+We96XsHRH2YwJy2X4qvIi3op9ZShvPA+iC9we8E8CACNqT8Pv0n02+PwigCa/8szja+Aa80m1Xft+Eyn9mAeABIlpLXicB4Bj53Tb98I8DCCEEEVVcORURVQO4C8DHhBBDXIqsxGcWXmnya8jrKns3gCVH+ZYmFET0ZgBdQoi1RHTB0b6fVxGvE0J0EtEUeCXrW/ibR/N3uxIi/Erd1XtQLt8gv3ZJe7nnfSn7zAj7UQURxeGR/c+EEL+R5op+ZgUhxACARwCcDW8JrwIvfp/+s8n36wD04pV/FkcT5wJ4CxHthie3XATgf1DZzwwhRKf82gXPsZ+BY+V3+2gnOA73H7xVShu8ZI5K3Cw/2vf1ZzzHXOhJ269BT/LcJF9fAT3Js1oESZ5d8BI8DfJ1o4hO8lx+lJ+V4GmP3wjZK/mZWwDUy9dpAE/Aa0nya+gJzA/J1x+GnsC8Q75eDj2B2QYveXlM/x0AuABB0rZinxlABkANe/00vBbxx8Tv9lH/RThCH/Ll8Co9dsJr6nbU7+kV3v8vAOwHUICnyf0NPO3yYQDbATzE/rMJwHfks74IYAWb56/hJbR2ALiB2VcA2CB/5tuQG+6O4vO+Dp7O+QKA5+W/yyv8mU8GsE4+8wYA/yzt8+Uf8A5JhElpT8nvd8j357O5PiufaytYhcax/HcAnfAr9pnls62X/zaqezpWfrfNTlsDAwODSYJK0PANDAwMDA4BhvANDAwMJgkM4RsYGBhMEhjCNzAwMJgkMIRvYGBgMElgCN/AwMBgksAQvoGBgcEkgSF8AwMDg0mC/w+2XN8sMD4iygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd27b691dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 / 25\n",
      "Epoch 1/1\n",
      "32550/50000 [==================>...........] - ETA: 3s - loss: 314.3629"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-3cb6a27d35a1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     22\u001b[0m               \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m               \u001b[0mnb_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m               shuffle=False)\n\u001b[0m\u001b[1;32m     25\u001b[0m     \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset_states\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/models.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m    963\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 965\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m    966\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    967\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1667\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1668\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1669\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1670\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1671\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[0;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m   1204\u001b[0m                         \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1205\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1206\u001b[0;31m                     \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1207\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1208\u001b[0m                         \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "epochs = 25\n",
    "\n",
    "print('Creating Model')\n",
    "model = Sequential()\n",
    "model.add(LSTM(50,\n",
    "               batch_input_shape=(batch_size, tsteps, 1),\n",
    "               return_sequences=True,\n",
    "               stateful=True))\n",
    "model.add(LSTM(50,\n",
    "               batch_input_shape=(batch_size, tsteps, 1),\n",
    "               return_sequences=False,\n",
    "               stateful=True))\n",
    "model.add(Dense(1))\n",
    "model.compile(loss='mse', optimizer='rmsprop')\n",
    "\n",
    "print('Training')\n",
    "for i in range(epochs):\n",
    "    print('Epoch', i, '/', epochs)\n",
    "    model.fit(cos,\n",
    "              expected_output,\n",
    "              batch_size=batch_size,\n",
    "              verbose=1,\n",
    "              nb_epoch=1,\n",
    "              shuffle=False)\n",
    "    model.reset_states()\n",
    "\n",
    "    print('Predicting')\n",
    "    predicted_output = model.predict(cos, batch_size=batch_size)\n",
    "\n",
    "    print('Ploting Results')\n",
    "    plt.subplot(2, 1, 1)\n",
    "    plt.plot(predicted_output)\n",
    "    plt.title('Predicted')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Keras online demo\n",
    "\n",
    "https://transcranial.github.io/keras-js/#/"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "30px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}