3_recurrent_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent Neural Networks (RNN)\n",
    "====================\n",
    "\n",
    "In this exercise we will work with Recurrent Neural Networks (RNN). A RNN is class of neural networks where the output not only depends on the current input but also on previous inputs along a given input sequence. This allows to exhibit temporal dynamic behaviour and contextual information in a sequence. Common applications for RNN are:\n",
    "\n",
    "- time series analysis\n",
    "- speech recognition\n",
    "- machine translation\n",
    "- image captioning\n",
    "\n",
    "\n",
    "Goal of this exercise\n",
    "========\n",
    "\n",
    "This exercise notebook should help you to experiment how recurrent neural networks are implemented, trained, and used for computer vision problems. Therefore, this notebook is structured as follows:\n",
    "1. Implement your own simple RNN class in Pytorch.\n",
    "2. Explore the backpropagation of the gradients in the RNN and discuss the vanishing gradient problem.\n",
    "3. Implement your own LSTM (Long-Short Term Memory) Network and show that this architecture improves the vanishing gradient problem.\n",
    "4. Build a RNN classifier for the MNIST dataset and train your model.\n",
    "5. Tune the hyperparameters of your model and submit your best model to the server to get bonus points.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import os\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using python:  3.7.5\n",
      "Using torch version:  1.3.0+cu92\n",
      "Using device:  cuda:0\n"
     ]
    }
   ],
   "source": [
    "import platform\n",
    "print('Using python: ', platform.python_version())\n",
    "print('Using torch version: ', torch.__version__)\n",
    "print('Using device: ', device)\n",
    "# Machine: 2015 13\" Macbook Pro, i5 dual core"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAAlCAYAAACAjKdZAAAUzUlEQVR4Ae2dW+hFT1XHV9lFsovipazACo0CTbEiqEiDeuj+EobiQ4WVaNBDqNBDF/CpFCVQ0BQ1utAFutdDQRkWiOIVCXowNSMqKitT1C7y+bm/P77/+c/sPWefOXPmnDMDh9l79lzWfGetNWvWzN4nYoaJwPkR+LSI+LbzkzEpmAhMBCYCE4GJQBGBT59zVRGb+WAgBD4vIv5/+WFgzTARmAhMBCYCE4EREfgcm68+u0RgOpF9RkR8VkQ8LCIeERGPiYj3RMS/lyqY6VeHABb5Z0YEDMTvsRHxjxHxDyfoKfX/91Lv50fEfyVtiD8xvHKB57lnpXTq2HpGnrTOtTI5uk6VNuXzVMj2qRfZeqjpV3j+YxHx7j7NX2QrW7K39VydJh9hVNleyJvRBgIjyNCXRMQHFzqZK/9njebvMUsM5tPvBWuF5rOrQgCm1bh7/LMn6qXa+LJM/Y8u0KIyilPalL4W5xYKW+3lymTIPlnSlM+TQdulYl/pOm+em6+6dH5nIyzuHavS9RaGTISlsqRvld9J/izWGIGRZOhJC09leech1nEy/EtEvCsivtHSfyjjSbDH8/KKEEDJwAN4p77W+vXHEfHXdt/i8lVLGz8XEb+bqfB/I+I/l3afnnn+6oj4jYh4Q8Kf71/KPCEiHp6U+8uI+PmI+M3FE+uPaY8fyvyL/EFEvDAifjEi3puk97yd8tkT7fZtwVt4Z/80Ipw3f/oEstWe+vPUiCfgwxnMRM0fRsRrFpn+gBIz8ccX2X7K4jFUlg9FBPijE9bKK/+Mz4vASDL0zxHxxRHxTcsuyF/UQIMSl4WPF2OGegTwoHxdffZhc9IH8UBrr6WvIOWiXwPieUYLNLFy2Qp+dosyP7hVwJ6/Y2nvjZY20uWUz5FG43BaWFRItnLe2sNrvP4SqffqD3Z02XVC6uneUd0sckYERpAh3+Fhx2M1OPMxwcxQj4D2+1sbIvUUtMvpxk/r/sgwqDV25HbVZFRjjGksVOYZB0BDGWgcMUz5bD8qLCK+sn21xRrFk8SXvHDtiVsqzyy2Dg3oGzCfRtWhyG3n78kLUDOKDGnR/3dbEPHau4jew7xb9V/zcw4Xg11rQ+QcmJ3KsPpm46/aScVpAd+aVX6qiGvHRILCCnnEMOWz7aho1dnLkPZzIpvKuG1Xm9bWG7dUng/1Jgv3S8a86QA2rKw3L2gsmQvOPZ6a86EFB8BdyE1sP6yHEcHZmhnqEfjq+qw3m1MufM5I/d8JUWDv24PePvS09JrXZ18RES8d2GM15TMdtePuUYgEzub1CH5+9ZU9GjxRG71xo713Wl++wK5rLnVG9GtqMs88ByHQmxdGkiHOATKXEX5/ibMRIOmXM7yyhWbiHQKsosCu1jsyMmzuJWrVHz8nsbknbeCkb+xB21pIV7eMCXvyW+HXlvEbme8lm8Qj07mF9SjPxSu9jj2Ixxi/ntuPrfHujRv0syhz/oeGmiAvbys9VtPmLeXpzQujyZDPldnzv35+49wutktjzFMe9j4HFs4srRSSttlQjocECa6UKp8eWAvQq7yKt/hZ/T3kLNYaDad4NuWzParirV6GlfiRmG2ESw29cQMn1x/gV7OwII8wv1SsR6e7Ny9oPEeSIdF0N3+kgv0NNoJyU8OYj48IPmbH6+98LDL9kKMVu8lL9lbfYj3/QruuvcQr810R8cTlFWNe5/yt5fMHa3XA1Hw2wz/s+rdWgDH+iobjd0x9P7rQJdepkXnQ5Vctq9dcIVYMvD79/Ij4joj47iXTf+QyW9qblms+xTBqmPI56sjU0YVhrPC+5cOCbD/Lc8VHB3ud9RIdlxT/fUIsHy/e+nDxryxlvjwpO9ot+v9xywdj4Y10joVHNAfz2ZfVj1KO1rmG9IwqQ3zK51uWeedBc0jqYktXCLLKWP0zod96wMUMFsJlLS69iYIhoNf7Kc8Y+D31Z92Ly4ot16betvNXUj0fyrtUp8ZUHhzKwQeElxT6WlOfVjXUt+VxWpq7j3zVSfk1D5q2CygM5ur3msdKB+o1wd03PNjFlM/2AyK+7OGx0pYUPMlY5uQJWSr+VUb77u+usSduItJ1EhhuvcSCPAtr1TFazFjn5hCdRUX3wRPSY4pH8qz35IVRZchtpQfxmAbNYwrIs+VnZA59K+NBjV1BAtYzypGJ3o0hsCGNiZ0fedgqTAO4Cuv7NwqWTOm5opwSgaFpR8aE6nJ6YEQF364k71pwJUYf1L+99Xl/cn1Zo4VnrnygJxeol36JRmd20lFSaZBSuAR+1vh6POUzHdHD7jX+PQwrN4w1huJV6NAZTZ5xP3LoiZtw8K1wMIL314Iwzsn9Wrlez1jcikbpRAwtpfkiG/3p8wV5RulXT14YVYZkxDMujNV9ENO6dexuN2X0jvUeWNqj/VP9ag44C4c01jdSAHZL4FXWJ376lAY3Rqh3LcjrQj79cuOH0aXnqTHn9bthpfzH1McKS/XIUPf2tq5l2FFHyfsE7/JTcHwpl+NX4ZHrm+oZIYY++jCyfI6A06E09JwUxP+KU57ziRa+HDn0xE04qE3ht4aR5Bq9OGLwvnDtIdVb6GKC+qT+5/TZkrVrpL70WJyo74pHkSGXXRnJd4OQ/heZBjMdIXdf9x7Y1NAQuC3jvX1yYVgTeMfTjVT6kGvbDYrSmFAnRpLjkPOQkY8VsvKt0ZkaVlpZO/2H1Of45PqZ1pveOw45AVa/HCM35uhzqsC0OoSnRw+XIJ+jY5ijr9ekIMNYspf7ThpyoefEI4deuKUYOD45PUB+7ayUnqd1nuNeC3HXV6LDdaV70j19JP7oxQsjy5AwYFye4ROcDhYzuPx/W+lQYGmCFVOcMuZ/7B62HOLjIF/rH3Wf8ttKKTZvTxJywuIHvfkn7VL4N3vAf2C91e798m/8pvKa+v6skHdPfYWqVpPfvPIUpuY/2PjvMOfbv0rKPCq5/+3lnv8CPFXAwMXohcZjwiXI5zH9oyyrvj3ezGPb7VHeXzxAv7pnVe27PiYtvcejjrEgr6XO4aj8LcQ137J62wKEf++oNTbHyvXrIoIDz66vROMP6CIiftKumQv4715+a3OBFbmqyxYy1AOQx7kS09tTNPzildafbM9qjRBWEEzOOcPBqqu6/EhVrsvIhKeED1fy1t7LGuFDz8G6ZWhZH2+81PKN98EVKjzIpKN6fmbJ+P1eIHON4YxxTmClCM9/r9WzPGoWMQHqj6DBMOelqG3sUuSztj+ej7Hkz3AZR4yO0hk6L7N1jZHmfzJfyv/I5QEfnKwtw5/BHqqH/MOuuW1/yBA/l2jVtod4iom5dajFoBduaf9YYGkOYsvF9QB58eySzlvBh45R2lbp/li5lsH8I4UGeLtM4d26WN4EfL3db10eO++OxgstZGgLs6bP5TrF8CmdX6FB34qrdbNqu2tt26lpZ85Umbtp9/YVRsYjyMrUD2szLvwesHeb9NO37ta2tjzfGp2er0V9wmeNv5IuPeBW5YWFlBOYkcbzNICX8qf4adWflml1D33eNtd7vVbXKJ948TCgci9eHDsGrqfSMWh1L/6rpdXb9QWtl3eZI3+uDeeFNX3g9dZej4hbSnuqBxxLrsEN2T5VaCnXORp1POHYfhw7747IC61kKIf7sWkP2AoUU/oBP32/KtcQqwCFl+tiJaaxZy7Pa/5SZKWqq37EKuuXzbtBZ1m5Ixy/tHwf4xAA+M5Jy9Cyvq3vSZXoLq0+/2QpkOPbDySVsRWIxwzjlVX/Kb9tgxJoFa5RPvn7p5+ICL4fhtdw9e8gDgTyXyPi1w8oIx1VW4ZvDG15l7x5JikFfb9K9x4/1W8KbXyf5Un52x7tuhwNt1wn3pUkfuki0yT/zvKMbwGeKrSU6xyNfMtK4bW6ODBuMe+OxgstZehAOA/Ofn/kxFeNa6sgtxjxFGwFX4Gt1btVzyU895VUzhOU81aQlnqm0jf1dMgR7NcwdKxz7QvDc+UTPntXk043WMB/SnPDQ/0kBl/nWeFCmh8K9TItrzHg6C+/Eo017d2CfOIBZ1w0RjW4tMpDu7Ue+D1tIrfiw7VtTmGwRo8+yXBKemv7uEZnbR2H5pPMC099e04v7xzzZnctLa3kOtce/K++lV5AypXzNMdobc7wMsden5oXWsrQsX3Nlfe55gXyWPn5jdIqiMFSYNVV8iAoD/G3202LPznFDft7VmfrS1aiz2pd6WIA8FVlznz4SpcD535e4mmZtmsM2EyxYZPoL+PoOOwh9jER8efLalVfTU/rSY1ZeFAfT+Wr7KcOHPhn6+bYcCnyeWw/z1E+5ZFT0PCdVmnpjAwyobNDZM/tCECrzt/8qtV5jsseuOX6lS7M8OD9QkTIk/XcXKHGaa3kOkeWH2nw81WeF+wxZEqh9bxbakfpPXihlQyJ5pPGGFau+DkgXJrwOFyq8CJdLCtM/kJE4GoF8fHlMKqyPmG5YEsw9yaE8q3FHJj0SWYt795nz17BYG+duXJ4MGRU8fxbc5kKaRxi5dzTjxWej5jMwXGFNaWgPGmcvgWDUcUqBgOrFOBlDo0L559aJi+2WWsWBqV6e6Zfknz2xOVS2/pEgfD0DbY3ZPLdbzFEBAYavPGciOAvtNiuZ9G5V7dmmhsy6aMJVcwnMka+fsPgSIoOd8t8LA8TeutjGQrxYvH3aeki/ZTzboaMsyYdI0MQDsYssD932bn4o2XOJ/3HI+KflkV76c36XOdl//Dszvnj38cpueFlIctFKU8XlZAm92vuYJ/KKE5XHDki19I43IcX5xQ/79caDbln2uqinymOcs2CjwKYCZO17QGeKZ+EjvpTHNVGrn21SXyufMUv0zpxK9d+qFN4iO9Wit1/MV5liF0I1sqO8OzS5HMvZtoGS2Vnb3215aTbTrm15rqhxHt+JKD0dyV+LEB4EbuOkJFR2/+9+XrgVqIN3efyzHWPrf0SPYemo8vgc34+57DFqH6V5gT4JH0BqMe8u9bHHrzQSoZ0qB+50bY6mEuGPI28tcHn1bt5uub8hr8h4I1RAUTJWoYIKQ6fSPVcz2qJvaR8eKAkFKlyExZuWEkxUibN7/12JSLDivxrhtVafc4APfNhCAuf9ByZ97d0LeFVHcSulErlHGfKoLwuKdyKfGqcrtGwkvyXeNaN53TSdF5F6bs+kF4lD54MyUYPPSt5ZNx6B/GK+kuMsXIpwel2fvd+5XSz5lv4JQ0ac+c18YeepWVa3ffgBe9XTu/XyJB0qc8/SmNMhLnGZ00WU+y8/Tv6VAmxT/xeUANKHl9NIeTpBK9yvroq1au81xD7V2HTAdHgeT8RKGFfgyF5tYohTsvo8Cb51lZvng+6SsHztaiPdtRfVyal9tP0dFWms1JpvvTelVWKWZp3xHthRlySo2uQT43THt44Ztx6TArOu6lh74vWtTEWneIHFkgefGFX4hPPf+y16GHcegd5F4RFrS7oTWeuPbZvRTex+N1lmPS0T85DuXqVdo55twcveP/3yJDmZ7dfwExzM5graHzciaRnpVg8eT/3q5L7hExJLDDlY2UEkHKZ+RkQLypFeQ7Bczp6XgsTsJIRpL9VSQXFMSU/OOmgOqsvbRViCWMQCH+saq45Y8U4EMu9qTyqjzr0NlopHzSTD3pa15diLzr38oT3L627dC+8KJtORqUyo6T7pHsu+cTAhof2/lCCdyu4DVClLzTRbGRv9rjHpACx0gPwoXSmL14Y3zWDyHkh1SXUL8Xea/HQC7fcQLtM9+pvjo49aW4gwAvcyxuDLvZx1HzgfLIlS5KjvTp2T5968cIxMqSyaf80t/qxEjCWjKb5S/fIL+Opef9+wqbhteCuLirgJ1djWk5Ak6e3okxp6X0v40EYEZcsX4QKg8fz+rWMIleqPJenyYUUBmFw+XEthkEoCao3zad0xrJ1fUvT95FvV2wpiPtCdiGloT7Zo+Il/OeYFTMO+MBXt+eQT5dj8cmeuKQnHHKNbW99oT72mIjcm+A45gwlx4ZrL4ucpkGKXbohfd76viduKe1uaKAbLy1oknceYPzAlOBeFOWBP3PjvhS5izQmlOkpR2p3dBkSvsLMj6cwN+0Nfv4Xz+NdYBV/f6PEQkwFKElNwoVsDzggXVt3qa5LTMdooN/gVCv4uCq3yoD/HoNkFAzdcEvduTU0shrAED0kIDAI/KXidm75xLiDh/f+oD9VaLnxY4x6TwjQ0XNSoD34EEwO0Q2Ukzc8N3lRpybgLQM8h/2etN64OY1MiOBQY5B6uZGu0YXiA+I00EfNB8wNNYF6xAc9593evLBXhlIMcVwIL+bWvcEXPXvrqCrnrlq3slEmPQe8itiZqSsC8ujlJoiuhNxwYyPK57kMKxn7eAxGDZq4Soan7yZoEibtmMliC4tLwG2rD9f2/Fxyfam8oOM2yFUaDpEfGWcn9xJKSfrkKfDv9yDTnsz7m0DA3a+H7mPfBEAdOjmifIomJofege1xPEKjBryFUt65hameqQ/aQs7lbdnH0XFr2ddLqEsydI55d3ReQBZY1PtOCTQjO+n2uTx/OU9iyge+LY2Nc9KgAXYlKU8FE+sMt42ADmhqIrhtNPr3fkT51BmhQ7d6+6PXv0XfakiVtxasTBDsCBCka5fbGd0IAiPK9SjQa/GBMUVwgyjdUpYuWrKuRsqb1rFaaO9DNSYlqQPabi3urXuWuw4ExOiaDK6jV5fRixHkk20q9AHnHKQfxBO41Emf+uJT/KTzVYxbGtywYoUtI0wvvqT55/31IjCCXI+Irm+lI0voHukaYtKQI7bRhWGNt0ovY8lYO3nf5Yp24o85dX9ygmcD3RHANSv+gPFn6IfACPLp4w8foND4yT0v3uiHyrgtSdmXDqZjiAov4lK+cXs4KWuBwAhy3aIfp6hDCw7JCTLFIXg/Z8Uz9A84bgU31mpfUNuqs/o52346TFldaGa8GQTE7H4m4GY6P0BHp3wOMAgVJKDEt/QokwR5iGe4bQSmXOfHHznKYYO3inR+tUGfxOCQ+wwTgeEQ0Hmr9ADhcIROgiYCE4GJwETg5hGQUeVnyG8elAnAeAhwlgY37NwSHG9sJkUTgYnARGAi8CkEtN26eobxk3V9Hwvp+6GBAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Recurrent Neural Network\n",
    "\n",
    "The recurrent loops in a RNN allow relevant information to persist over time. A simple RNN architecture is shown here:\n",
    "<img src=http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png width=\"150\">\n",
    "\n",
    "A simple RNN takes not only an input X at time step t but also passes a hidden state that is the output of the previous time step into the network. The output of a RNN cell at time step t reads in Eq. 1:\n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "In this task you have to implement a simple one-layer RNN as a class in Pytorch, where you can choose a relu or tanh activation in the cell.You can see the architecture of a simple RNN in the figure below.\n",
    "\n",
    "\n",
    "<img src=http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png width=\"600\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DONE: Implement the RNN class\n",
    "from exercise_code.rnn.rnn_nn import RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luckily, Pytorch already has implemented a simple RNN in their library and you can call the RNN with <code>nn.RNN</code>. We will use the Pytorch RNN function to check if we have built the correct cell and compare the output of both functions. We also compare the running time of both classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference between pytorch and your RNN implementation: 0.0\n",
      "Cool, you implemented a correct model.\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import timeit\n",
    "\n",
    "# choose your network parameters\n",
    "input_size = 3\n",
    "hidden_dim = 3\n",
    "seq_len = 10 \n",
    "\n",
    "# define the two models\n",
    "pytorch_rnn=nn.RNN(input_size, hidden_dim)\n",
    "i2dl_rnn=RNN(input_size, hidden_dim)\n",
    "\n",
    "# initialise both rnn with same values\n",
    "for p in pytorch_rnn.parameters():\n",
    "    nn.init.constant_(p, val=0.3)\n",
    "for p in i2dl_rnn.parameters():\n",
    "    nn.init.constant_(p, val=0.3)\n",
    "    \n",
    "X=torch.randn(seq_len, 1, input_size)\n",
    "\n",
    "output_pytorch, h_pytorch = pytorch_rnn(X)\n",
    "output_i2dl, h_i2dl = i2dl_rnn(X)\n",
    "\n",
    "# The difference of outputs should be 0!!\n",
    "\n",
    "diff = torch.sum((output_pytorch-output_i2dl) ** 2)\n",
    "print(\"Difference between pytorch and your RNN implementation: %s\" % diff.item())\n",
    "if diff.item() < 10 ** -10:\n",
    "    print(\"Cool, you implemented a correct model.\")\n",
    "else:\n",
    "    print(\"Upps! There is something wrong in your model. Try again!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import timeit\n",
    "# runs=10 ** 4\n",
    "\n",
    "# print(\"Time Pytorch RNN {} runs: {:.3f}s\".format(runs, timeit.timeit(\"pytorch_rnn(X)\", \n",
    "#                                        setup=\"from __main__ import pytorch_rnn, X\", \n",
    "#                                        number=runs))\n",
    "#      )\n",
    "\n",
    "# print(\"Time I2DL RNN {} run: {:.3f}s\".format(runs, timeit.timeit(\"i2dl_rnn(X)\", \n",
    "#                                        setup=\"from __main__ import i2dl_rnn, X\", \n",
    "#                                        number=runs))\n",
    "#      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From now on we will use the Pytorch module that is faster and optimised in performance. However, it is always a good exercise to build the functions by yourself and we really advice you to do the exercise!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vanishing Gradient\n",
    "\n",
    "As discussed in the lecture, the simple RNN suffers from vanishing gradients in the backpropagation. The hidden state is manipulated in every time step along the sequence and the effect of the past inputs to the final output vanishes with the distance in time. In the next cell we will explore the vanishing effect of previous inputs in the RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "############################################################################\n",
    "# TODO: Define a RNN and explore the gradients on the output h_T wrt. the  #\n",
    "# input at time t and plot your result. What behaviour do you observe?     #\n",
    "# Hints:                                                                   #\n",
    "#   - use one input feature                                                #\n",
    "#   - pytorch allows backward() pass wrt. to any vector                    #\n",
    "#   - backward() can only be applied to scalars and not to output tensors  #\n",
    "#   - choose a good representation of the gradient plot                    #\n",
    "############################################################################\n",
    " \n",
    "    \n",
    "    \n",
    "############################################################################\n",
    "#                             END OF YOUR CODE                             #\n",
    "############################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert alert-info\">\n",
    "    <h3>Note</h3>\n",
    "    <p>It can be seen that the gradient of the output at time t wrt. to a previous input decreases exponentially. Hence, the final output does not change significantly for changes in the previous input and hence the RNN does not have memory.</p> \n",
    "<h3>Question</h3> \n",
    "<p>In order to better understand the vanishing gradient problem, calculate the gradients \n",
    "dh_t/dV, dh_t /dW, and dh_t/dX_0 analytically for t=3 and h_0=0 using Eq. 1. This exercise might seem a little bit tedious but it is really useful. Can you explain the vanishing gradient mathematically based on your findings?</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Long-Short Term Memory Network (LSTM)\n",
    "The vanishing gradient problem had been known for some time until Schmidhuber (1997) developed the Long-Short Term Memory Network and showed that this architecture can overcome the problem. <br> \n",
    "A LSTM is a more advanced recurrent network architecture that is able to learn long time dependencies. The architecture of a LSTM is composed of a **forget**, **input**, and **output** gate and the cell can remember values over arbitrary time intervals. Despite various different and exotic LSTM architectures, the standard LSTM cell is shown in the figure below:\n",
    "\n",
    "\n",
    "<img src=http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png width=\"600\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to a simple RNN the LSTM cell has a hidden vector and an additional cell state vector. __What size does the cell state have?__ <br>\n",
    "The operations inside the LSTM are given as \n",
    "\n",
    "<img src=https://wikimedia.org/api/rest_v1/media/math/render/svg/2db2cba6a0d878e13932fa27ce6f3fb71ad99cf1  width=\"600\">\n",
    "where \n",
    "f_t: forget gate,  <br>\n",
    "i_t: input gate, <br>\n",
    "o_t: output gate, <br>\n",
    "h_t: hidden state vector, <br>\n",
    "c_t: cell state vector, <br>\n",
    "x_t: input vector, <br>\n",
    "t is time step, \n",
    "<br> \n",
    "<br> \n",
    "and<br> \n",
    "sigma_g: sigmoid activation <br> \n",
    "sigma_c and sigma_h: hyperbolic tangent function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next step you should implement your own LSTM with the operations stated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DONE: Implement the RNN class\n",
    "from exercise_code.rnn.rnn_nn import LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference between pytorch and your RNN implementation: 4.413136522884997e-14\n",
      "Cool, you implemented a correct model.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# choose your input parameters\n",
    "input_size = 3\n",
    "hidden_dim = 3\n",
    "seq_len = 10 \n",
    "\n",
    "# define the two models\n",
    "pytorch_lstm=nn.LSTM(input_size, hidden_dim)\n",
    "i2dl_lstm=LSTM(input_size, hidden_dim)\n",
    "\n",
    "# initialise both lstms with same values\n",
    "for p in pytorch_lstm.parameters():\n",
    "    nn.init.constant_(p, val=0.3)\n",
    "for p in i2dl_lstm.parameters():\n",
    "    nn.init.constant_(p, val=0.3)\n",
    "    \n",
    "X=torch.randn(seq_len, 1, input_size)\n",
    "\n",
    "output_pytorch, (h_pytorch, _) = pytorch_lstm(X)\n",
    "output_i2dl , (h_i2dl, _ )= i2dl_lstm(X)\n",
    "\n",
    "# The difference of outputs should be 0!!\n",
    "diff = torch.sum((output_pytorch-output_i2dl) ** 2)\n",
    "print(\"Difference between pytorch and your RNN implementation: %s\" % diff.item())\n",
    "if diff.item() < 10 ** -10:\n",
    "    print(\"Cool, you implemented a correct model.\")\n",
    "else:\n",
    "    print(\"Upps! There is something wrong in your model. Try again!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import timeit\n",
    "# runs=10 ** 4\n",
    "\n",
    "# print(\"Time Pytorch LSTM {} runs: {:.3f}s\".format(runs, timeit.timeit(\"pytorch_lstm(X)\", \n",
    "#                                        setup=\"from __main__ import pytorch_lstm, X\", \n",
    "#                                        number=runs))\n",
    "#      )\n",
    "\n",
    "# print(\"Time I2DL LSTM {} runs: {:.3f}s\".format(runs, timeit.timeit(\"i2dl_lstm(X)\", \n",
    "#                                        setup=\"from __main__ import i2dl_lstm, X\", \n",
    "#                                        number=runs))\n",
    "#      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Gradients \n",
    "Analogously to the RNN, calculate the gradients of the input wrt. to the output of the LSTM and compare it against the RNN gradients. __What do you see?__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "############################################################################\n",
    "# TODO: Define a RNN and LSTM and explore the gradients on the output h_T   #\n",
    "# wrt. the input at time t and plot your result.                           #\n",
    "############################################################################\n",
    "\n",
    "\n",
    "\n",
    "############################################################################\n",
    "#                             END OF YOUR CODE                             #\n",
    "############################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST image classification with RNNs\n",
    "\n",
    "In the previous exercises we already have classified images with a Fully Connected and Convolutional Network. In this exercise, we will solve the problem of image classification with a recurrent neural network.  \n",
    "\n",
    "For the experiment we use the MNIST handwritten digits dataset which we already know from the autoencoder exercise. This dataset consists of images of the 10 different digits (10 classes). The images have the resolution 28 x 28. The idea for the RNN classifier is to interpret the image as a sequence of rows. This means that we pass the rows through the RNN and use the final hidden state for classification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert alert-info\">\n",
    "    <h3>Note</h3>\n",
    "    <p>\n",
    "    In this semester you have seen three different types of neural networks, namely Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), and now Recurrent Neural Networks (RNNs). We have seen that we can use all three architectures for image classification. However, it turned out that some models are better than others for image classification. Try to think about advantages and disadvantages of the models, regarding # of parameters, transformations of the object in the image (scaling, rotation, translation,...), training time, testing time, over-fitting, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define data loader\n",
    "from torchvision import transforms\n",
    "import pickle\n",
    "\n",
    "class Unsqueeze(object):\n",
    "    \"\"\"Adds a channel dimension that that our 2 dimensional input (H, W), \n",
    "    fits the 3 dimensional (H, W, C) expectations of pytorch's ToTensor function which\n",
    "    expects a PIL image. This is very inefficient but you most probably will use pytorch's\n",
    "    PIL image loader. Check out the documentation and make it more efficient :)\n",
    "    \"\"\"\n",
    "    def __init__(self, dimension=0):\n",
    "        self.dimension = dimension\n",
    "    def __call__(self, numpy_array):\n",
    "        extended_array = np.expand_dims(numpy_array, self.dimension)\n",
    "        return extended_array\n",
    "    def __repr__(self):\n",
    "        return self.__class__.__name__ + 'dimension={}'.format(dimension)\n",
    "\n",
    "    \n",
    "# transformation of data\n",
    "transform = transforms.Compose([\n",
    "    Unsqueeze(dimension=3),     \n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=(0.5,), std=(0.5,))\n",
    "])\n",
    "\n",
    "\n",
    "class MnistDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, images, labels, \n",
    "                 transform=None):\n",
    "        super(MnistDataset, self).__init__()\n",
    "        self.images = images\n",
    "        self.labels = labels\n",
    "        self.transform = transform\n",
    "       \n",
    "    def __len__(self):\n",
    "        return len(self.images)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        image = self.images[idx]\n",
    "        label = self.labels[idx]\n",
    "        if self.transform: \n",
    "            image = self.transform(image)\n",
    "        return image, label\n",
    "\n",
    "    \n",
    "# loading the train data\n",
    "with open(\"../datasets/mnist/mnist_train.p\", \"rb\") as f:\n",
    "    mnist_raw = pickle.load(f)\n",
    "\n",
    "X, y = mnist_raw\n",
    "############################################################################\n",
    "# DONE: Set a useful training/ validation split                            #\n",
    "############################################################################    \n",
    "\n",
    "train_split = 0.8\n",
    "\n",
    "############################################################################\n",
    "#                             END OF YOUR CODE                             #\n",
    "############################################################################\n",
    "\n",
    "\n",
    "train_dset = MnistDataset(X[:int(len(X) * train_split)], y[:int(len(X) * train_split)], transform=transform)\n",
    "val_dset = MnistDataset(X[int(len(X) * train_split):], y[int(len(X) * train_split):], transform=transform)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAG1CAYAAAD9WC4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdebxV0//H8ddu0KCBBg2UvkmlklJIhkI0KYWvJiUyJiFDREhEZPxKyJj8yByJwheVIZklV6bIkDLUNw2m9u+P3Wftc+89dz7j7v18PDzudc6556zVPmeftT/rsz7L830fERERkSgrl+4GiIiIiCSbBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5KVlwON5Xi3P857yPG+D53nfeJ43OB3tSBbP80Z5nveO53l/eJ53f7rbk2ie51XyPO+ercduved5H3ie1zPd7Uo0z/Nmep73o+d5//M8b7nneSenu03J4nne7p7nbfY8b2a625Jonue9urVvv2/977N0tynRPM8b6Hnep1vPqV96nndQutuUKDHHzf77x/O8/6S7XYnmeV4Tz/Pmep73m+d5qzzPu83zvArpbleieJ63h+d5//U8b53neV94ntc/1W1IV4RnKvAnUA8YAkzzPK91mtqSDD8AVwH3prshSVIBWAl0AWoClwKPep7XJI1tSoZrgCa+79cA+gJXeZ7XIc1tSpapwJJ0NyKJRvm+X23rfy3S3ZhE8jzvcGAycCJQHTgY+CqtjUqgmONWDagPbAIeS3OzkuF2YDXQAGhHcH4dmdYWJcjWgdtsYA5QCzgVmOl5XvNUtiPlAx7P87YHjgHG+77/u+/7i4BngKGpbkuy+L7/pO/7TwO/pLstyeD7/gbf96/wfX+F7/tbfN+fA3wNRGow4Pv+J77v/2H/u/W/3dLYpKTwPG8gsBZ4Od1tkVKZAFzp+/5bWz+P3/u+/326G5UkxxAMChamuyFJ8C/gUd/3N/u+vwp4AYhKIKAl0BC4yff9f3zf/y/wOin+3k9HhKc58Lfv+8tjbvuQ6BzYbY7nefUIjusn6W5Lonmed7vneRuBHOBHYG6am5RQnufVAK4ExqS7LUl2jed5P3ue97rneV3T3ZhE8TyvPNARqLt1muC7rVMhVdLdtiQ5AZjhR3NPpJuBgZ7nVfU8b2egJ8GgJ6o8oE0qXzAdA55qwP/y3LaOIBQrWcbzvIrAQ8ADvu/npLs9ieb7/kiC9+ZBwJPAH4X/RdaZCNzj+/536W5IEo0FmgI7A3cBz3qeF5VIXT2gInAswXu0HdCeYJo5UjzP25VgmueBdLclSRYQXPj/D/gOeAd4Oq0tSpzPCCJzF3ieV9HzvCMIjmXVVDYiHQOe34EaeW6rAaxPQ1ukDDzPKwc8SJCPNSrNzUmarSHYRcAuwBnpbk+ieJ7XDugG3JTutiST7/uLfd9f7/v+H77vP0AQSu+V7nYlyKatP//j+/6Pvu//DNxIdPoXayiwyPf9r9PdkETbei59geCianugDrAjQW5W1vN9/y+gH9AbWAWcBzxKMLBLmXQMeJYDFTzP2z3mtr2I4HRIlHme5wH3EFxhHrP1DR11FYhWDk9XoAnwred5q4DzgWM8z3svnY1KAZ8gnJ71fN//jeBLI3aKJ4rTPQDDiG50pxbQGLht68D8F+A+IjRw9X3/I9/3u/i+X9v3/e4EUde3U9mGlA94fN/fQDCKvdLzvO09zzsAOIogUhAJnudV8DyvMlAeKO95XuUoLS/cahqwB9DH9/1NRT0423iet9PWpb7VPM8r73led2AQ0UrsvYtgANdu6393AM8B3dPZqETyPG8Hz/O622fQ87whBKuYopQbcR9w1tb37I7AuQSrYSLD87zOBFOSUVydxdbI3NfAGVvfpzsQ5Ct9lN6WJY7neW23fg6rep53PsFqtPtT2YZ0LUsfCVQhmNN7GDjD9/0oRXguJQg1XwQcv/X3yMypb51LP43gS3JVTH2MIWluWiL5BNNX3wG/AVOAc3zffyatrUog3/c3+r6/yv4jmG7e7Pv+mnS3LYEqEpSIWAP8DJwF9MuzaCLbTSQoKbAc+BR4H7g6rS1KvBOAJ33fj3Lqw9FAD4L36hfAXwSD16gYSrDwYzVwGHB4zCrYlPCimewuIiIiEtLWEiIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnmF1obxPC+rl3D5vl9kcTH1MfMV1ceo9w/Ux2ygPka/f6A+ZoOC+qgIj4jINqRRo0asXbuWtWvXMnr0aEaPHp3uJomkRKF1eKI6youlPmY+XVWqj9kg2X1s0KABAD/++GOJ/q5WrVoAnHtuUMNu/fr1NGzYEIALLrgAgL/+Kt7OMPosqo/ZQBEeERER2WZFbX8nEZFIGj9+PADff/89AFdfXbzdI8444wwANmzYAMDcuXNZunRpElooktkU4REREZHIy5gIT5MmTQBcAl2nTp0AGDx4MCtWrEhTqxKrUaNGANx4440AHHvssQD06tWL559/Pm3tKkiLFi3cHP+IESNy3Reb+zVv3jwAFixYwA033ADAn3/+maJWltzhhx8OwDXXXEOHDh2AsD+WHzF79my6du0KwCefBPvajh8/npycnBS3VgTOO+88Tj75ZAAGDRpUrL/ZddddAfjll18AmD59OgD//PNPElookvkU4REREZHIy4hVWieccAJXXXUVALvsskuu+x588EGGDRtWqufNtGz0K6+8EoBLLrkk1+3jxo1j8uTJpXrOZPbxgw8+YM899yz24z3P48UXXwTg+uuvB+Cll14qzUvnkuiVIdbGQw89tETt2Lhxo7vKnjVrVon+tjCZ9j5Nhqj0sW7dugCsWbMm333J7OMVV1zhXttWW8WLou6www4ArF271t1Wv359AFatWlWal85Fq7TUx2xQUB/TMqVVrVo1AGbOnAlA7969qVAhflOeeeaZlLUr2Twv/vvsgQceSHFLiuf555/n/fffB+C7774DYNq0ae7+vn37ArDffvsB0LZtW7p16wZA586dAWjfvj0AX3zxRWoaXQzr1q0r1d9VrVqVBx98EAj6CjBp0iQgTAjNZDZdXLNmTXfbPffcA8APP/xQ4N/VqVMHgJdffpkPP/wQoNQXIZmqZcuWABx88MHutlNOOcX9bp/d2rVrA+E00RNPPME111yT9PZdccUVBd5XvXp1mjZtCsBRRx0FhBdXkJiBjpRdxYoVOeSQQwDYbbfdgPB4lSsXTrbYZ7FKlSqMGTMGCBPVo2L48OFA8Hm67rrr4j7mjTfe4N///jeQuPewprREREQk8tIypfWf//wHgFGjRhX52I0bN3LhhRcCMHXq1BK9TiaF7jp16sQrr7wCwHbbbZfrvurVq7Nx48ZSPW8m9bF27do899xzAOyzzz4AtGrVCoDPPvus1M+b6DD6TjvtBMCUKVMYMmRIrvt+/vlnAM4880y6dOkCwMiRIwt8rt69ewPwwgsvlKQJuSTzGLZr185NK1oBuljjxo0D4Nprry3wOSz6sWzZMnelaQn4xZXO9+n2228PhP2IjdxYRKdFixbWBpfAblGdNWvWsHLlSiBMbrf3ydVXX82iRYvsvrT0sXPnznz55ZdA2I8FCxYk+mWAzJrSql69OhBO45mBAwe6aTyLkGzZsoXPP/8cwEWhLWodKxnH8LDDDgPg1FNPdRGL4vrf//4HwF133QXgvgvLItXvUzs+nTp1cu23c+uWLVsK/Lty5cq5c9dpp50GBMfs77//LvI1VXhQREREtlkpz+E58cQTXSEss3DhQpo1awaE5dNN1apV3dXnN998A8CcOXNS0NLEuuCCC/JFdubOnQvAH3/8kY4mJdwvv/zi8rIswmPRkbPPPjtt7cpr9erVQFCQza4AjZXYX7duHbNnzwbCiJwlLGeTMWPG5IvsWM7YW2+9xd13313kc+RNss8GlmvVsmVLqlatChQexVm2bBkAixYtcqUHFi5cCATRnG+//TZ1jS+mSpUqAfD555+7JOqffvopnU1KCIvctG7d2t1mJSIsJxBg9913B2CvvfYCcpfKMBZB8H2fHXfcEYD+/fsD4UxDslhUwsqQVKlSxd338ccfA7B48WIgSCw/8sgj8z1HjRo1cj2XRfIefvhhF/3JNO3atQPC/CSLosbmxxWXLSyx6Nx5553HrbfeWuq2pWzAYwOaU045hfLlywNBEiQEX4pLliyJ+3f//POPS3K2Gi9z584tNBSWSayvjRs3znefDXiiVBfDTrg2cLBVXtWqVeP3339PW7vi2bBhQ6HJxtaHt99+G8iuAY99CdjJHcJw+E033QQU/b6zL5yShuHTwaatZsyYAcDRRx8NBF94Nl38f//3fwBuCgrCqZ9sqq9kCa52HAubcs0WzZs354ADDgDgnHPOAXIPeGxgWlgKRlFsCuv+++8v9XMUV6dOndzxqVy5MgAfffQRgwcPBsIpfntMvMFOLFuRZ4tGxo0b5wIB9j1iAYF0atmyJfPnzwdwA8x4LAl5+PDhLF++PNd9J510EgCXXnppvr/r27dvmQY8mtISERGRyEt60rItN3/jjTeAcKoD4MADDwSCUJct7zUW6ps+fXq+sHv79u354IMPinztTEjotSXb1n8Iw3N2RWNLXEsjE/oYjyV52q7M++23H++8806pnitdiZJWqdaqYNuUCITJhHZ8816llEQyjqEtba1fv75rm33eLOG2KBYlstIEENaAueWWW0rSnKS/T5944gkgDKNbRODqq692kZ1kR3FS9Vm0pc0WPS3pFW+fPn1cpKSk6QGJ+ixa4rhdxVerVs0lt8b7TooX4bHK0RaltXSIAQMGxP07S9AvaBn01scl5Bg+8sgjHHfccUA4hd6uXbt8y6utREb16tV5+OGHAXj11VeBYJrVIkE2LT127FggSA0xr732GhBEiYpTHiMZ71NbEDB79mw3mxNvFsbKYNiU8+uvv17gc/7zzz/5nuO1115zSeeFUdKyiIiIbLOSnsMzcOBAIHdkx5K4LOrx5ZdfuvwCG+Xbnkax85J2WyYmEOZlia52RRzLRvxliexkKtunKu9S0UzVpk0bICwoF+v8888Hckd2jCW5liWyk0yW+AlhOYfiRnYKk6mJkparZBEAi0BddtllaWtTosVGLqD4kR0rHGm5Fe3atXPJr+lYAHLkkUe67wBLKIcwN8kSsK3IpeWoQPgd8Mgjj+R73osuuggIv3MgjPBMnjy50MhOoln03l4b4hfPs+KtL774ouubRa5i2WfX8ginTZvGW2+9BYRLvE8//XSX55oqlqBs763atWvnKqIY6/HHHy9WZMci0fGep6DivcWlCI+IiIhEXtIiPLacLm/Z9QcffJAJEyYA4dXYqlWrXGTAciKsSJ/lgEA43/nrr78mq9kJY6tG4q1wsXnMKLJoSOyVWyazPdz69OlTor+zq0W7CsmUVYOWG2H//g8//HCu7UASxZZEN2rUKO3bhvTv39+dS+xnNq26Ki47R8bmVBXn8RZNiV31VJbVTmU1Z84cF42xshA//PCDi8pZ5CJeYcB47Lms3Els32zfvNtvvz0BLS++2OjEihUrCnyc5d/kLVlSEOvbO++8w/HHHw+Eqw933nnn0jS11Nq0aeNy52xF1pYtW9xqXIuQWx7S22+/zebNmwt8PlvNZlvgbNmyJd95tazv26QNeGygk3cz0ClTpsQNi1tCa97E1kGDBiWphclVr169fLetX78eCJPStgU29ZOpe8HE2wSyOJ588kkgnLK844474m7mmGrjx48HwtDv2rVri1WZtLjspGQJmePHj6d58+YJe/6SsIuK448/Pl+o2waye++9tzvGTz/9NADvvvtuCluZGB07dnT70xVWbdeqiJ900kluvzNLKP3qq68A+O2333Ltp5YO9vmxnzadUxI2XXn55ZcD8b8Me/ToUdomlslXX33lkqjts7JgwYKEpTFUq1aNoUOHArjPd6qn12+99da45VbsfWd1zIrLxgyxpTTyKulz5qUpLREREYm8pER4atWq5ZKxjBVYsgqTRbGpEQtXQuHLCTNNvOq0VhQtEwpEJYsl0NkVt03pleYKLhUs7Gqh1pIWcbP3te/7Sa/cWhw27WuR1caNG9OzZ08gjCxu2rSpWM9llWxj2dJeC5//+eefLskwtqBfKljk4qijjso3pWVJ6CeffLJ7L1588cVAMM1hV8eJSOROhe7du7uinvGm9Pfdd18AV8Jjzz33ZO3atUBYaM+mCg499NC0J3OX9nxgKQ6XXnopI0aMiPuYiRMnpj1t4P3333cJvZZEXb58effvHm/K1d7PFuFYu3Ytn3zyCZA7CRqC97wdczu+d9xxR4J7Ed+xxx4L5F6IZO+10aNHFysKY5WzL7zwQvddX9iUnE0LPvroo6VpsqMIj4iIiEReUiI8l112mbvCtPL1tn9PUUlHtt/IbbfdFjSwQgU+/fRTAFeYKdO1adOGfv365bs9ypEdgFGjRrndw/NecWcqi4jY1W/evbUgjDY+8cQTrqhWXpMmTWLp0qVAmHCfDldffTUQFvbq3bu3OybvvfceUPy92+yKM5Z9ri1f6aKLLkp5ZMdYtMnzPBfFeeqpp4BwmSwE0RHAfSa7d+/urobtajXTVaxY0SV/5tWgQQO3m7aVWXjzzTdd9NLKf3Ts2BGAO++80+UTZgsrc3HnnXcC0KtXL3efJbbacu677ror7RHlMWPGuL2j2rZtCwTvNfss2kIC276mX79+7rNluWnFZc85ZMgQl6RtpU+SwbZ6snw+CCPkhx9+OEcccQRQ+Ln/hBNOAIq/2MPOZ2XdKy6hlZbtpJOTk+MSGa3OQ1GrYOxvr7jiCiBMSPN932Xf25u9uNJVhXjAgAEuc94sXLjQvRESmdyaCZWW9957byBIOLf3k03pTZkyBaBMibPpqrSc1/bbb+/Cx7ZXUyzbG86Oc3El4xjaF37Xrl1dtfOyWr58uftSsUroxR3sJKOPEydOBIKpKjuB2oDH9s+CcMWaTSnHruqyKVf7u7JI5mdxypQp7rxhAz2b5jjxxBPd/kMW8j/ttNPcYL5Vq1YA3HzzzUBQ2d7qtZR0Y9h0fRZtCijeBYet+rX3Q1kk8hjWrVsXCC9CBg0aVOLBTF7z5s0DgtpEBx10EBAOZJs0aeKmlmyT1XgX2Ynq4yuvvBJ3Q9DirFy1xyxYsMBNicd7jCXa77bbbkU1JxdVWhYREZFtVkIjPHalH7vs0/a9sKvfgtgeOLZ01DzwwAMMHz68JM1w0hX9ePLJJ11/zCWXXOJ2t02kZPfRksvOO+88IP5I265kateu7a6crSLo448/DlCmEHqmRHggrJdhIePYisaZFOExhxxyCJ06dQIKX+4Z68033wTgrLPOynX70KFDeeihh0rTjIyIRJotW7a496lFj21KqCySHeGx6SpLZD311FOBIPL47LPPAuFUQbly5VxlZlu2bUvWly5d6uqdFTeB3aTjszho0CBmzpxpr5/v/kRFMLc+f8KPoU0BzZ8/330W47FpIZuWWrZsmTt/WmVsi1zGzhJYcvSkSZPcMnyL9MVWnTaJ6mP37t3jVuq26I21MbYkiSWT2zRrTk6OSxmwlBV7n5YrV47TTz8diF99ujCK8IiIiMg2K6FJy3n3eimu3r1751tSZ9VE7SomG9jVYmy+kiWlvfDCC2lpU2nYnOrYsWPdMvOSzj3b8thzzjkHCK4qbQl3aXdNTwab/77yyiuBcJfxc889N25Uyq5aMj0Z27zyyisuiTpv1fOCWK5HvB2qo+DJJ590CcwW9UpEhCeZ/vzzTxc5jBdBtCXKVhqhe/fubg8tOwc99thjAFxwwQUljuykk+2onldJS0iki+X7derUKV8E3KoS//XXXy5KZwt9iuuDDz4A4Prrr+eQQw4BoGnTpmVveBEWL15caN6U5RMVtefbHnvsAYTV280HH3yQa/FBIijCIyIiIpGX0AiPjTSLy+abJ02aRP369QFYsmQJgFt1kAnl+otSvnx5IIzsxO6jYqP2kv7bpMOoUaOA4EoBYMOGDS5f4N577wXCLTMmTJjgVgDYUsFy5cq5zHxbJWB5B23atHFLSW07hjlz5qS18FvXrl3dKjJrr1m8eHG+eeNq1aq5q7XY3B2TTVfNhbHl6HY1anPwCxcuTFub4rH30WuvveaW3BfHzz//7KJXFgXJdPfcc4/7fFpOSCz7XA4ZMgQI8pRsR3HbSyuTIquFsaKRFpFs2bKlO15W+uGSSy5Jy07vpWHfZRDuEn7fffcl/HXq16/voiTFLT1RFmvXrnWR8bKwWYS82518+OGHCS/lktABT7w6EbaJqOnQoYPb58YSmitUqMDnn38OhFNY9sbOBlaPwOp9QDhQsy/UTHfssce6papW1XLMmDE899xzQLgE1qrV5uTkuOMYrz6SJc/ZQOK0006jUaNGQJi49uqrr7qwdEE1RpLp0EMPzTfQMdOmTXMnj//+979AECYu7ERly56zWYsWLdhzzz1z3WZh92+//TYdTSqQJTuedNJJ+dpcmNhl6dnIElLtc9esWTN3jFatWuV+WumAbGPTPTZQ8H3fnU9tijydta5KKnZq3DYLTQQbIFhpgdh9JydPnpyw14kSTWmJiIhI5CU0wmOj8L///tstFbTkVSskuOuuu7qt5M2cOXPc8rkNGzYkskkpkTeKBeHywWy5yjrzzDN55plngHCq0fd9F260xDNL7ixqib0ladvP+++/35UXsCTErl278uGHHwK4fXFKu+y5NP766y+XIGjTksbzPG655RYgvEKzcHQ8r776qisKls3q1q3rdnm2YpGZWuHcdkFv1aoVTzzxBBBMj0N4HsnJyXEJ95agvNNOO7mpVzs/Zbovv/wy7nkmSmypcrzighZhLsnUZaaITcuwEh6lZRWnhw0bxgUXXACEe1Bt2rTJTV8+//zzZXqdVGnSpEmB+7rFTgUmiiI8IiIiEnkJjfBY3k1OTo5LVq1Vq1aunxAmVFlS6EUXXZSVkR0Tb9lkpuU7FGX+/PnueFh0qk+fPq5Qlu3X8t1335Xq+X/88UeXhGgJ3BdeeKHLoZk6dSqQ2gjPxIkT3dYItkzZEkNjWYKy5SXF8/jjj7s8imwWW1Jh2bJlAAlJTEwGi84cdNBB7vhZwU9LIM/JyXFbS1jOz5YtW3jyyScBXGRI0sO2IBo/frwrhmj5VcuXLweCz2k2RnbML7/84n4fNmwYEBYZtP3OYrfesT21dtllF3r27AnAv/71LyDYqwqCJHXbL8wiy+eee27cXdgzXXH300qEhFZaNs2bN3fhNktAMzfddJObHrHk2GRJVXXXlStXAtCwYUMAPv/8c5eQXdoBQnElqo+dOnVyX+xW6TNTpKK6q01ptW7dGghWnJ1yyilAuMFk7IoeG9zYqph58+a5eicllQlViO2L5+WXX3Yh8o8++ggIK7mWRbL7aIsdbNqqQ4cOQLDix1ZNWnXs8ePHuwFPIlcJZsJxTLZEfRbzrsQaMWKEW4llF8RW6TuV047JOIa2gm7q1Kn5VndakvnTTz/tpvLsIjPeajyzYsUK929X0u/RTHqfNmnSxC1YyqtixYqlfl5VWhYREZFtVlIiPJkik0ayyaI+Rr9/kPw+WoXW2D3vsinCk1fjxo2B3FE5i+Yka7o5E45jsiXis1ipUiU3TWp79G39WwAXfbOd7FNJxzCQqj7Wr1/fpRXsuuuuue5ThEdERESkFBKatCwi2enrr792P21H+Ntvvz2dTSoTi+Jk2+KBbcHo0aNzRXaM5XZaQVOJvlWrVrkcSSsDYoVpk0ERHhEREYk85fCojxlPOTzqYzZQH4vXvx122MFtw2L7602fPp0LL7wQyL0VQ6rpGAai2kcNeNTHjKcBj/qYDdTH6PcP1MdsoKRlERER2WYVGuERERERiQJFeERERCTyNOARERGRyNOAR0RERCJPAx4RERGJPA14REREJPI04BEREZHI04BHREREIk8DHhEREYk8DXhEREQk8jTgERERkcjTgEdEREQiTwMeERERiTwNeERERCTyNOARERGRyNOAR0RERCJPAx4RERGJPA14REREJPI04BEREZHI04BHREREIk8DHhEREYk8DXhEREQk8jTgERERkcjTgEdEREQiTwMeERERiTwNeERERCTyNOARERGRyNOAR0RERCJPAx4RERGJPA14REREJPI04BEREZHI04BHREREIk8DHhEREYk8DXhEREQk8jTgERERkcjTgEdEREQiTwMeERERiTwNeERERCTyNOARERGRyNOAR0RERCJPAx4RERGJPA14REREJPI04BEREZHIq1DYnZ7n+alqSDL4vu8V9Rj1MfMV1ceo9w/Ux2ygPka/f6A+ZoOC+qgIj4iIiEReoREeERHJfDvuuCMA7dq1A+Cyyy5z940fPx6ARYsWpb5hIhlEER4RERGJPM/3C56qi+o8Xiz1MfMpb0B9zAap7uPo0aMB6NChA4ceeigADRo0iH0tAH777TcA/v3vfwPwyiuvlPo19VlUH7OBcnhERERkm6UIj/qYMDVr1mT33XcH4IQTTgBg0KBBANSqVQt7rx133HEAPPHEE8V63lReVdasWROAwYMHc/jhhwPQo0cPAKpUqcKbb74JwIUXXggkJi8ik45hsqiPgUT00SI1jzzyiL1uQa+V6/4XX3wRgJ49e5b6tRXhUR+zgSI8IiIiss3KmAjPCy+8AED37t0BWLNmDQATJ07kP//5T6meM5NGst27d2f27NkAPProowAMGzaszM+bzj42a9YMgMMOOwyAUaNGscceexTUBnel+eGHHwJB7kFxpOKqsk6dOgAsXLgQgBYtWvDXX38BMG/ePCBob/369QHcfRateuaZZ0r92pn0Pi1Ms2bNOOOMMwA477zzSvS32dLHskhVH++55x4Ahg8fbq+b7zHLli1z0cfbb78dgN9//x2AFStWlPq1FeEpWx9tNd3BBx8MhNFjiyznZRFyy9EqLP+qS5curF27FgjPsfFsy5/FtCxLb9q0KQAXXHABACeeeCIVKjNZwOQAACAASURBVARN2bJlCwC1a9cG4Oabb6Zt27YAnHLKKaluasJcdtllVKxYEQgHCtnIBgYjRoxg3LhxAFSrVg3IfeL93//+B8Aff/wBQI0aNahUqRKAm/Zq1aoVy5YtS03Di9CyZUsgGOhAEP63L/WlS5e6x9mXx+mnnw7AlClTAFi8eDE//fRTytqbSpUrVwbg6quvTnNLtm02lTV06NB8933zzTdAcIEI8Pjjj7sBThQ1btyYyZMnA8EXPeAuttatW5e2dhVmxx13ZO7cuQDsu+++ue7bvHkzCxYsyHVbu3bt3Dn1xx9/LPB5LUgwY8YM9zgrT5ANGjVqBIQJ908//TQA9evXd/8ml156KVD2FAJNaYmIiEjkpTzCs9tuu/H888+73/P6+eefAXjooYeAIDJw4oknAnDkkUcCQdLdBx98kIrmJkytWrXc75kS1SiO7bffHoCBAwcCuOnF7bbbLu7jP/30UyBMjPzuu+/c39l0yN9//w2EUaBMYMnIduX1+++/k5OTk+9xFtWyq5LevXsDcPzxx3PDDTekoqkJc+CBB3LdddcB8NhjjwFw00035Xtc3759gSDCYI+T1Ntll12AMGpqkdU///yTk046CYDXXnstPY1LMovATps2DYD999/fRYzNEUccAZCx79HGjRu7yMtbb70FwJAhQ4CgdEDeyFS1atUYO3YsQNxonX0ubYqzXLlyHHXUUclpfAJUqlSJ9u3bA2EkvVevXq7N69evB8KZghUrVripvxkzZgBwwAEHFBrtKooiPCIiIhJ5KYvwWN7K3Llz40Z2nnvuOSDME1i8eDEA5cuXd1edH330ERAkfGVbhCdWWUaoqWZXGBbZKMoxxxwDhJGdeOwKtbDHpNo///wDwDvvvFPo4ywp8NlnnwXCCE/16tWT2Lrk6dSpEwAbN24E4kd49ttvv5S2KdkOO+wwFx0xlqz+66+/pqNJRapevTqnnnoqEJRHgDDf8brrrotsZAfgzDPP5JprrgFyf87s/DFq1CggXPiSqT788EM3g/Hee+8BhSeQ//77725bEDvmpl27di7aZbMHV111lcvlyiSWeH3++efnyy2aOHGiy8WyKJa9rzds2MBpp50GhH0s63dn0gc8NtCxAU3sYOeHH34AgjfqLbfcAuROEIXgi8i+ZMwFF1zArbfemrQ2J5KFoXfeeWdXF6O49WfS7cwzz3TJYoWt5vvzzz8BGDlyJMuXL4/7mNq1a7v+289slk1JgQWxLwoIq/HGY9OZUPSAMBOVKxcEsu3kOXXqVHdf3lo1n3zyiUuMjH2cvcdtQGRfXKkycOBAl+xv7JhZIn1U2GfryiuvBIJp5tWrVwO4nxBOt1uSq33XfPnllylra0nZd9nJJ58MhCkCBZ03jS3qse+9IUOGuAtHm860ukzp1rBhQwCeeuopIFykcuutt+abcvvxxx/dxWZeHTp0cM9l30NlpSktERERibyk1+Gx0N1ee+2V776bb74ZKLqmx8477wzAt99+CwTJTTvssEORr50J9QY6duwIhFN0EEzTJUoy+3jGGWdw22232evku9+SdC1JMN7Vf79+/YAgqmXPcffddwPh0u6iZFLtD1sC+/LLLwNhvYsjjjiCX375pVTPmer36YEHHggES+8t8dOStWOPoR0fu6r85ZdfXLJhSRPO0/VZrF69Otdffz0Qv6xF3ghPUfdZhKdu3br5Hp+MPjZv3hyAt99+O980nEU77Co4FZL1WaxSpQojR44E4OKLLwbC/vXt25cvvvgi399YTZtVq1YB8N///hdIbiVpKNv7tFWrVgBuubVNQZ1yyinuuzKW1f2ypF2refb000+7782S1lVK9neGRRwt7eScc84Bip9QX7VqVQBmzZrlon22SKS4VGlZREREtllJy+G54oorAGjdunW++2xEftdddyXr5TPGmWee6X7P1ITIgixZssRV9rRIgO3HM3/+fBcNsGXm8cRLUM/WAnYtW7Zk5syZuW679957AUod3Ukli+zYMS1fvrxb0hovOnfIIYcAYf7AQw89lFGlBArTrVs3AO68807+9a9/AWGk5tFHH3V5H8ceeywAe++9NwBNmjQp9Hljy0ukwtdffw0EkXJbohtFAwYMcMvKL7roIgAeeOABIKxqnpe9F+24WrHaTGYlSc4++2wAbrzxRgBef/11V3bFosdHHXWU+3xa0rL9/6hRo1xeWTrZv7mVtxgyZIhb4GLnGVuCXxSLZlqCert27Yo9C1BcivCIiIhI5CUlwjNgwAA3D2tXh7EsQ/2zzz5LxstnlNhtJBKxs3YqvfPOO+5KuaQsx8rm5SHcH80KTKXb3nvv7aIexiIdHTt2dKtgbKXdhAkTXP7VfffdB+ReyZOpLGpx//33A2EO2fvvv1/oFVQ2r6az1S9NmjRxV8J25XjVVVe5lSGzZs0CwivoqlWruhwQ++wecMAB7nlTvV2D5XxY7hiEJQQyfRl2Sdx///3u/VlctsrQos95t2bIZFZY15bWz5gxw207YUVQW7Vq5Qq8WoFCi0xmCjt/dO7c2f2Ml4tUGDsH26oui6LedNNNbnV3oiRlwFOrVq24Ax0IElZL+sa05aTmjjvuKHXbUsX2H6pXrx4QfHm8/vrr6WxSSl177bUA7Lrrru42GxzkLTOQarHVTgt6nxbEEuezpbRA06ZN3Ze/7WFnJ9ljjjkm7pJQ2/PNqtsa+6LNBvZeu/XWW90xs+nIeH3etGmT+5l32jITxCZO2yIBm/rZFpUvX54xY8bkus2S07OJJfIOGDDAfT8cdNBB7v7GjRsD8P3336e+cUVo2LChC15Y4nRRgx1LsLel6nvvvbe7OLYpTKvbU9zNpUtCU1oiIiISeUmJ8NioL5YtGRwzZgwbNmwo9nMdcMAB+ZaTWsHCTGZRBEva9X3fTYNEmS1bznvMXn/9da666qp0NCkfSzD+6aefXMmDvD755BMX0Xj33XeBYEmoXZlYSNrC8DfccENGXoXdeOON+ZbpXnLJJUDBy1ntGLZp0ybX7dkS1YJwGa/v+24qy5bJ7rffflmzn51N/7733nsusXrYsGFAmAA7ZcqU9DQujfbZZx8X/bDUiEyZKi8JW9Tz4IMPutus6vdBBx3kyoJYcn1BRfrSoU2bNi6CP3/+fCD+8nEbDzRo0ICjjz4aCEsP3HTTTS7CE7u7AiTnfa0Ij4iIiEReQiM8Nj9nuQKx3n//fYBiR3es+ND48ePZaaedgHD5czbsRRW7HB2CJZSFLd+OiocffhgIcw6szLvNy2aClStXAtC2bVtXnj6v1atX51v2WaVKFbefkZU6t6JanTt3diXeMyF6YFGNvn37uuRjW9JqEasDDjggbl6Z7Z9mf2fJsVZkMZ0aNWrkrnYtuTPesldbsjx16lSX6Dl06FAguHLs1atXKppbZl999RUQRDEswmP22GMPICjAZ0uaLfpTp06dQosqmuOPPx4IIprZwL4Xpk+f7m6zz2Q25ZjZHnbPP/88ANttt51b4HHnnXcCQaKy5UI+/vjjAJx77rlAyYsNJsP//vc/tyWGRXFOPfVUtxeWsbyeX3/91c18xO6jaMvRbZHA6NGjARKesAwJHvDYyb9GjRruNgvVTZgwoVjPYW9oC2cdfvjhrg7D5ZdfDoRJe5ksb1XUF198kXXr1qWpNcllib9XXXUVe+65JxDWzrD9cDJxuue3334rdA+pvDZt2uT2fLNaGRZyPvjgg90gaPDgwQluacnF1j+yLzwbkA0YMAAIEuvt4sG+HNeuXetCzPZ3xa2jkQovv/yy65tdQNxyyy1ceOGFBf6NvfesH6NHj3Yn13gVfDPRE088ke+iwQY3ffr0iVsfqDgDnjfeeAMI9uOyaYZMZOcYq+HVpk0bdxFtA99sYCsm7cvcBi69evXKdyFv0+ZAvhVsgwYNYvPmzUlrZ3G89dZb7uLDVhEuX77cVYf+/PPPgXDAU1DdILsgsX00bTo6GTSlJSIiIpGX0L20LIIRG92wq7HihuDatm0LhFNgECYpJ2o/jVjJ2ofJRq19+/YFglCkRQcSKdV9tJokFStWdGFJq2Rr4VkIE2MnT55c5tfMpL208rIaEgsWLHBX0pYIbRXFi5KMY2hJ49dee60rkWC7hsebWrX7PM9zjzc2DX3JJZe4fbVKKlF9/OGHH9wVZCy78h8/fjwQLq2vWbOmO0YTJ04EgqkP25MokXV1kvlZbNKkiZt+tP7bMcs7hWCKuj/2MT/99BM9evQAwuTueNLxWaxQoYJbcm6zCJs2bXJJy4ncuT7Z51ObirNoqy29LuzfHMIprf79+wNBpWaLLpdUJuwxaU499VQ3m2OLKxJRvkV7aYmIiMg2K2l7adkydKuuWxRL4sqb6/PXX39l5Z5bdlVpV/3ZWhXV8lIs78MSzMqXL+/yX2Ir8loBt3nz5qWymWkTW1gxkyoT25Xk9OnTXSVay/MoLOm/efPm5OTk5LrNorMlKSeRLJ9//rkr5hnLkiYt18P2nerUqVO+XJZRo0alvGJyWa1YscIlFlv/LXJTUJTe7rdCrXZOit3f0B5Tp04dV6DS9rSyz3K69ezZ00V2LA+kX79+CY3spELTpk057rjjgDB3qrjJ4pZcbrk/55xzjlsgkg37+OU1YsQIIMjVXbJkCVD8sUJZKMIjIiIikZe0CE9JrgorV67slt9ZWW1b5TNp0iQ3954tDjvsMHc1bf3PlKul4thvv/2AIP/GrpTjXUVaH2NzBWyVnS19tuz9bNtHrCiWP2BLKCHM2/r111/T0qaC/PHHH0DxyjnEW+1jV/yZUA5i0aJF+fY/gzDqEW+11k8//QSEK+pseX62OeGEEwCYM2cOEBY3LYr9ezVo0KDQx+2///5AmHdpq2bSLXZ12t133w2Ehe6yyWmnnebyW61sSUE7wedlK7JspWHXrl3dd2Wm7a9VmOrVqwNhHtq6devcjvG2xD2ZkjbgKU7VS9ur5/rrr3cHz2qYXHbZZUC4oVg2id0w1Gq+2H4+mcymrSy5uk6dOm6gY8t3Y5c7m8JC6/ac6Rrw2AmmTZs2pV5ebc/RpEkTt2HhwIEDgbAEw+rVq139jIKWX2aD3r17u9+tBkwmlVOYOHGim0q1BOVq1aq5Kq15E8VXrVrFDTfcAGTPEvSC2IDTppmt3llRiyFip7AK88033wCZM0ViF8GDBg1yKRJW5iIbVa1a1S3VtvpkJWWpAhdddJGrw5RNAx6rtm/n0WRsEFoYTWmJiIhI5CU0wmOVLqtVq+b2N8mrUqVKtG/fHgj30+jVq5eL7PTp0wfIjEqSpbXPPvu4RMlsudqvXLmyS1q0aY0FCxYwadIkAI466igAzjjjDPc3Nk03btw49/9Wbdv2YZo7d24KWl8wS9idM2eOK1hpSzwXLFhQYEi5atWqroCgTZPERu6MRQ1OOumkSEzbWWE0CI9dJlWw3bx5s1vGui3uIQVh+YcrrrgCgE8//dRd7ZfWmjVr3NRRKpJHC2PnGktUXrRoEd27dwfil1TIFvXq1XN78Vkpj5JOG65evRrIrM9kcTVs2NAtx7fpq5tvvjmlbVCER0RERCIvoYUHLYnupZdecvPNlhNgV/4XX3yx2/fFbNy4kX79+gFhyf5ESFeBpZycHLd8e+rUqQCcddZZiX4ZIHF9nD59OieeeCIQbIMBwRX0+eefDwRbfORlORS2b1OyJKLY2bBhw1zSquXkrF69usCibBUqVKBOnTq5bluzZo1LxrZomBXJsr2bSiOTCoF9++237LLLLkCYf5WIrVwyqY/Joj6WrX+dO3cGwnw/S9Rt3759gTMGiZbMY9isWTO3zYJFhm338NWrVxcatbE9/6xEy8CBA92edyWNdKb6fWrFFe+66y6XrGzH2vLGEq2gPiZ0SsuqRY4bN859IXz99df5HmdfMhbW6tmzZ1Yk9RbFpjxsBQ8UvTIiU8SuImvVqhUAzz77rJsSsoGxPW7w4MFZtVJixowZbiBnX+QHHXSQq4q877775nr8+vXruffee4Fw6nX9+vVp378mWbp16wYEqyfWrl0LBFMlIqmw4447uilUSwMYMmQIQMoGO8n2xRdfuJV2tsDBkpe/+eYbVy/JVnnWr1/ffZdYFWILHDz33HPcfvvtqWt8GdiAp3Xr1q7OXrrqYGlKS0RERCIvoVNasWx6y6a0zj77bCCYFrClaVYpMlnSFWK+7rrrXH0Wixx89NFHiX4ZQGF0iH7/IPl9tDIQY8aMcVPOiUzCzoQ+Jpv6WPz+We0um9q499573VSqJU/Pnj279A0tpVQdQytnYfV42rRp4/Z3q1u3buxrAfD9998DYZLv1KlTSx1tTlUfbYGL1dlZs2ZNrtmPZNJeWiIiIrLNSlqEJxPoiisQ9T5GvX+gPmYD9bH4/bPkVdtL6t1332XYsGFA/uKRqaRjGEhEHy2P1/bJfPbZZ1NWOFIRHhEREdlmKcKjPmY8RXjUx2ygPka/f6A+ZgNFeERERGSbpQGPiIiIRF6hU1oiIiIiUaAIj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5FQq70/M8P1UNSQbf972iHqM+Zr6i+hj1/oH6mA3Ux+j3D9THbFBQHxXhERERkcjTgEdEREQiTwMeERERibxCc3hk27T99tsDcPTRR1O+fHkA7r///jS2SERKq1y54Lr2wQcfBGDevHnMmDEjnU0SSQtFeERERCTyFOERp3HjxgA88cQTAGzatInOnTsDcOihhwIwd+5cAD799FP3d8cccwwAb775Js8//3zK2ptIp512GgA77bQTAKeccgovvfQSAG+88QYAd999d3oaJ1IGTz75JABHHXUUAPXq1VOEJ0NUqVIFgHHjxrnz6B577AHAqlWrGDp0KIA7F0nZKMIjIiIikef5fsHL7ZO9Fr9du3YANG/enMLa8fnnnwPwwQcflOj5M6HewCuvvAJA165dmTBhAgBXXHFFwp4/kX288sorgeBqI85z2Ovlu++bb74BYN26dYwcORIIrk4AVqxYUZyXLlSyan9MmzaNk046CYCKFSvmfU7XV/v5999/c/vttwNw7rnnluYl40rV+3TEiBEA3HXXXQAuP6u4unTpwmuvvVaq186Ez2LdunUB6NChA/369ct1m/1/7HH/7LPPgOD4t27dusjnz4Q+xmratCkAy5YtA6BSpUoAHHjggbz++uulek7V4UlsHx966CEABg0alO/c6nke//zzDwAXX3wxADfeeCMAW7ZsKfVrZtr7NBkK6mPKp7TatGnjvhT79OkDQMOGDQsd8Pz0008AzJ49G4CxY8eyfv36JLe0bGxQ07Vr17S2oyQ2btxY5GPsRNm0aVMaNGgAwK677goEH9AXX3wRgLVr1wJwyCGHAPDFF18kvL1l1aNHj3wDnYcffhiA1atXu9vq1asHwMCBAxk9ejQA3bp1A2DPPfdMRVMT4uijjwbiD1oL06RJEyCYGrn55psBmDhxYkLblkyXXHIJACeffDIQTN3av0G8gXzegW6LFi1S1tZE2WGHHdz50gY6duzefPPNtLVLAp06dQLg2GOPBeJ/Jn3fdwnnkydPBsLzUrZPSdoF4/7778+///3vXPfZoO68885L+OtqSktEREQiL2URnp49ewLwwAMPUKtWrRL9bf369QE49dRTgWD64eyzzwaKF5VIhy5duuS77fLLLwfg1VdfzfUzU9hVw5dffgnA6aefzj777APA7rvvDgTTVhAk21l0xCJ2y5YtY/z48UCYeGeJvr1792bDhg2p6EaxnXjiia6djz76KBD27++//3aPq1Ah+JgsX76cyy67DAj7Z1fN55xzTmoaXQr22evQoUOp/n7UqFEA1KxZk9q1ayesXamwZMkS1++8UR2AX375BYA1a9YAQUmGO++8E4BrrrkmlU1NqMsvv5w2bdoAweIDCM69ULbpkFSw5OqPP/4YCCKTu+yyS4GPt/OPfU5j2fE94IADWL58eaKbWmq9e/cGwjZ7nue+H/7v//4PCCLLs2bNAoJZEAintrIxwtOoUSN3nhwzZgwQRBsHDBgAwJQpU3Ldt3jxYndeThRFeERERCTykp60bJEOm0+uVq1avNfh/fffB2DmzJlAMAcNcOmll8adZ7dkrxNOOKHA105XctYVV1zhRuuFsQjPhAkTSh3tSXYfO3bsCMA777xTrMfvtddeALz33nu5bu/bty/PPfdcqdqQKYmSjRo1ckm7ltdiuUqdO3cmJyenVM+bzGPYpUsXtyy5Zs2aue6Ld0Ucq1WrVgA8++yzQJCrddtttwElj2il+rN48MEHA0FUw8otWPLu008/zVNPPQXAzz//nOtn1apV3e8llQnJoBY9//jjj11UYPjw4UAY4SmLZH0Wa9asyXXXXQfAsGHDANhuu+3iPT8Av/76q/vOiOeggw4CwsUI3bp1cwtICpPsY2jL0BctWgSEC3c8z6N58+ZA7nzHvOfTP/74Awj69+6775aqDal+nzZq1AiAG264weUuPfbYY0DuPB27z3LMxowZw0033VSq10xp0nL16tV55plngDBpt7Aw6s0335wvQckGPL1793ZfurHPkcnJwMUZ7EDYh9deey3jprdMcQc6xla95B1IL1myJGFtSpeVK1e6k7Kt1rL3adeuXUs94Emmtm3b5hvoFJclo9vPTGbVwfv37w+EIX/f9920hk0jfPvttwU+T6ZOkRfXoEGDgGAKxAZ4L7zwQjqbVCgbmN57773861//ynXfb7/9BoR1hABuvfVWAH7//fdCV4DaKlE7H2UKG5C2bNky1+2zZs1yqQSxbAWsqVy5MhDWC8tksQMdCAY0xx13HABvvfVWvsfbhUkyaUpLREREIi+hEZ7DDjsMCEJRFlK0qEzsFf8PP/wABEvSIBit52VTBd26daNXr15AWD+kSpUqGZ94VxKJrMuTTu3ateO+++5LdzOSyipOm82bNwPw1VdfpaM5BbLQf+yVpCWNL1iwoFjPcemll+a77aqrrkpA6xLPIju251vs0nJLtD7llFMAXGJ9FF1wwQXud0sPsLIemcQiOzbNXbVqVXefLRzo0aMHUPIo83HHHeciO9b3r7/+umwNTpDvv/8egIULFwJw+OGHA8F5pKTlIjKdRXZs2fn+++8fN7JjU1mWoG1TWqWdziqMIjwiIiISeQmJ8NgVlBUMKqwq6dKlS91SXhvtFmbdunWuGNy1114LwM4775yrGBzAI488UsrWp57l61hRvqgYPHiwK0YYJXYFMmbMGI444gggjJacfvrpAMyfPz89jStAnTp1gHCPMAhzy+zzV9zniFXahN5kOvjgg3Pl7ADccsstAEyaNMldKWZ7fk5hzjzzTCDMm3jqqae4/vrr09mkAnXr1s0tN46N7FiulRWkLWlkx1iuFoS5XImo+J5IixcvBsIIz6BBg1x0LvYzVtr8u3Sy86VFdixiEy+606hRI7cMfeXKlUC4LD0ZFOERERGRyCtzhKd69epuFB0vsmNbQFhUY8SIEW4kX1KW17Pzzju7JYt5s92zge2pFRW2is6KFMa66KKLgMyMDMRju8PXrFmTE088EcBFdWrUqOF2LbarkKVLl6ahlUWz7RRixe5wHyUtW7bMtx1E7FWi7TgdZfY5++uvv4BgCXps8cxMctZZZ+WLXPzyyy9u6xOLfpSU7Tbev39/l1tnsw6ZxnKXxo4dCwRL8K2oYOyKZbvNWGTZ8mAzzaOPPppvqwhbgh5P7NYShUWCEqXMA56TTz6Z7t27F3i/Vdo9//zzy/pSLmHSprggTKzMhsTfTK2wXBqVKlVySaA2fVCuXDn+/PNPINyA1BLXMp0N2i351ZY5Q3hyuuaaa1y11kwdwDVr1gwIvlTyiq0wXBz2+JL+XaotWLDA1TU58MAD09ya1Js0aZKruWPTWFb3LBPZoAzCEgF77723W4ZeUvb+tIq922+/vZu2jd0TL5O8/fbbQJioe/zxx7vaVrb454033nDLuI0tjvjwww9T1dQSOe6441ybrW82oIlNYbG+2sKlVNGUloiIiERemSM8Xbp0iXsFaBVpExHZMbaUMfb1krGjarLYv0k2syTdo48+2i3RtmmEP//800XcsiWyY6ziaWxkx9iOxTk5Ofz6668pbVdpxVviOn36dCBM3o2tlmy3xb5H804TPf7448lpbBnl5OQwadIkIIzG2ZTe1VdfnbZ2JZvtgn7ooYe696hN5WSy0aNH8/zzzwNh9eeyTL/ZwhWb0gKYM2dOGVqYOraooFy5cq4fe+65Z66fsQrbWSBT2NSUTSfa9HJsNCf2MbH7aiWbIjwiIiISeWXeS+uff/7JdzX5ww8/uH144hUVLCnLFbF9fMqXL++Stgp7nVTvGWJ5RAVtLWHJyonMN0p2H23PqB9//BEI91XKW4APgu0WRo8eXdqXKlAq9tKyHYrtvVypUiW3ZLR69eoAvP/++y5HIpFlEBJ5DO142ZJe2/qigOd0/bVkyEWLFrkIquXDxC4dtrIDtrt4cSX7fWqLFyxyYNthXH311W47gpK2uaRSfb6ZNm0aEEZdITwPCif4GwAAIABJREFU9u3bt1h7R5VUpuxrF8uS8W0vqldeecUVq7WcwuJK135o9erVc8fTdouP/Xya/fbbDyj9kn3IjD3fzLfffutKKVgOViJ2SC+oj0kZ8KxcuTLfviglZeHa9u3buy+j2L02LEmxsD21Un1gi6qUmYzkz2T2cezYsYwaNQoIV8jtscce7n5LoLPN/r788sukfKmk6yRrK7b+85//AMF70QYGljh/6qmnlvl1knEMbfq3bdu27jYLn9sxjHdCtdu3tivffZm+eajVDrKpuRYtWrjzh4XOk5Vwnqo+li9fHgiSWgH23XdfrrzySgBGjhwJBIMAew8kUiYNeGwKyCpK2/u2R48evPjii6V6zlR/Z9gK0GuvvdZtFGo2bNiQb4rdFk20bds2V/J3SWTCgCfvRqEQfr9bPZ6yKKiPmtISERGRyEtKhOfHH390U01Wh6cwFtKCcDmthah79eqV74pz9erVjBgxAghD2PEowuNet1R9nD59uqtFk/cYTJs2jYceeghIbt2Era+Z1qtKu5KcPn26u+KySI9FeF544YVSL6tN1xXXLbfcku8926VLF5fAnXe/ugULFpS6Oni6+rhkyRI6dOgAhFMfXbt2TUskEhLTx+HDhwO4fetefvllV13YIgCNGzd2SaKJ/Hym+7Nomjdv7iJcVul/8uTJQFiTqDSSfQyrVKkChMnHVtKjQoUK7rvynnvuAYIyGFOmTAHy15I68sgjC/3uK0wmRHhil65bRCeRu6UrwiMiIiLbrDIvS589ezZ9+/bNdVuDBg14+umnAfjggw8A3Bzz8OHDc0V0AM4999wS7RTbv3//UlfjTIZsKHpYGvvss4+L7NiyV7vqHzlypMsXMBs3bnRXV5Y3UdqoRyaxBOVVq1a5/dz23XdfABflGjlyJHfccUd6GlhKZ599dr7b6tSp43aYts+k7UPUr1+/lLUtUSZNmuQKlrZo0QIIikz27Nkznc0qE3vvmRkzZrhFBLvssgsQfO6sqF+UWIRk/Pjx7LjjjgD88ccfAMybNy9t7SqOypUru2X4xx57bK773nzzTVfSIzbZ3B5vEZ5MLwJaXLH9L6wSc6IpwiMiIiKRV+YIz8KFC90yuli2esquPApb1VGuXLl8+QKx3n//fSAsub1u3brSNjdtsnE7Cd/33VW+HZ/CInFVq1Z1S2SPPPJIICz89vfff7sr07p16wLBKpPSzkOnku3btn79enbaaadc90XlisvEW8Fkq0Gy8XP31FNP8dRTTwHw4IMPAjBkyBC3HU6mRwXisRWsZsWKFa50h0ViFy1alLH7LZWF7Sg+ePBgd9sZZ5wBkJRl+InUrVs3VxzRzqOvv/46EERP4xU1fe+993L9f0lmQjKRze7E7reVyuLBZR7wTJ8+nW7dugHE3VOrOF+UW7ZscffbxqJPPvkkEOyfZbUlMvWE26VLlyIfU9pkz3S67bbbXN+GDBlSrL+xJc/16tUDYOrUqUCwsawNDmw5+z777OOqiV533XWJa3gJ2JeH1doxTZs2dQnbNkCzzQ0hfF9bguHChQuT3lYpm9jK0TZVUL9+/XQ2qVTyDronTpzIAQcckOu2JUuWpLJJSWcDHZv2gaDuF8D999+fjiaVSt4LJAsExA527DGdO3d271Njic2WnJ5t8u6dlcrpLNCUloiIiGwDyhzh+f33313hOatouvvuu7P33nsX+DeWZGbhugULFvDMM8+45wP45JNPytq0lCms+GE2u+eee9xxsaup2KquNm0Vr5pvrVq1cv2EcJ8fu/o87rjjWLVqVRJaHp/ttxRbOXj33XcHKHYSq/XBEpStmF1UdOnSJV+Cuv1/punYsSMQRGzefffdIh9v0wdDhw51UbtsZNW+e/ToAZCrwKBd+ds+RlFw5JFHMn78eCBYvg2wdOlSF+3Jpmkea6tFcaxA5pIlS9x99nnr0qVLvr5dc801QFDkNRvlTdZO9f58mXkmExEREUmgMkd4IMy7sTyPqlWrur1M4rGibdmQsLqtswJt9jO2iJkV/LJikdOmTXOl/e2KzOTk5Lh8Hks+X7lyJXfddVcSW5+bLdm1olflypVzV4x5+b6fr3T7zJkz3dX1Z599lsSWpk///v3z5d1ZSYlMY/kNLVq0cAsbjJXF6NevHzk5OUC4N1hszmA2WrBgARDmx5199tku2dze23aOzWa77bYbECSb5y34OXbs2IzN6SzI0qVLXeJ8//79gTB3sKBZAks8tz0KZ8+eneRWplaqSyeUudJyJktVRcnCNg211VnJSlrOhKqZyZas6q5t2rThkksuAcIB3erVq4Fg5Yvt0ZNsmXQMb775ZjeAtXODVfEty4qmZPTRpqUuvvhijj/+eCAchMdOy8WborPfW7duDeAGRWWRSccxWVJRadkGN3PnzgXCgSoEAx3AVSBOtFQdQ5tCj61tZbsHWEXwp556irvvvhtI7MAgne9T68d3330HBBuGJmLvrLxUaVlERES2WYrwJKCPFo6MrQNhkZ0JEybk+v9E01Vl9PsH6Ynw2MKBPn36APDNN9+U+nmT3UfbL+uGG24Acl8l235ndjU9ffp0V5tn/vz5pX3JfDLpOCZLKj6Ldgxja7fNmjULyF1/Jxl0DAOpivB07tw5GS+jCI+IiIhsuxThUR8zniI86mM2UB/L1j9bVPDxxx8DUKNGDQB++ukn2rdv735PJh3DQLL6aDvcG0V4RERERBIsIcvSRUREysK2zLBSEVbupE+fPkmP7EhqJCuiU1ya0lIfM56mtNTHbKA+Rr9/oD5mA01piYiIyDar0AiPiIiISBQowiMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEngY8IiIiEnka8IiIiEjkacAjIiIikacBj4iIiESeBjwiIiISeRrwiIiISORpwCMiIiKRpwGPiIiIRJ4GPCIiIhJ5GvCIiIhI5GnAIyIiIpGnAY+IiIhEXoXC7vQ8z09VQ5LB932vqMeoj5mvqD5GvX+gPmYD9TH6/QP1MRsU1EdFeERERCTyNOARERGRyNOAR0RERCKv0BweKbvDDjsMgJEjR7LddtsBMGHCBADeeeedtLWrrM4991yOPfZYADp37uxu/+9//wvAJ598AsBtt90GwPLly1PcQpFtV6dOnXj22WcBOOiggwDIyclJZ5NE0k4RHhEREYk8z/cLTsZOZab2jjvuCMCUKVMAOOmkkwBYtGgRffr0AWDt2rUles50ZqPvsssuAMyZMweAPffc09336KOPAjBo0KAyv06y+1i5cmUA9t9/fwDGjRsHhJGrWJs3b6ZixYoAlC9fHoD169cDcOGFF3LnnXeWqg3JXBkyfPhwAK688kogPG55nh+Axx57jPfeew+Aa6+9trQvmc+2vGoilvpYdnYeffXVV905Z8SIEQDcd999ZX5+rdJSH7OBVmmJiIjINisjcni6d+/OrbfeCkCzZs0A2LJlCwAdOnRwt2V6zovl6JxwwglMnjwZgB122AEA3/exaNorr7ySngaWUOXKlVm4cCEQHIe8/vnnHwCGDRsGwHPPPUfbtm0BOPvsswE45phjALj++uv58ssvAXjppZeS2/BiatCgAddccw0AO+20EwAff/wxEOQgtW/f3j0Ogr4cffTRANSoUQOA8ePHA+G/RZQdccQRLv/MIn7ZqEqVKu6YWv7ZpZdeCsCsWbNctC8bj2mPHj2A3BFlEQmkZUqrXr16AG5QMGjQICpUiD/2uvjii7nuuutK9TrJDt0dfPDBQHhyOeeccwBo2rQpb731FhB+Mfi+z8aNGwGoXr16aV8yn2T2sW/fvjz99NNx7/vss88YOnQoEH8guvPOOwPw2muvAbDbbru56S0bXGzevLlY7UhWGH377bfnjTfeAIJjBnD++ecD5Jp+swH3Pffcw4EHHpjrOerWrQvAr7/+WpomAKkPMZ9xxhkAzJw50x2Twtg0yYIFC2jevDkAhx56KACvv/56sV4zE8Lo/fv3B4Jj3KlTpwIfZ4Oh1atXl+j509nH+vXrA/DVV18BwcXKiy++CMDAgQMB+O2338r8OumY0qpcubI7j55wwgnWDnr16gWEn8ElS5YAsGzZMr755hsAJk6cCBR/8JrOY9ioUSMATjnlFCD43PXr1w+Ahg0b5nv8rFmzAPjuu++AIN3g77//LvJ1MuGzWLt2bSBIXbH37pgxY4Aw2BFr7ty5AAwYMMB9jxZGU1oiIiKyzUr5lFbPnj256667gPijVmNX3rfccktK2lVSEydO5MILLwRw0SkbaY8dO9a1+48//nB/k+lTcnnVqFGDn3/+GcAl6y5btgwIrpLjjcTN999/D8Cnn34KBBGUatWqAWEScLpt2LCBvfbaK9dtNhW3/fbbs2HDBgC++OILIIhw5I3wZANLxLZlyi1btgTg3Xff5e233y7y7y1BvXXr1m5a1pLSM8Vuu+0GhEuwLdLRvXt3t+ihTZs2ABQW1X7mmWfYtGlTMpuaFEceeSQQLjL4/fffOe2004DERHbSoXHjxgDMmzfPRRbt3BF7DO33jh075voJ0K5dOwAXKck0J598MgCDBw+mVatWANSpUwcI+mp9i/eeHTBgQK77li9fzt133530NpdWw4YNXfqDRZltJgDg66+/BsLvDvsbwEXzevXqxeOPP17qNijCIyIiIpGX9AiPXXFYQbqOHTsW6+rQRumXXnopkyZNAsioK69Zs2ZRpUqVXLdZrlFBc/8WHckWM2fOdHOnVhKgsKhOFNiy+jZt2rB48eJc99WqVcv9bseyOPPJ6dS0aVOee+45AHeVbFeGxYnuAHGvqEpaIiKZzjnnHJd8bgsHLGcj3rlm3bp1LvI6bdo0AFauXAnAjBkzsi5ZuWrVqlxyySVAGAHJyclhxYoVaWxV2R1++OFA+L4tjUyNqr/88stAmDRfsWJF/vrrLwDWrFkDwMKFC1m6dGmuv7N/k2bNmrlcSFPYjEk6nX766UBQmmTXXXcFwqjUd9995/Jdzz33XAB+/PFH97eWT2ezJ7vvvnuZ2pL0AY8lCu63334l+js7cY0bN47WrVu73yEzKoYuXbrUJbhGWWmTcW0wGPsh/Pbbb4HMXv1iK8gaNGhApUqVgLBWz4gRI9xJ6ayzzgKKn3idLjNnzqRFixZAUEcIwunJknr55Zfd9JbVdIm3ei9VatasCQTTkHa+MPEGOvPnzweC4xgbNs9WNrgZM2ZMvi8SW3WWzWxFJIRfeHbOjTfFYysIbcoW4g/W02WHHf6/vTOP03Je//h7yhKTooOTPQon+86xRHZaSCH7STiIHEt0EmVJOMh+OCWUKEvWIjtH5IRIOJFjT5ZI2X6l5vfH8/p873tmnplmeZb7uX3e/8zM8zxzz/2de/t+P9d1fa6VOfroowHYY489gGgc9913H7fccgsQFXpkQxWEkOz7KESLCSWax6/RU089FYAHHniAuXPn1riNCy+8sNLPBx54YFjcNASHtIwxxhiTevKm8Oy3334A3HTTTdXeGzVqFACDBw8GMuV3Cn1JzlKS0sUXX8xBBx0ERN4ne++9d752O2ccddRRxd6FoiI/EHnZAMHTZ+HChUXZp7ogZaB58+Y8+OCDQDSWioqK0Bvs+eefL8r+1ZXRo0cDsP322zNmzBggctxt6P9/r732CivSpHgp1YTCdU8//XRwO68aoix1VCwRX/XLR0pqVikzZ84cIGMRoQKRH3/8sdrn9Fy44YYbwmsPP/wwkLHPKDZSuy+88EL69u2b9TPNmzcPKlapoxDWSSedVOn12bNnh9L7utCrV6+wDamZTZo0TqOxwmOMMcaY1JMXhWfTTTcN5XGKs4uKiorQS0rmUPoaR/G/qVOnhm3J6G/XXXflpZdeyseu5wyVQ/7eUJnhpZdeWun1BQsWcNtttxVjl+qEcs3Gjh0LkNUIc/78+Ykpqc9G06ZNg8mcFMY+ffpwxx13AMlW1hrCDz/8AMArr7wSSpiFVGQpXQCdOnUCYIsttghJ54888ghQe6l6UpFZJkTKh3oQpgEVq8ihvSoqMJASqyTeefPmcfXVVwPJOK7nnXceEFleQHVrjgMPPDAYKO6+++5AZUuTqrRq1SpsY9q0aUDGzb7YrLnmmiGqo//9+PHjgSiXpyaUM6nozhVXXFHt+EmtbShWeIwxxhiTevKi8PTs2TNU52hVKav+GTNm8Pjjj9d5W6+99lrIzJ80aRKQmTEnXeFRXDmOzNDSjMzO2rdvX+n19957L9F5L8pJkX1C+/btKS8vByIb9BYtWoSY8uuvvw5ECkISyvUHDBgQ8uLEnDlzEmXnUCiUa6WvkN20TsZvuegkXmgOO+yw8L1MMvW1b9++4fzVvVitF8aPH581FyZp1KTsCN1rVPEkBg8eXOeWJ4VACuvMmTO5+eabgej5ED8XZcWiFgvZqpEUORg7dmz4XalZSbjOZXkBhOorVVfXtH+yHbj77ruB7NGRt956C2h8H8qcTngOOOAAAM4+++zwmkqR1WcqF6ghZ5KR82tcukxyOCQXtGzZktNOO63Sa5rkHHHEEUXYo7qjnlI6hyHygFCZa69evUKoaOTIkUAkOysUVgzkFq2yXIgSByX35wqdw0k6l8ePH1/pRlsfFDpXMmSSw65CpdbyZIGoP2FdvL5WW2218JAsVXbeeWeGDRtW6bUhQ4YA2Qtlion8kD7++GOeeOIJICqzVxgHMkm9ECWcr7DCCmGSoEmdvN7atWvH8OHDgWSV3sdRz0j1BpszZ054TQUUQHDgjzfaFrLQkB1GXXr/1YZDWsYYY4xJPTlVeCTFLb/88mEGq5Ly3yvxXihKskwro0ePDrN0KQAqC/3qq6+Ktl8NRRYJ+vr6668HwzolIkqG/f7770PItdB0794dyJxjCpuqX10uOPTQQ8P3tfX2KRavvfZaOL+kdIiPPvoo9BATXbp0Yf311weixFc5FU+YMCGUQycN9ViSsqPVcn3ZZZddSlbhkQntPffcE1Q5JcVefvnlQDLCyzWhvmZdu3YFCAU8LVu2DKFH2Sc89thj3HXXXQAMGjQIiFIFpk6dGlTcJDFs2LCQgqIu6HGDXh0bqVmrr756MCTUM0NpMC+99FIwn2yssiOs8BhjjDEm9eRE4VEPjI4dO4bXZJOdhDYQSWDevHk89dRTxd6NnKIZuWK0MpsEQnKelJA0MH/+/GB1rq7p6oXTv3//kChZqIRQxf/j3aHjpcqNRSaMVVUTiJK7k8DHH38cVsdKKn/55ZcBqvUigsz9SgZ96qDepk0bANZbb73EKjzffvstULu9gM69Bx98MLQSUSKzSvUPOuggdtppJ4DQxyjpKCdS+S2tW7cO+Up9+vQBkt/XLo56aSnx/KGHHqrWm3G//fZjzz33BDL90iBKPK+aqJ0k1O9K5eVxlJOk8/S+++4LPQp1XivfV4VOuSQnEx45feoBOGXKlJxK6u3atav0c1WJuhRo2rRptX4/pc4uu+wCRJPbOMqqT2I1iCYKCxcurHdoRr20JEkrhNShQ4fgn6FmnflGfy9b36hjjjkGyEjo9fGuOOaYY0KITB5a8r+KoxtxscJ4NVHX+478dzbZZJN87k5eeOihh4Ao8XPWrFkhaf76668HMpMjnRe6P8sja8GCBcyaNaug+9wYWrVqxYsvvghEFWdvvPFGcEBXs81SRBOfTp06BXdoVXA1a9YsPDPUxFmTgdo8eoqNJp7Zkqk1cdPiP96QWZWS+ZjoCIe0jDHGGJN6Gq3wHHHEEWHWJgYPHlxrB9T60LlzZ6677jogmjlee+21Odl2IZk9e3bOEq/yjfqatWrVKoSpevToAUQhAIiS0uL+JpLNJacnCSk777//PpDp89XQbvBKPtQKrVu3bkEJKZTCI3lfCbtyXAW488478/I3P/jgAyC/q7BC8NFHH2V9fZNNNkl8z62qndF79OjB9OnTK32mvLw8qD29evWq9N6kSZNCeCzJKMRz4403hnuNypQ7depU0spOVV588cVwPUthXbJkSTjGAwcOBJbuTZR01BNtxx13DK/pmKqDej6xwmOMMcaY1NNohSdedq5Ss1yskJRUd8kll4S+RjKaKsWeQKNHjw7lzUlFeSlaTcSTYWsjngdz7LHHAtHqTP1jkpAIqlXF2muvnbNtxntuVf3/Kd8n3yh5c6+99mKbbbYB4OijjwYyx0Y9epT/IEXuiy++CMaQb775ZtieXKQ/++wzIDKPXGuttUIpbFpdw6X+JZmtt94agMWLFwMwffr0cB4qgffss8+udp7LIkKJ90lFYznyyCOByu69Gl8S7ie5QDYew4cPD73esqFnaymiCNC4cePCGPXMmDx5cq3jzjVWeIwxxhiTehqt8Kyzzjrhe8Xg5s+fX+/tKG9EBmBqT7HccsuFKhOtnJPOiiuuGFbV2dBKW1VOQOgNVmgVSK0UBg0aFBQdGXp98skn4X+v9gUqx47z66+/AhklQJUTKreUUdaECRPCCnPjjTcGMqXrUhEKQdX494ABA4Khl0p3G7NN9YTRCrVQCo/+znPPPRd6zcSN5XQuqlWG+PLLL2vNg1DehM5XiMqDk9YXTaZ8MipraJXob7/9lrN9yjdS6jp37hzul6rciyPVSn3Dkm4VcvrppwNw1VVXhdfGjBkDlE4Z/dJQnzPZO2yxxRbF3J28ovPugAMOCMqO7h+HHXZYQXNbGz3hkfMsRDebSZMm1SnspDLJnXfeOVyw22+/faXPjBkzJoRJiolu/rvtthuQ8RxSEmy8iR9kJgyawIlBgwaFJmp6IMbL1CVZqgQ/32WHcr/UA7+8vDwkj+m9V155JfjMxHtMQSZp95prrgGiJN0ZM2aE7al8UknOm2++eTjZ5Tg9YsSIPIysZpSoqcnlmWeeGUKy8g1SuXJdkwNVng1RCEihhqSgSU0ukjwV7koach9WQYOSybMdxw4dOgS/nlJGZec6Z6siywD1WEv6ZGHfffcF4LLLLqv0+rx588JCOC3IQVkTnalTp4Zxa7HStm3b8L2KJEqJbt26AZEtAsDXX38NRM/MXBU31RWHtIwxxhiTehqt8Fx55ZVhlRxPWNWsTiGa448/PvyOZNcddtgBoFL4R/K8VmrqDltsDj74YKDhnXh//vnnGlf+06dPD8ZShTKUUgdwSauQSSqDSG48+uijQwdihS7nzZsHZFbUUoTiXHDBBUCUYK7u4gsWLAjbl9KjUFih0P9Wqsy1117LcccdB0Ty+dChQ4FMibfKvFX2PWnSpKDOqbxy8803D9uXfUIpJtVnQ4nP8Q7pSU28V+K2rjEdnzPOOCOcZ0oQveWWW0JYVeeiruts53TSUIK5OkjHUWj2rLPOYuTIkUDyFMdsdOvWLZTRS/nWs+D8888vaOg7n8hoT89MnX9PPPFEMNSV0tOkSZPwXCilUGvnzp2ByNk77iCtoopCKzvCCo8xxhhjUk9Zbdb6ZWVldfLdHz58OFBZxakPP//8Mw888AAQrbSz9cCpLxUVFWVL+0xdx6h8I6kDbdq0CX2LZASmMtC2bduG3AGN69Zbb61RvWnMqrmhY9Qxkz3966+/HvIgZCp38MEHh27SshpQrpV6FRWCpY2xrscwG0o0lpohRWrttdcOak7cWLEmnnvuuVAyq+TsupLL8zSXqBO8yoKHDRsWlJP6dqQu1BjfeecdIFIwP/3003BslQwb7/4uZSiu1DWUQo1R9xuZQFZUVAR1UV/juZW5JNfXohLin3rqqXDMhO4xypssBPk+hsrJUY6j7in3339/yLFSdKS8vDw8b1544YWG/slq5HOMK620UsjR1NhkGNyzZ8+CmbLWNMacTHiUvKtk1wEDBlRL2hUzZszgwQcfBKIHwxtvvFHvh0RdSOqDJJd4jPkZ30477RR8dfr376/9CO/rBqQJ7Z133tngvmFJO4aaBCqkp2t5hx12aHDScqHGqKR/9epZd91149vXvvDpp58CUbJzLnpLJe045oNcXYtKuNYDUMcBosR4LaBzsfitK/k8hm3atAmTGqVx1Pb8HTFiRJik57LiMx9jlG/ezTffHCp6NTaFKs8666z67WgjqGmMDmkZY4wxJvXkROFJKl5xZUj7GNM+PijsGKWKqHRWXiGDBw9u8DYLPcaePXuGr126dNH2gUwZ93nnnQfUP/xYG0k7jvkgV9fiiSeeCGQSyIVCHx07dgTgtddea9hONoJ8HsPVVlstjEnpD9mev+oarv9RrsnHGGUbcNFFF4XrTOq30iYK6bdjhccYY4wxv1us8HiMiccKj8dYCniMdR/f5MmTgSj3A+DSSy8FIjPUYpDvY6hy9PHjx+vvAZlu6VK7Hn74YSB/FiX5GGOvXr2ATDGMjuMVV1wBwC+//FLvfWwsVniMMcYY87vFCo/HmHis8HiMpYDHmP7xgcdYCljhMcYYY8zvFk94jDHGGJN6ag1pGWOMMcakASs8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvDFE7BsAAAWZklEQVQYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvV4wmOMMcaY1OMJjzHGGGNSjyc8xhhjjEk9nvAYY4wxJvUsU9ubZWVlFYXakXxQUVFRtrTPeIzJZ2ljTPv4wGMsBTzG9I8PPMZSoKYxWuExxhhjTOrxhMcYY4wxqccTHmOMMcaknlpzeIwxxpQOf/zjHwFYf/31w2sffPABAHPnzi3KPhmTFKzwGGOMMSb1WOEpEBUVFSxZsgSApk2bFnlv6s9mm20GwIwZMwBYb7316NChA0D42qRJE44//ngA3n77bQBeffVVAK6++mr++9//FnSfjfm90axZMwBefvnl8NrkyZMBOOCAAwD48ccfC79jZqmUlWUKiy6++GIABg4cSEVFRaX3nnrqKQB69OjB/Pnzi7CXpY0VHmOMMcaknkQoPNtssw1HH300AH/729+AaEZ7wQUXMHbsWABmzZpVnB1sBGeeeSYAS5YsCQpPqVBWVsbGG28MwPPPPw/A7NmzAVhjjTX4wx/+UO13tCKRIqSvXbt25d577wWgb9++lT5baJZddlkANtlkE7beemsAdthhByBS30488cRwDsZXWVOmTAFg2rRpQEa5Avjwww8LtPf5Ya211gLgiy++CK+1bt0agOnTpwOw6qqr0qRJ6a2RevXqBcDaa68NwN13313r8VpxxRUBOP300wG44oor8ryHjadNmzYAvP7669Xe033Tyk6yOeywwwAYMGAAQKXnhe5Be+21FwBXXnklJ598coH3sPQpq+2hk2/zIT0wp02bFm642Rg4cCAAo0aNAirflGujmAZL66yzDkCYrP35z38OJ+0555wDwLBhwxr9d/I5xpVWWik87NZbb72GbCIrurDvv//+On0+F2ZnLVq04KyzzgKgU6dOQGaiLb766isgCgm0bNmyTvumh8m+++7LJ598UqffqUqxztNmzZpx2223AfDrr78C0Lt37/B+u3btAJg5c2Z4raHh2EKN8cQTTwQyCyXIJPEus8wy2j4AixcvDg8TXZ/vvvtu2Ebnzp0B2GWXXQD46aefWGmllZb6t4t5v9lyyy0BePrppwEqLUY+//xzALbffnsgOtcbgo0H8zfGt956C4gWiQDffvstAM2bNwei+9P//vc/tt12W4B6h7ZsPGiMMcYYk2KKGtI69thjAWpVdwAuvfRSIFIGOnXqFEIrSWWnnXYColBJPGm5VGjWrFm9lJ1p06YxadIkADbffHMA9tlnHwCWW2658DkpB4Vk9dVXD6v+xYsXAxl15vbbbwfg2muvrfT5bKGbdu3a0aNHDwDOP//88BrAdttt12CFp1j079+fnj17AmS9nlq0aFHoXWo0Z599NhCFr7LRtGnToFQdc8wxS91meXk5Dz74IADdunXLwV7mlnXWWSeEsrKdt7omG6PsFBKFFONqoo6rwtHZ7qXrrrsuQEiPADjppJMAgpKZVLbccks22GADAL7//nsAbr75ZoYPHw5k7l8Q/R8OP/xwysvLgforPIVGVgl77LEH3bt3Bwj30bKyshD5ePTRR4FIpf36669zvi9WeIwxxhiTegqu8Gy11VY8+eSTAKyyyir1+t0tttgCgFatWiVe4RFacZWVlYXvc5G7kyS0wjjnnHN47rnnKr2nRGXN6IvFTz/9FBKNtRr+61//Wq9ttGrVKuR1iIULFwJ1zytLAloJn3LKKeG1W2+9tdrnunbtWunnJ554Ir871gi22morIJNYnQ8OOuigvGy3ISifY4UVVgAypcrZlB0ZDSbtftOiRQsuueSSGt8//PDDgezHsmohQTbi70lpT7rCs/LKKwdl64477gDgwgsvDO9/9tlnQKR+3HnnnUGBXbRoERDl+yQFKXR///vfgagIAKJj9P777wfVTrlzyrGU+p5LCjbhUTVL9+7dq1X3zJ07lxdffBGIQgv77rsvAP369asUDik1JL02adKk5EJaS0MS+aBBgwCqTXaSxJdffsluu+0GwC+//FLj5yQdr7nmmhxyyCFAJIuvssoqIQH2u+++A2DXXXcFKif2Jp0bbrgByDxQlJQen/DoQaMbkKhrknkxePPNN4Hopt+qVavw3pdffgnA448/Hl5TqFXFBbWxZMmSOoW+CoUqWffbbz8ANtpoo2qfmTp1anjAxBOyk0B5eTl9+vQpyN9SeKiUeOONN6q9tuaaawLRvejkk09mtdVWA6JzfquttgrnehLQNaPzcMmSJUydOhXIVJkBPPTQQyEt4L333gMI99077rgj55W8DmkZY4wxJvXkXeFRuaSS/dZZZ52w4pB0N2XKlOAGKl566SUgI2HKl6CUkJSaLaSVBoYPH07//v2B2ldRH330UbXX9t9/fwAuv/zy/OxcDdSm7PTr1w+ILBAUNojz6quvBqXy3//+N5CfxLp8IaddqadlZWXhGHzzzTfhc23btgWisn2FRvIhMecK3V+U+BlHDt8nnHBCeE1KnsIItVFRUZGIhHRJ/+qTVTW8CjBv3jwAhg4dyn/+85/C7Vw9WLRoUVjNKyy3aNEiNtxww0qfk4r66aefhtdeeOEFAF555ZXw2hlnnAFkrD+q8q9//SuHe54/fvvttxABUOn58ssvH8Z21FFHAZVL1qV+KESflJCWQm26l4q+ffvyz3/+s9rnZe2h81Xl9s2bN2fBggU53bf0PIGNMcYYY2ogbwpPVSOseExdyVgq9awNzfJLDcXZSzmHZ+7cuSE/R47RMqabOHEi//d//7fUbTzzzDNApKBAdvWkWMhQTol12fZNCtZNN90UetkkvRQ0G9dccw0QWQSMGzeOBx54oNrnpMBpBZmUlWNNbLDBBgwZMgQg5FiJOXPmVEr+FKWkzIndd98diJyjsyElIJ6vJKRm9evXj7333rvSe6ecckrok5dvvv3226CA6/pbuHBhsPAQMkysab+kDsWT78Xdd98NZLdbSCKTJ0/m/fffBwgJ3ccdd1wwi8yG7AaU5JsUZB8jJfLhhx8G4JZbbqnT7//0009AJofwoYceAmpX5+uDFR5jjDHGpJ68KTwyf4orO0LVH7UpPIceeigQdY4tNVQ+Wco5PEuWLOGmm24Colycuqhy2YgbTCWp2kcx4pEjRwKRkhVH9gmjRo3i559/BqJVTLaVdNJQLF190d555x0gYzz422+/Vfpsu3btQusTcdVVV1XbpvIMysvLQ45PsWjRokXo+1WV5s2bhzL8qnmCpYAUq3POOSeYXVblu+++4/rrrwcqWwdsuummQKQAqPqlqpICcNddd4XS/kKgVby+Qv1tD8477zwgUr7iSM3U9Zp0lllmmfDM0P0mm7qjKMENN9zAueeeW7gdrAft27ev82c322yzYEYo2xkpkXfddVeIMMh8uLHkZcKz2mqrZU2oEwofZEMypcrv6pJUmET0cC/lkBZEIcW77rqrUdspVqPQuqKwx4gRI6q9p3L2gQMHBgdf9WDSxarQbdLYddddQw8xobL0bIm4nTt3DtecvD/GjRsHZCYWf/nLX4AodHLVVVdx33335WXfl4YWRVdeeSUrr7xy1s80b948SOnyghoxYkRI9Ex6aEsWHpdddlm195Sg3KlTJ1599dVK72277bah8GP55Zdf6t/ZaKONwiRe3llJZtVVV+XUU0/N+t7kyZNLxiZiu+22AzL3E4WAsjFhwgQgWrzITiKJPPvss0CU1iEPq9tvvz00odbku3PnzrU+G7I1w20MpSU5GGOMMcY0gJwqPDLxGjduHDvuuGODtiG5dc8996z2ntwyVWaaZLKFtKZMmVLMXSoK2XoPJVHpkvSd7dzSa88++2xYYakPnBLyunfvnignYpWGjhw5MnR+lxu0EjrjtGnTBoh69UBUBi0lZejQoaGcW6Z3SuIuJLrPSJVbWr83/S90Lnbr1i2smLt06ZKv3cwJsu6II/O2oUOHAlRSd9SXcNy4cbUqOx9//DEQhU9atmwZEplLQeE57bTTqqVLyAi1X79+OUtyzTXqf6UiDoXlajLXVWhOqR25LtPOBwrzK/n6tNNOAzJGhPUx8Jw2bVrOrRWs8BhjjDEm9eRU4dljjz0Asqo76lA9YsQIfvjhh2rvK/EsW66IymKVmFc10TKJZMvhSQMyvornTPz4449AZO8fRzkI8aRlJQiXGh9++CEnn3wyEJ3PKhEeO3Zs6BCv3JdiIjUj3pleq/ps5l9SeNZee+1wnNZYYw2gch8i2RIUQ9mpipKRl0a2HAHdq6RmaBWahJyesrKycJ7FlW6NQ2aRKtmFqCO1VMZsBoxS+PbZZ5/Qf0mfb9myZUm08FH+Z9W8NIB77rkHILGGiy1atAgKxwUXXFCn35Hpro5XKaBn3uDBgwFC78zRo0eH/loi2zUstb1fv345L4jI6YSnNj8A9eqJNxATu+66a6j+kfweRzJ7oXwickG2kFapoMqQU089tZrHhZJ2Jc0CwY9nwoQJIalV4417usiTpxT72wiNVUn1ci9u3bp1eGhKpi4meqDHH/YqJNh5551r/L2KiorwO7/++isQhVD69OkTKryKiSaUSjA/88wzQxKn+hAp0XqjjTYKya2qUoPo/FUi82uvvQZEPX6KSZcuXUJ1ZBw9/LJVSuo9VWbFUVhETtnt27cPTSjlqg1ROCzJqGon7rekUFbSXZX33XffsGivyjfffMN1110HRAv8uvrWJJ2XX34ZqHyuye09WxqAJrX56M2YDtnBGGOMMaYWGq3wrLTSSqGHycEHH1ztfc2+s/XhUbfXRx55pJqyI/n98MMPD31XSolsIS11gk8qkhulxtXm6BpHyZGHHHJICE2qJDbuXKyEO4WDShkd1w8++ADIKDxJCluOHz8eyPQXUqflujhc//TTTyEZWNtIQh+pbKi8/oUXXgi2AFV9VyZNmhRCHUcccQQAQ4YMCQ6/uk6lHMyePbvRFgwNRerpsGHDqr03ffp07rzzzqy/t+KKK9ZYog2w1157AXDggQcCmfCl7D80/o8//pgbb7yx4TufZ+QqrHvNkiVLwr6r87qcipOGkv+zOUKrh91+++3HW2+9BUSl6hBds6Wsimdj6623BipHPpR8/9hjj+Xt7ybnDm2MMcYYkycarfBssMEGtbrNamau2DpEM96+ffsC2fN2pAZ89tlnldw4S4Fx48ZlzeFJeh6PlDopO2VlZSGPQ8ds4cKFQGZFqLJDxc7Ly8tZddVVgchsSixZsqTGFWopo7wkmRMmhUcffTR8Ve5K1ets9dVXDwnJOm5PPfVUVoWh2Gj/unXrxj777FPtfSUb6z6j8xSinAgpQgsWLKiWOC/FY+jQoUVTeKTExA3o1LNt//33Z86cOZU+L5Xq/vvvr5YMGkd9DeNI9RozZgyQcd+Wqp40OnXqRIcOHYBIWV28eHE4T+PJ20lEtgnKq4PofFXXAak7VVHHAp27pY7UZimqFRUVQeVSPp2S6/OBFR5jjDHGpJ689dKShXS2uKWqBrLFnVUFo74xygEqJeKVLlqRTJkyJfHGg9tss02lnysqKkJlnEoMv/zyS6CyYieTr/Hjx9doE/7DDz8EhSue6yUlolTzepQXkWRqstkfPHhwUE6EKs2Shio3svU6g2h1KDM+VcNku388+eST4X8Sr9wqNrq24khZPeOMM8JrKjmX0lVTW42qKBdy9OjRoSIoqQZ9kKnehYzlg/4PYubMmYmohqwLMimF6HmgZ1+8dYLydfR8TCNStOKmmFJ78qnsiLxNeOTXoiRW9X159dVXw80pjhKWlNgrJ9RSJB6+UkiradOmoVS2tj5jxaSqf0VZWRnbbrstEF2YVWV1iOTZuXPnhgdo1YlPq1atQhKs/jeff/55KEsslQmPQgdKuttkk03Ce/FJYJLRA7J3797hWKgce/bs2UXbr9rQxLt3797Bw0ShgnioWBMj9fp65plnQuNbFU6cddZZ/OlPfwKqn6ctW7YMbu+F9nNRUnUchdoa8nDX9aYS91deeQWI7AaSityf5Wqu/wFE+57NwT2pxPtB6j5a1Vpgs802C5P0+PNBKSFpoWqD2rfffptRo0YV7O87pGWMMcaY1NNohWfx4sXVDMpat27NhhtuCMDEiROBqKzuuOOOC2ErKQP9+vULM95SS1DORraQVikoGDp+SkKWuR7AAw88AES9o5o3bx6Osbosx92Us/H5558DcPXVVwMwatSogq02V1xxRZo1axa+j+9PTcgxXMmhAB07dgSgf//+lT47ceLEEJ5LOpLM11xzzeCSnc19OUnEbS30vcqQW7duXa3kXg6uvXr1CgqQ+hHVxvjx44vm1Ct1qm3btiGckw051atLfdyQ9d133wUyZm+6tpLYu642dP9Rb684UiJnzZpV0H1qDOpdd8YZZ/Ddd98BkZmulMi2bdtWO4fffPNNPvzwwwLuaf6p2iNz5syZ1awk8okVHmOMMcaknkYrPDNmzAg27eoTcscddzB58mQAtt9+ewB69uwJwNNPP13S+Tl1IVsOzy677FKr+pEEpELJyGvRokXhe5Ws6ytEuRPxcb399ttAxjASCKrKzJkzw0pTCl8h6dq1K//4xz+AqDPxuHHjQr5Atv5vMoKrreu0lMvzzz8/qCVJRUnp6n4OUZ+spJoL1sZGG20EZNopqLdX69atq30u23VXVYGVslxMhVn5cR07dqxkPlcV9RcqJZWjLsiINlsZvZSOiy66qKD7lAuU2zd27FiOPPJIIGM0WBXlz6lNyMSJE0siMlBXttxyy2q9swo9F8hb0rKy6tWjRr4sckRNMzVVaSXdaVnoIhswYEB42Et6jScQaoy6AQ8ZMiSxYzzyyCODB4To06dPqAxQUn28F5Emb/KEgqhRqmRqhfoKKcs2FFU4KaT39ddfJ/Z41Yd33nknVCyde+65QDThXnbZZcOiIz7xUQNiefSoJ1pNfiiFZPHixaGI4/eEkquzNT7V8SllHn744TDhEfKLuuyyy4I3VCGqlYrBCiusUG3xWGjvJ4e0jDHGGJN6ymoLs5SVldUrBhMPaWmlpQTVYlBRUbFUa+P6jjFpeIx1G1+vXr1Cd22pMUcddVRI1JaKE19dKmSQb/WmUMdQXcaldPXq1atgJaHFOk979+7NGmusUe31e++9F8ht/yVfiw0f33bbbRe6Y8dVZMgUN5xwwglA/hOwfQwz5GOMN9xwQ/AfkrtythB0LqhpjFZ4jDHGGJN6cqrwJA3P1jOkfYxpHx94jKWAx9jw8S233HLBnV/Gj+Lnn38OSkBa1NZiUqwx3nzzzfTu3RvImH9ClLeVa6zwGGOMMeZ3ixUejzHxWOHxGEsBj7Fx41M7oieffBKA1VdfHchU+qpbuqqa8oWPYYZ8jHHWrFkss0ymMLxNmza53nwlahqjJzweY+LxhMdjLAU8xvSPDzzGUsAhLWOMMcb8bqlV4THGGGOMSQNWeIwxxhiTejzhMcYYY0zq8YTHGGOMManHEx5jjDHGpB5PeIwxxhiTejzhMcYYY0zq+X/UWDbIV3o8KwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 50 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize some examples from the dataset. Stolen from other notebooks\n",
    "# We show a few examples of training images from each class.\n",
    "X=train_dset.images\n",
    "y=train_dset.labels\n",
    "\n",
    "\n",
    "classes = list(range(10))\n",
    "num_classes = len(classes)\n",
    "samples_per_class = 5\n",
    "for y_hat, cls in enumerate(classes):\n",
    "    idxs = np.flatnonzero(train_dset.labels == y_hat)\n",
    "    idxs = np.random.choice(idxs, samples_per_class, replace=False)\n",
    "    for i, idx in enumerate(idxs):\n",
    "        plt_idx = i * num_classes + y_hat + 1\n",
    "        plt.subplot(samples_per_class, num_classes, plt_idx)\n",
    "        plt.imshow(X[idx])\n",
    "        plt.axis('off')\n",
    "        if i == 0:\n",
    "            plt.title(cls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a classifier based on a RNN where you sequentially feed the rows in the network and use the final hidden state for prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=https://cdn-images-1.medium.com/max/800/1*Cm_c-I02rBa1rtLZXBhNUw.png width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from exercise_code.rnn.rnn_nn import LSTM_Classifier, RNN_Classifier\n",
    "model_rnn = RNN_Classifier()\n",
    "model_lstm = LSTM_Classifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from exercise_code.rnn.solver import Solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START TRAIN.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/ipykernel_launcher.py:14: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Iteration 50/7500] TRAIN loss: 2.148\n",
      "[Iteration 100/7500] TRAIN loss: 1.513\n",
      "[Iteration 150/7500] TRAIN loss: 1.222\n",
      "[Iteration 200/7500] TRAIN loss: 1.072\n",
      "[Iteration 250/7500] TRAIN loss: 0.968\n",
      "[Iteration 300/7500] TRAIN loss: 0.879\n",
      "[Iteration 350/7500] TRAIN loss: 0.859\n",
      "[Iteration 400/7500] TRAIN loss: 0.824\n",
      "[Iteration 450/7500] TRAIN loss: 0.778\n",
      "[Iteration 500/7500] TRAIN loss: 0.771\n",
      "[Iteration 550/7500] TRAIN loss: 0.716\n",
      "[Iteration 600/7500] TRAIN loss: 0.675\n",
      "[Iteration 650/7500] TRAIN loss: 0.663\n",
      "[Iteration 700/7500] TRAIN loss: 0.673\n",
      "[Iteration 750/7500] TRAIN loss: 0.683\n",
      "[Iteration 800/7500] TRAIN loss: 0.601\n",
      "[Iteration 850/7500] TRAIN loss: 0.574\n",
      "[Iteration 900/7500] TRAIN loss: 0.590\n",
      "[Iteration 950/7500] TRAIN loss: 0.572\n",
      "[Iteration 1000/7500] TRAIN loss: 0.542\n",
      "[Iteration 1050/7500] TRAIN loss: 0.663\n",
      "[Iteration 1100/7500] TRAIN loss: 0.547\n",
      "[Iteration 1150/7500] TRAIN loss: 0.525\n",
      "[Iteration 1200/7500] TRAIN loss: 0.516\n",
      "[Iteration 1250/7500] TRAIN loss: 0.550\n",
      "[Iteration 1300/7500] TRAIN loss: 0.541\n",
      "[Iteration 1350/7500] TRAIN loss: 0.577\n",
      "[Iteration 1400/7500] TRAIN loss: 0.515\n",
      "[Iteration 1450/7500] TRAIN loss: 0.502\n",
      "[Iteration 1500/7500] TRAIN loss: 0.488\n",
      "[Epoch 1/5] TRAIN acc/loss: 0.844/0.488\n",
      "[Epoch 1/5] VAL   acc/loss: 0.868/0.443\n",
      "[Iteration 1550/7500] TRAIN loss: 0.473\n",
      "[Iteration 1600/7500] TRAIN loss: 0.484\n",
      "[Iteration 1650/7500] TRAIN loss: 0.472\n",
      "[Iteration 1700/7500] TRAIN loss: 0.423\n",
      "[Iteration 1750/7500] TRAIN loss: 0.457\n",
      "[Iteration 1800/7500] TRAIN loss: 0.432\n",
      "[Iteration 1850/7500] TRAIN loss: 0.410\n",
      "[Iteration 1900/7500] TRAIN loss: 0.464\n",
      "[Iteration 1950/7500] TRAIN loss: 0.471\n",
      "[Iteration 2000/7500] TRAIN loss: 0.383\n",
      "[Iteration 2050/7500] TRAIN loss: 0.395\n",
      "[Iteration 2100/7500] TRAIN loss: 0.387\n",
      "[Iteration 2150/7500] TRAIN loss: 0.387\n",
      "[Iteration 2200/7500] TRAIN loss: 0.350\n",
      "[Iteration 2250/7500] TRAIN loss: 0.355\n",
      "[Iteration 2300/7500] TRAIN loss: 0.355\n",
      "[Iteration 2350/7500] TRAIN loss: 0.304\n",
      "[Iteration 2400/7500] TRAIN loss: 0.396\n",
      "[Iteration 2450/7500] TRAIN loss: 0.401\n",
      "[Iteration 2500/7500] TRAIN loss: 0.336\n",
      "[Iteration 2550/7500] TRAIN loss: 0.373\n",
      "[Iteration 2600/7500] TRAIN loss: 0.346\n",
      "[Iteration 2650/7500] TRAIN loss: 0.364\n",
      "[Iteration 2700/7500] TRAIN loss: 0.369\n",
      "[Iteration 2750/7500] TRAIN loss: 0.343\n",
      "[Iteration 2800/7500] TRAIN loss: 0.312\n",
      "[Iteration 2850/7500] TRAIN loss: 0.290\n",
      "[Iteration 2900/7500] TRAIN loss: 0.355\n",
      "[Iteration 2950/7500] TRAIN loss: 0.326\n",
      "[Iteration 3000/7500] TRAIN loss: 0.307\n",
      "[Epoch 2/5] TRAIN acc/loss: 0.875/0.307\n",
      "[Epoch 2/5] VAL   acc/loss: 0.902/0.348\n",
      "[Iteration 3050/7500] TRAIN loss: 0.319\n",
      "[Iteration 3100/7500] TRAIN loss: 0.253\n",
      "[Iteration 3150/7500] TRAIN loss: 0.342\n",
      "[Iteration 3200/7500] TRAIN loss: 0.378\n",
      "[Iteration 3250/7500] TRAIN loss: 0.297\n",
      "[Iteration 3300/7500] TRAIN loss: 0.314\n",
      "[Iteration 3350/7500] TRAIN loss: 0.291\n",
      "[Iteration 3400/7500] TRAIN loss: 0.262\n",
      "[Iteration 3450/7500] TRAIN loss: 0.363\n",
      "[Iteration 3500/7500] TRAIN loss: 0.252\n",
      "[Iteration 3550/7500] TRAIN loss: 0.290\n",
      "[Iteration 3600/7500] TRAIN loss: 0.269\n",
      "[Iteration 3650/7500] TRAIN loss: 0.277\n",
      "[Iteration 3700/7500] TRAIN loss: 0.249\n",
      "[Iteration 3750/7500] TRAIN loss: 0.270\n",
      "[Iteration 3800/7500] TRAIN loss: 0.319\n",
      "[Iteration 3850/7500] TRAIN loss: 0.284\n",
      "[Iteration 3900/7500] TRAIN loss: 0.252\n",
      "[Iteration 3950/7500] TRAIN loss: 0.245\n",
      "[Iteration 4000/7500] TRAIN loss: 0.250\n",
      "[Iteration 4050/7500] TRAIN loss: 0.344\n",
      "[Iteration 4100/7500] TRAIN loss: 0.298\n",
      "[Iteration 4150/7500] TRAIN loss: 0.313\n",
      "[Iteration 4200/7500] TRAIN loss: 0.284\n",
      "[Iteration 4250/7500] TRAIN loss: 0.326\n",
      "[Iteration 4300/7500] TRAIN loss: 0.265\n",
      "[Iteration 4350/7500] TRAIN loss: 0.289\n",
      "[Iteration 4400/7500] TRAIN loss: 0.270\n",
      "[Iteration 4450/7500] TRAIN loss: 0.383\n",
      "[Iteration 4500/7500] TRAIN loss: 0.259\n",
      "[Epoch 3/5] TRAIN acc/loss: 0.906/0.259\n",
      "[Epoch 3/5] VAL   acc/loss: 0.925/0.270\n",
      "[Iteration 4550/7500] TRAIN loss: 0.229\n",
      "[Iteration 4600/7500] TRAIN loss: 0.273\n",
      "[Iteration 4650/7500] TRAIN loss: 0.256\n",
      "[Iteration 4700/7500] TRAIN loss: 0.278\n",
      "[Iteration 4750/7500] TRAIN loss: 0.300\n",
      "[Iteration 4800/7500] TRAIN loss: 0.240\n",
      "[Iteration 4850/7500] TRAIN loss: 0.253\n",
      "[Iteration 4900/7500] TRAIN loss: 0.241\n",
      "[Iteration 4950/7500] TRAIN loss: 0.285\n",
      "[Iteration 5000/7500] TRAIN loss: 0.194\n",
      "[Iteration 5050/7500] TRAIN loss: 0.292\n",
      "[Iteration 5100/7500] TRAIN loss: 0.295\n",
      "[Iteration 5150/7500] TRAIN loss: 0.266\n",
      "[Iteration 5200/7500] TRAIN loss: 0.235\n",
      "[Iteration 5250/7500] TRAIN loss: 0.274\n",
      "[Iteration 5300/7500] TRAIN loss: 0.292\n",
      "[Iteration 5350/7500] TRAIN loss: 0.228\n",
      "[Iteration 5400/7500] TRAIN loss: 0.229\n",
      "[Iteration 5450/7500] TRAIN loss: 0.266\n",
      "[Iteration 5500/7500] TRAIN loss: 0.234\n",
      "[Iteration 5550/7500] TRAIN loss: 0.251\n",
      "[Iteration 5600/7500] TRAIN loss: 0.231\n",
      "[Iteration 5650/7500] TRAIN loss: 0.202\n",
      "[Iteration 5700/7500] TRAIN loss: 0.250\n",
      "[Iteration 5750/7500] TRAIN loss: 0.280\n",
      "[Iteration 5800/7500] TRAIN loss: 0.220\n",
      "[Iteration 5850/7500] TRAIN loss: 0.216\n",
      "[Iteration 5900/7500] TRAIN loss: 0.278\n",
      "[Iteration 5950/7500] TRAIN loss: 0.464\n",
      "[Iteration 6000/7500] TRAIN loss: 0.353\n",
      "[Epoch 4/5] TRAIN acc/loss: 0.844/0.353\n",
      "[Epoch 4/5] VAL   acc/loss: 0.902/0.323\n",
      "[Iteration 6050/7500] TRAIN loss: 0.277\n",
      "[Iteration 6100/7500] TRAIN loss: 0.274\n",
      "[Iteration 6150/7500] TRAIN loss: 0.211\n",
      "[Iteration 6200/7500] TRAIN loss: 0.226\n",
      "[Iteration 6250/7500] TRAIN loss: 0.249\n",
      "[Iteration 6300/7500] TRAIN loss: 0.209\n",
      "[Iteration 6350/7500] TRAIN loss: 0.250\n",
      "[Iteration 6400/7500] TRAIN loss: 0.221\n",
      "[Iteration 6450/7500] TRAIN loss: 0.202\n",
      "[Iteration 6500/7500] TRAIN loss: 0.216\n",
      "[Iteration 6550/7500] TRAIN loss: 0.194\n",
      "[Iteration 6600/7500] TRAIN loss: 0.202\n",
      "[Iteration 6650/7500] TRAIN loss: 0.203\n",
      "[Iteration 6700/7500] TRAIN loss: 0.213\n",
      "[Iteration 6750/7500] TRAIN loss: 0.274\n",
      "[Iteration 6800/7500] TRAIN loss: 0.224\n",
      "[Iteration 6850/7500] TRAIN loss: 0.236\n",
      "[Iteration 6900/7500] TRAIN loss: 0.202\n",
      "[Iteration 6950/7500] TRAIN loss: 0.228\n",
      "[Iteration 7000/7500] TRAIN loss: 0.231\n",
      "[Iteration 7050/7500] TRAIN loss: 0.180\n",
      "[Iteration 7100/7500] TRAIN loss: 0.243\n",
      "[Iteration 7150/7500] TRAIN loss: 0.221\n",
      "[Iteration 7200/7500] TRAIN loss: 0.203\n",
      "[Iteration 7250/7500] TRAIN loss: 0.220\n",
      "[Iteration 7300/7500] TRAIN loss: 0.236\n",
      "[Iteration 7350/7500] TRAIN loss: 0.210\n",
      "[Iteration 7400/7500] TRAIN loss: 0.196\n",
      "[Iteration 7450/7500] TRAIN loss: 0.196\n",
      "[Iteration 7500/7500] TRAIN loss: 0.223\n",
      "[Epoch 5/5] TRAIN acc/loss: 0.969/0.223\n",
      "[Epoch 5/5] VAL   acc/loss: 0.940/0.221\n",
      "FINISH.\n",
      "START TRAIN.\n",
      "[Iteration 50/7500] TRAIN loss: 2.250\n",
      "[Iteration 100/7500] TRAIN loss: 1.568\n",
      "[Iteration 150/7500] TRAIN loss: 0.791\n",
      "[Iteration 200/7500] TRAIN loss: 0.574\n",
      "[Iteration 250/7500] TRAIN loss: 0.439\n",
      "[Iteration 300/7500] TRAIN loss: 0.410\n",
      "[Iteration 350/7500] TRAIN loss: 0.417\n",
      "[Iteration 400/7500] TRAIN loss: 0.321\n",
      "[Iteration 450/7500] TRAIN loss: 0.332\n",
      "[Iteration 500/7500] TRAIN loss: 0.289\n",
      "[Iteration 550/7500] TRAIN loss: 0.250\n",
      "[Iteration 600/7500] TRAIN loss: 0.239\n",
      "[Iteration 650/7500] TRAIN loss: 0.239\n",
      "[Iteration 700/7500] TRAIN loss: 0.223\n",
      "[Iteration 750/7500] TRAIN loss: 0.213\n",
      "[Iteration 800/7500] TRAIN loss: 0.213\n",
      "[Iteration 850/7500] TRAIN loss: 0.181\n",
      "[Iteration 900/7500] TRAIN loss: 0.162\n",
      "[Iteration 950/7500] TRAIN loss: 0.163\n",
      "[Iteration 1000/7500] TRAIN loss: 0.155\n",
      "[Iteration 1050/7500] TRAIN loss: 0.201\n",
      "[Iteration 1100/7500] TRAIN loss: 0.180\n",
      "[Iteration 1150/7500] TRAIN loss: 0.184\n",
      "[Iteration 1200/7500] TRAIN loss: 0.153\n",
      "[Iteration 1250/7500] TRAIN loss: 0.168\n",
      "[Iteration 1300/7500] TRAIN loss: 0.156\n",
      "[Iteration 1350/7500] TRAIN loss: 0.168\n",
      "[Iteration 1400/7500] TRAIN loss: 0.163\n",
      "[Iteration 1450/7500] TRAIN loss: 0.101\n",
      "[Iteration 1500/7500] TRAIN loss: 0.142\n",
      "[Epoch 1/5] TRAIN acc/loss: 1.000/0.142\n",
      "[Epoch 1/5] VAL   acc/loss: 0.956/0.146\n",
      "[Iteration 1550/7500] TRAIN loss: 0.118\n",
      "[Iteration 1600/7500] TRAIN loss: 0.110\n",
      "[Iteration 1650/7500] TRAIN loss: 0.122\n",
      "[Iteration 1700/7500] TRAIN loss: 0.107\n",
      "[Iteration 1750/7500] TRAIN loss: 0.146\n",
      "[Iteration 1800/7500] TRAIN loss: 0.147\n",
      "[Iteration 1850/7500] TRAIN loss: 0.098\n",
      "[Iteration 1900/7500] TRAIN loss: 0.106\n",
      "[Iteration 1950/7500] TRAIN loss: 0.122\n",
      "[Iteration 2000/7500] TRAIN loss: 0.096\n",
      "[Iteration 2050/7500] TRAIN loss: 0.122\n",
      "[Iteration 2100/7500] TRAIN loss: 0.098\n",
      "[Iteration 2150/7500] TRAIN loss: 0.108\n",
      "[Iteration 2200/7500] TRAIN loss: 0.116\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Iteration 2250/7500] TRAIN loss: 0.114\n",
      "[Iteration 2300/7500] TRAIN loss: 0.079\n",
      "[Iteration 2350/7500] TRAIN loss: 0.088\n",
      "[Iteration 2400/7500] TRAIN loss: 0.125\n",
      "[Iteration 2450/7500] TRAIN loss: 0.082\n",
      "[Iteration 2500/7500] TRAIN loss: 0.107\n",
      "[Iteration 2550/7500] TRAIN loss: 0.115\n",
      "[Iteration 2600/7500] TRAIN loss: 0.102\n",
      "[Iteration 2650/7500] TRAIN loss: 0.097\n",
      "[Iteration 2700/7500] TRAIN loss: 0.082\n",
      "[Iteration 2750/7500] TRAIN loss: 0.116\n",
      "[Iteration 2800/7500] TRAIN loss: 0.088\n",
      "[Iteration 2850/7500] TRAIN loss: 0.074\n",
      "[Iteration 2900/7500] TRAIN loss: 0.074\n",
      "[Iteration 2950/7500] TRAIN loss: 0.078\n",
      "[Iteration 3000/7500] TRAIN loss: 0.090\n",
      "[Epoch 2/5] TRAIN acc/loss: 1.000/0.090\n",
      "[Epoch 2/5] VAL   acc/loss: 0.975/0.082\n",
      "[Iteration 3050/7500] TRAIN loss: 0.067\n",
      "[Iteration 3100/7500] TRAIN loss: 0.066\n",
      "[Iteration 3150/7500] TRAIN loss: 0.058\n",
      "[Iteration 3200/7500] TRAIN loss: 0.052\n",
      "[Iteration 3250/7500] TRAIN loss: 0.076\n",
      "[Iteration 3300/7500] TRAIN loss: 0.082\n",
      "[Iteration 3350/7500] TRAIN loss: 0.110\n",
      "[Iteration 3400/7500] TRAIN loss: 0.094\n",
      "[Iteration 3450/7500] TRAIN loss: 0.084\n",
      "[Iteration 3500/7500] TRAIN loss: 0.082\n",
      "[Iteration 3550/7500] TRAIN loss: 0.075\n",
      "[Iteration 3600/7500] TRAIN loss: 0.075\n",
      "[Iteration 3650/7500] TRAIN loss: 0.102\n",
      "[Iteration 3700/7500] TRAIN loss: 0.078\n",
      "[Iteration 3750/7500] TRAIN loss: 0.073\n",
      "[Iteration 3800/7500] TRAIN loss: 0.080\n",
      "[Iteration 3850/7500] TRAIN loss: 0.063\n",
      "[Iteration 3900/7500] TRAIN loss: 0.070\n",
      "[Iteration 3950/7500] TRAIN loss: 0.064\n",
      "[Iteration 4000/7500] TRAIN loss: 0.072\n",
      "[Iteration 4050/7500] TRAIN loss: 0.084\n",
      "[Iteration 4100/7500] TRAIN loss: 0.066\n",
      "[Iteration 4150/7500] TRAIN loss: 0.072\n",
      "[Iteration 4200/7500] TRAIN loss: 0.064\n",
      "[Iteration 4250/7500] TRAIN loss: 0.072\n",
      "[Iteration 4300/7500] TRAIN loss: 0.054\n",
      "[Iteration 4350/7500] TRAIN loss: 0.061\n",
      "[Iteration 4400/7500] TRAIN loss: 0.058\n",
      "[Iteration 4450/7500] TRAIN loss: 0.092\n",
      "[Iteration 4500/7500] TRAIN loss: 0.056\n",
      "[Epoch 3/5] TRAIN acc/loss: 0.969/0.056\n",
      "[Epoch 3/5] VAL   acc/loss: 0.978/0.074\n",
      "[Iteration 4550/7500] TRAIN loss: 0.059\n",
      "[Iteration 4600/7500] TRAIN loss: 0.050\n",
      "[Iteration 4650/7500] TRAIN loss: 0.051\n",
      "[Iteration 4700/7500] TRAIN loss: 0.071\n",
      "[Iteration 4750/7500] TRAIN loss: 0.070\n",
      "[Iteration 4800/7500] TRAIN loss: 0.058\n",
      "[Iteration 4850/7500] TRAIN loss: 0.055\n",
      "[Iteration 4900/7500] TRAIN loss: 0.077\n",
      "[Iteration 4950/7500] TRAIN loss: 0.047\n",
      "[Iteration 5000/7500] TRAIN loss: 0.048\n",
      "[Iteration 5050/7500] TRAIN loss: 0.062\n",
      "[Iteration 5100/7500] TRAIN loss: 0.048\n",
      "[Iteration 5150/7500] TRAIN loss: 0.059\n",
      "[Iteration 5200/7500] TRAIN loss: 0.053\n",
      "[Iteration 5250/7500] TRAIN loss: 0.055\n",
      "[Iteration 5300/7500] TRAIN loss: 0.049\n",
      "[Iteration 5350/7500] TRAIN loss: 0.059\n",
      "[Iteration 5400/7500] TRAIN loss: 0.054\n",
      "[Iteration 5450/7500] TRAIN loss: 0.051\n",
      "[Iteration 5500/7500] TRAIN loss: 0.062\n",
      "[Iteration 5550/7500] TRAIN loss: 0.060\n",
      "[Iteration 5600/7500] TRAIN loss: 0.058\n",
      "[Iteration 5650/7500] TRAIN loss: 0.081\n",
      "[Iteration 5700/7500] TRAIN loss: 0.056\n",
      "[Iteration 5750/7500] TRAIN loss: 0.053\n",
      "[Iteration 5800/7500] TRAIN loss: 0.045\n",
      "[Iteration 5850/7500] TRAIN loss: 0.052\n",
      "[Iteration 5900/7500] TRAIN loss: 0.068\n",
      "[Iteration 5950/7500] TRAIN loss: 0.046\n",
      "[Iteration 6000/7500] TRAIN loss: 0.066\n",
      "[Epoch 4/5] TRAIN acc/loss: 0.969/0.066\n",
      "[Epoch 4/5] VAL   acc/loss: 0.978/0.075\n",
      "[Iteration 6050/7500] TRAIN loss: 0.062\n",
      "[Iteration 6100/7500] TRAIN loss: 0.046\n",
      "[Iteration 6150/7500] TRAIN loss: 0.043\n",
      "[Iteration 6200/7500] TRAIN loss: 0.049\n",
      "[Iteration 6250/7500] TRAIN loss: 0.047\n",
      "[Iteration 6300/7500] TRAIN loss: 0.036\n",
      "[Iteration 6350/7500] TRAIN loss: 0.034\n",
      "[Iteration 6400/7500] TRAIN loss: 0.039\n",
      "[Iteration 6450/7500] TRAIN loss: 0.033\n",
      "[Iteration 6500/7500] TRAIN loss: 0.041\n",
      "[Iteration 6550/7500] TRAIN loss: 0.030\n",
      "[Iteration 6600/7500] TRAIN loss: 0.042\n",
      "[Iteration 6650/7500] TRAIN loss: 0.069\n",
      "[Iteration 6700/7500] TRAIN loss: 0.050\n",
      "[Iteration 6750/7500] TRAIN loss: 0.048\n",
      "[Iteration 6800/7500] TRAIN loss: 0.050\n",
      "[Iteration 6850/7500] TRAIN loss: 0.052\n",
      "[Iteration 6900/7500] TRAIN loss: 0.055\n",
      "[Iteration 6950/7500] TRAIN loss: 0.072\n",
      "[Iteration 7000/7500] TRAIN loss: 0.052\n",
      "[Iteration 7050/7500] TRAIN loss: 0.045\n",
      "[Iteration 7100/7500] TRAIN loss: 0.042\n",
      "[Iteration 7150/7500] TRAIN loss: 0.054\n",
      "[Iteration 7200/7500] TRAIN loss: 0.045\n",
      "[Iteration 7250/7500] TRAIN loss: 0.034\n",
      "[Iteration 7300/7500] TRAIN loss: 0.072\n",
      "[Iteration 7350/7500] TRAIN loss: 0.041\n",
      "[Iteration 7400/7500] TRAIN loss: 0.052\n",
      "[Iteration 7450/7500] TRAIN loss: 0.034\n",
      "[Iteration 7500/7500] TRAIN loss: 0.049\n",
      "[Epoch 5/5] TRAIN acc/loss: 1.000/0.049\n",
      "[Epoch 5/5] VAL   acc/loss: 0.969/0.110\n",
      "FINISH.\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "                 dataset=train_dset,\n",
    "                 batch_size=batch_size,\n",
    "                 shuffle=True)\n",
    "val_loader = torch.utils.data.DataLoader(\n",
    "                dataset=val_dset,\n",
    "                batch_size=batch_size,\n",
    "                shuffle=False)\n",
    "\n",
    "solver = Solver(optim_args={\"lr\": 1e-3})\n",
    "\n",
    "# train rnn model\n",
    "solver.train(model_rnn, train_loader, val_loader, log_nth=50, num_epochs=5)\n",
    "\n",
    "# train lstm model\n",
    "solver.train(model_lstm, train_loader, val_loader, log_nth=50, num_epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train your RNN classifier and try to tune the hyperparameters. With your simple RNN classifier you should exceed an accuracy higher than __90%__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to improve your model by using a LSTM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from exercise_code.rnn.rnn_nn import LSTM_Classifier\n",
    "#model= LSTM_Classifier()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train your LSTM model again and see wether it improves performance on the validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test your Model\n",
    "When you are satisfied with your training, you can save the model. In order to be eligible for the bonus points you have to achieve a score higher than __97__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Model\n",
    "\n",
    "When you are satisfied with your training, you can save the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model... models/rnn_mnist_nn.model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LSTM_Classifier. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LSTM. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Linear. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Sigmoid. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
      "/home/cengerkin/anaconda3/envs/i2dl/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Tanh. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    }
   ],
   "source": [
    "os.makedirs('models', exist_ok=True)\n",
    "model_lstm.save(\"models/rnn_mnist_nn.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}