DLBook_ch06.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deep Feed Forward Networks aka FFNN, MLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depth = Number of Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Width = Number of Neurons in a Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "output layer = Final layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training Data specify directly what the output layer must do, but not what each hidden layers do - the learning algorithm must decide how to use these layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"feedforward networks as function approximation machines that are designed to\n",
    "achieve statistical generalization\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear models are appealing because they may be fit efficiently\n",
    "and reliably, either in closed form or with convex optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To extend linear models to represent nonlinear functions of x --> apply Kernel Trick"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to choose the mapping $\\phi$ that provides a new representation for x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Use a very generic $\\phi$ - like RBF Kernel, but generalizes poorly, and cannot solve advanced problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Manually handcraft $\\phi$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Use Deep Learning to learn $\\phi$ \n",
    "<br>\n",
    "$ y = f(x;\\theta,w) = \\phi(x;\\theta)w $\n",
    "<br>\n",
    "can have advantage of both 1 and 2\n",
    "<br>\n",
    "#1. generic - choose a very broad family $\\phi(x;\\theta)$\n",
    "<br>\n",
    "#2. hand-craft - find right general function family"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Example: Learning XOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear model cannot represent XOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple FFNN: $ h = f^{(1)}(x; W,c) $\n",
    "<br>\n",
    "$ y = f^({2)}(h;w,b) $\n",
    "<br>\n",
    "$ f(x;W,c,w,b) = f^{(2)}(f^{(1)}(x)) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "affine transformation from a vector $x$ to a vector $h$\n",
    "<br>\n",
    "activation function $g$ - element-wise activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "    f(x;W,c,w,b) = w^{T}max[0,W^{T}x+c] + b\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-linear activation function (ReLU) mapps both $ x = [1,0]^{T} $ and $x=[0,1]^{T} $ to a single point in feature space, $ h = [1,0]^{T} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Gradient-based Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-linearity of a NN causes most interesting loss functions to become non-convex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stochastic Gradient Descent applied to non-convex loss functions has no Convergence guarantee, and is sensitive to the values of the initial parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.1 Cost Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. $\\textbf{ Negative Log Likelihood aka Cross Entropy between Training Data and Model Distribution}$\n",
    "<br>\n",
    "$ J(\\theta) = -E_{x,y ~ p_{data}} logp_{model}(y|x) $\n",
    "<br>\n",
    "Advantage of this approach is specifying a model $p(y|x)$ automatically determines a cost function log $p(y|x)$\n",
    "<br>\n",
    "Also, the log undoes the exponent of some output units and prevent saturating functions.\n",
    "<br>\n",
    "1. $\\textbf{ Functional }$\n",
    "<br>\n",
    "Instead of learning a full probability distribution $p(y|x;\\theta)$ we often want to learn just one conditional statistic of $y$ given $x$\n",
    "<br>\n",
    "A Functional is a mapping from functions to real numbers.\n",
    "<br>\n",
    "We can think of learning as choosing a function rather than merely choosing a set of parameters\n",
    "<br>\n",
    "For example, we can design the cost functional to have its minimum lie on the function that maps $x$ to the expected value of $y$ given $x$.\n",
    "<br>\n",
    "Unfortunately, mean squared error and mean absolute error often lead to poor\n",
    "results when used with gradient-based optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "x = np.linspace(-100,100, 10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n",
    "\n",
    "for i in x:\n",
    "    softmax_i = softmax(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Softmax Function')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEICAYAAABBBrPDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X+cXXV95/HXm0kmFkIChCENSehEiT9CVcBpah/+2NaIBrYSXEGHdTV06aZ2zbZua7dhKTxYqltxVVwXqg8oqZFVEkq1Ttu0yK/+oFtDBoxAjJEhBDMkhoGEQAKZzNz57B/ne5OTy7137uTHnTMz7+fjcR8553u+53u+58zN/dzv93vu+SoiMDMzq+WE0a6AmZkVmwOFmZnV5UBhZmZ1OVCYmVldDhRmZlaXA4WZmdXlQGHjiqQPStomaa+k80a7PkUj6WuSrhntetjY4kBhhSPpnZL+n6Q9knZJ+hdJv9Tg7l8AlkfE1Ij4gaStkt57POvbqFSXV1IQK7/OPI7Hu0LSg/m0iPhERPzx8TqmjU+TRrsCZnmSpgF/A/w2cCfQCrwL6G+wiF8ANh6f2h0TH4iIe0e7EmYj4RaFFc3rASLijogoRcQrEfG9iHgUQNIJkv5I0tOSnpX0DUnTJU2RtBdoAX4o6UlJtwNnAX+dvr3/N0ntkkLSb6Quqt2SPiHplyQ9KukFSTeVKyPpdZLul/S8pOckfVPSKbltuySdn9bPTHl+dSQnLOlXJfVWpB1sCUm6TtKd6VxfkrRRUkcu71xJ35bUl+p5k6Q3AV8DfiWd+wsp79clfSa373+S1JPOoyvfwknX6ROSnkjX6WZJGsm52fjgQGFF8xOgJGmVpAslnVqx/Yr0+jXgtcBU4KaI6I+IqSnPWyPidRHxMeCnZN/ip0bE53Pl/DIwH/gI8GXgauC9wDnAhyX9m5RPwJ8AZwJvAuYC1wFExJPAHwLflHQi8OfA1yPiH47FhahwMbAaOAXoAm4CkNRC1gJ7GmgHZgOrI2IT8AngX9O5n1JZoKT3pHP7MDArlbG6ItuvA78EvDXle/+xPjErPgcKK5SIeBF4JxDArUBf+qY7M2X5KPCliNgSEXuBq4BOSSPtRv3jiNgfEd8D9gF3RMSzEfEM8M/Aeak+PRFxTwpEfcCXgHIQISJuBZ4A1pF92F49zHH/KrVaXpD0VyOo74MRsTYiSsDtZB/cAAvJgtgfRMS+dE4P1izlcB8FVkbEIxHRT3Ytf0VSey7P5yLihYj4KfAAcO4I6mzjhAOFFU5EbIqIKyJiDvCLZB+EX06bzyT75lv2NNlY20xGZmdu+ZUq61MBJJ0habWkZyS9CPxf4PSKsm5N9fw/6QO3nksi4pT0umQE9f1Zbvll4DUpOM4Fno6IwRGUVXbYtUyB93myVkmt407FJhwHCiu0iPgx8HWyD2KA7WQD1mVnAYMc/kF/WBFHWYU/SWW8JSKmAf+BrDsKAElTyYLYbcB1kk47gmPsA07MldkCtDW47zbgrBotquHO/bBrKekkYAbwTIPHtgnCgcIKRdIbJf2+pDlpfS5wOfD9lOUO4L9Kmpc+pP8nsKbON+qdZGMZR+pkYC/wgqTZwB9UbP/fwMMR8ZvA35INII/UT8haCP9W0mTgj4ApDe77ELAD+JykkyS9RtI70radwBxJrTX2/RbwG5LOlTSF7Fqui4itR3AONo45UFjRvEQ20LxO0j6yAPE48Ptp+0qyPvp/Ap4C9gP/pU55fwL8URoT+PQR1Od/AOcDe8gCwbfLGyQtARaTDRoD/B5wvqSPjuQAEbEH+M/An5F9m98H9Nbd6dC+JeADwNlkA/e9ZAP0APeT3Sr8M0nPVdn3PuAa4C/Jgs3rgM6R1N0mBnniIjMzq8ctCjMzq8uBwszM6nKgMDOzuhwozMysrnHxUMDTTz892tvbR7saZmZjysMPP/xcRAz7m51xESja29vp7u4e7WqYmY0pkp4ePpe7nszMbBgOFGZmVpcDhZmZ1dVQoJC0WNLmNMHJiirbp0hak7avKz+mWNIFkh6W9Fj69z25fd6W0nskfaU8IYqk0yTdkyZLuafKfARmZtZEwwaK9CTLm4ELgQXA5ZIWVGS7EtgdEWcDNwI3pPTnyCaNeTOwlOwZPWVfBZaRTR4zn+yZOQArgPsiYj5wX1o3M7NR0kiLYiHQkyaKOUA2A9aSijxLgFVp+S5gkSRFxA8iYntK30j2hMwpkmYB0yLiXyN72NQ3gEuqlLUql25mZqOgkUAxm+yZ92W9HD6xyWF50uOe95A91z7vQ8AP0sQuszn86Zj5MmdGxI5U1g7gjGqVkrRMUrek7r6+vgZOw8zMjkQjgaLaZOqVj5ytm0fSOWTdUb81gjLriohbIqIjIjra2hqd48XMbHzY1z/Il763mR9ue+G4H6uRQNFLNt1i2RyymbGq5kkzbU0HdqX1OcB3gI+nyejL+efUKHNn6poi/ftsoydjZjZR7Osf5Cv39/D49j3H/ViNBIr1wPw0o1gr2cQmXRV5usgGqwEuBe6PiJB0CtlkL1dFxL+UM6cupZckvT3d7fRx4LtVylqaSzczs2Qo9cGcoGodNMfWsIEijTksB+4GNgF3RsRGSddLujhluw2YIamHbJav8p1Ky8lm3rpG0ob0Ko85/DbZjF49wJPA36X0zwEXSHoCuCCtm5lZzlCadO6E4x8nGnvWU0SsBdZWpF2bW94PXFZlv88An6lRZjfwi1XSnwcWNVIvM7OJqpSaFCpCi8LMzIqnPIt1iwOFmZlVc7DrqQmf4g4UZmZj0KExCrcozMysinKg8BiFmZlVNeQxCjMzq6eZt8c6UJiZjUFDQ9m/7noyM7Oq3KIwM7O6fNeTmZnVdXAwuwlNCgcKM7Mx6NDtscf/WA4UZmZjULjryczM6imlu54cKMzMrCo/68nMzOryXU9mZlZXFGmGOwBJiyVtltQjaUWV7VMkrUnb10lqT+kzJD0gaa+km3L5T87NeLdB0nOSvpy2XSGpL7ftN4/NqZqZjR/liYsKMcOdpBbgZrJpSXuB9ZK6IuJHuWxXArsj4mxJncANwEeA/cA1ZDPZHZzNLiJeAs7NHeNh4Nu58tZExPIjPiszs3Hu0BhFMVoUC4GeiNgSEQeA1cCSijxLgFVp+S5gkSRFxL6IeJAsYFQlaT5wBvDPI669mdkEVbSup9nAttx6b0qrmiciBoE9wIwG63A5WQsicmkfkvSopLskza22k6Rlkroldff19TV4KDOz8aFoz3qqVo04gjy1dAJ35Nb/GmiPiLcA93KopXJ44RG3RERHRHS0tbU1eCgzs/Hh0BhFMVoUvUD+W/0cYHutPJImAdOBXcMVLOmtwKSIeLicFhHPR0R/Wr0VeFsDdTQzm1CGCtb1tB6YL2mepFayFkBXRZ4uYGlavhS4v6IrqZbLObw1gaRZudWLgU0NlGNmNqFEE39wN+xdTxExKGk5cDfQAqyMiI2Srge6I6ILuA24XVIPWUuis7y/pK3ANKBV0iXA+3J3TH0YuKjikL8j6WJgMJV1xVGcn5nZuNTMFsWwgQIgItYCayvSrs0t7wcuq7Fve51yX1sl7SrgqkbqZWY2UZUKNphtZmYF46fHmplZXX7Wk5mZ1TXkx4ybmVk9Jc9wZ2Zm9UTBnvVkZmYFU749tsVdT2ZmVk3RnvVkZmYFU25RyC0KMzOrZqiJExc5UJiZjUHlrqcWD2abmVk17noyM7O6woPZZmZWT9EmLjIzs4I5+DsKj1GYmVk1Q36Eh5mZ1VO4x4xLWixps6QeSSuqbJ8iaU3avk5Se0qfIekBSXsl3VSxzz+kMjek1xn1yjIzs0NKRXp6rKQW4GbgQmABcLmkBRXZrgR2R8TZwI3ADSl9P3AN8OkaxX80Is5Nr2eHKcvMzJKizXC3EOiJiC0RcQBYDSypyLMEWJWW7wIWSVJE7IuIB8kCRqOqljWC/c3Mxr2hoeAEFed3FLOBbbn13pRWNU9EDAJ7gBkNlP3nqdvpmlwwaKgsScskdUvq7uvra+BQZmbjRymiKXc8QWOBolpN4gjyVPpoRLwZeFd6fWwkZUXELRHREREdbW1twxzKzGx8yVoUxQkUvcDc3PocYHutPJImAdOBXfUKjYhn0r8vAd8i6+I6orLMzCaa0lCxWhTrgfmS5klqBTqBroo8XcDStHwpcH+U792qQtIkSaen5cnArwOPH0lZZmYTUSmiKZMWAUwaLkNEDEpaDtwNtAArI2KjpOuB7ojoAm4DbpfUQ/btv7O8v6StwDSgVdIlwPuAp4G7U5BoAe4Fbk271CzLzMwyQ0PRlGlQoYFAARARa4G1FWnX5pb3A5fV2Le9RrFvq5G/ZllmZpYp2mC2mZkVTGmoOT+2AwcKM7MxaWgoaGnSJ7gDhZnZGNTMwWwHCjOzMaiZg9kOFGZmY5AHs83MrK7Bgv3gzszMCmZoyGMUZmZWR9Ee4WFmZgUzFMV6KKCZmRWMWxRmZlZXKfDtsWZmVls2mN2cYzlQmJmNQe56MjOzukoezDYzs3qG3KIwM7N6CvcID0mLJW2W1CNpRZXtUyStSdvXSWpP6TMkPSBpr6SbcvlPlPS3kn4saaOkz+W2XSGpT9KG9PrNoz9NM7PxZWioQF1PklqAm4ELgQXA5ZIWVGS7EtgdEWcDNwI3pPT9wDXAp6sU/YWIeCNwHvAOSRfmtq2JiHPT689GdEZmZhNA0VoUC4GeiNgSEQeA1cCSijxLgFVp+S5gkSRFxL6IeJAsYBwUES9HxANp+QDwCDDnKM7DzGxCKdoMd7OBbbn13pRWNU9EDAJ7gBmNVEDSKcAHgPtyyR+S9KikuyTNrbHfMkndkrr7+voaOZSZ2bhRtBnuqoWsOII8ry5YmgTcAXwlIrak5L8G2iPiLcC9HGqpHF54xC0R0RERHW1tbcMdysxsXCla11MvkP9WPwfYXitP+vCfDuxqoOxbgCci4svlhIh4PiL60+qtwNsaKMfMbEIp1GA2sB6YL2mepFagE+iqyNMFLE3LlwL3R0TdFoWkz5AFlE9VpM/KrV4MbGqgjmZmE0ozWxSThssQEYOSlgN3Ay3AyojYKOl6oDsiuoDbgNsl9ZC1JDrL+0vaCkwDWiVdArwPeBG4Gvgx8IiyqHhTusPpdyRdDAymsq44RudqZjZulJo4cdGwgQIgItYCayvSrs0t7wcuq7Fve41iq55hRFwFXNVIvczMJqqhofDTY83MrLZSeCpUMzOrozTk+SjMzKyOoSjW7yjMzKxgmjmY7UBhZjYGeTDbzMzq8mC2mZnVNeiJi8zMrB7PcGdmZnUV7aGAZmZWIBFBRLHmozAzswIpDWXPXHWLwszMqiqFA4WZmdUxNJT9664nMzOrajBFikluUZiZWTXlMYpJLQ4UZmZWxUApBYoitSgkLZa0WVKPpBVVtk+RtCZtXyepPaXPkPSApL2SbqrY522SHkv7fEVpmjtJp0m6R9IT6d9Tj/40zczGj0N3PTXnu/6wR5HUAtwMXAgsAC6XtKAi25XA7og4G7gRuCGl7weuAT5dpeivAsuA+em1OKWvAO6LiPnAfWndzMySg2MUBep6Wgj0RMSWiDgArAaWVORZAqxKy3cBiyQpIvZFxINkAeMgSbOAaRHxrxERwDeAS6qUtSqXbmZmwGABu55mA9ty670prWqeiBgE9gAzhimzt0aZMyNiRyprB3BGtQIkLZPULam7r6+vgdMwMxsfBg8OZhek6wmoFrLiCPIcTf5XZ464JSI6IqKjra1tJLuamY1pB+96KlCLoheYm1ufA2yvlUfSJGA6sGuYMufUKHNn6poqd1E920AdzcwmjIFSNkZRpF9mrwfmS5onqRXoBLoq8nQBS9PypcD9aeyhqtSl9JKkt6e7nT4OfLdKWUtz6WZmxqEWxeQmDWZPGi5DRAxKWg7cDbQAKyNio6Trge6I6AJuA26X1EPWkugs7y9pKzANaJV0CfC+iPgR8NvA14GfA/4uvQA+B9wp6Urgp8Blx+JEzczGi/JdT826PXbYQAEQEWuBtRVp1+aW91PjAz0i2mukdwO/WCX9eWBRI/UyM5uIync9TS5Q15OZmRWIHzNuZmZ1DfhZT2ZmVk/p4NNji/M7CjMzK5DyQwHd9WRmZlX5MeNmZlbXwUd4uOvJzMyqGSx5hjszM6tj0F1PZmZWT8ldT2ZmVs9gAR8KaGZmBTLY5IcCOlCYmY0xg/4dhZmZ1XOoReExCjMzq6I05DEKMzOro/wID/+OwszMqioNBS0niGyC0OOvoUAhabGkzZJ6JK2osn2KpDVp+zpJ7bltV6X0zZLen9LeIGlD7vWipE+lbddJeia37aJjc6pmZuPDwNBQ07qdoIEZ7iS1ADcDFwC9wHpJXWk607Irgd0RcbakTuAG4COSFpBNi3oOcCZwr6TXR8Rm4Nxc+c8A38mVd2NEfOHoT8/MbPwplaJp3U7QWItiIdATEVsi4gCwGlhSkWcJsCot3wUsUtYmWgKsjoj+iHgK6Enl5S0CnoyIp4/0JMzMJpLBoeIFitnAttx6b0qrmiciBoE9wIwG9+0E7qhIWy7pUUkrJZ1arVKSlknqltTd19fXwGmYmY0Pg0NDTGrSrbHQWKCoFraiwTx195XUClwM/EVu+1eB15F1Te0AvlitUhFxS0R0RERHW1tb7dqbmY0zg6Vo2q+yobFA0QvMza3PAbbXyiNpEjAd2NXAvhcCj0TEznJCROyMiFJEDAG38uquKjOzCe3A4FDTfmwHjQWK9cB8SfNSC6AT6KrI0wUsTcuXAvdHRKT0znRX1DxgPvBQbr/Lqeh2kjQrt/pB4PFGT8bMbCI4UBqidVLzAsWwdz1FxKCk5cDdQAuwMiI2Sroe6I6ILuA24HZJPWQtic6070ZJdwI/AgaBT0ZECUDSiWR3Uv1WxSE/L+lcsi6qrVW2m5lNaAOlIVqb2KIYNlAARMRaYG1F2rW55f3AZTX2/Szw2SrpL5MNeFemf6yROpmZTVQDpShc15OZmRVINkZRrMFsMzMrkAOl4g1mm5lZgQw0eTDbgcLMbIxp9mC2A4WZ2RhTxN9RmJlZgQyUgsnuejIzs1oODLrryczM6sgGs317rJmZ1eDbY83MrK4BD2abmVk9A6Xw7yjMzKy6iHDXk5mZ1TZQyuZ+a/WznszMrJqB0hCAWxRmZlZdOVB4jMLMzKo6UNQWhaTFkjZL6pG0osr2KZLWpO3rJLXntl2V0jdLen8ufaukxyRtkNSdSz9N0j2Snkj/nnp0p2hmNn4cGEwtiiIFCkktwM3AhcAC4HJJCyqyXQnsjoizgRuBG9K+C8imRT0HWAz8aSqv7Nci4tyI6MilrQDui4j5wH1p3czMyAWKgnU9LQR6ImJLRBwAVgNLKvIsAVal5buARZKU0ldHRH9EPAX0pPLqyZe1CrikgTqamU0I+weyQPGaycUKFLOBbbn13pRWNU9EDAJ7yObDrrdvAN+T9LCkZbk8MyNiRyprB3BGtUpJWiapW1J3X19fA6dhZjb27R8sATBlcsswOY+dRgJFtZt1o8E89fZ9R0ScT9al9UlJ726gLocKibglIjoioqOtrW0ku5qZjVn7B7JA8ZpJxQoUvcDc3PocYHutPJImAdOBXfX2jYjyv88C3+FQl9ROSbNSWbOAZxs/HTOz8a2/oF1P64H5kuZJaiUbnO6qyNMFLE3LlwL3R0Sk9M50V9Q8YD7wkKSTJJ0MIOkk4H3A41XKWgp898hOzcxs/DnYomhi19Ok4TJExKCk5cDdQAuwMiI2Sroe6I6ILuA24HZJPWQtic6070ZJdwI/AgaBT0ZESdJM4DvZeDeTgG9FxN+nQ34OuFPSlcBPgcuO4fmamY1pB8comnjX07CBAiAi1gJrK9KuzS3vp8YHekR8FvhsRdoW4K018j8PLGqkXmZmE82hu56KNUZhZmYFMRpdTw4UZmZjSFF/R2FmZgVR1NtjzcysIPYPlmhtOYETTvB8FGZmVkX/wBBTmtjtBA4UZmZjyv6BUlMHssGBwsxsTMkChVsUZmZWw/6BoaYOZIMDhZnZmPLyQIkTpzT0W+ljxoHCzGwM2dc/yEmtblGYmVkN+/oHOcktCjMzq2Vv/yBTHSjMzKyWrEXhriczM6th34GSu57MzKy6gdIQBwaHmNrqQGFmZlXs6x8EKGaLQtJiSZsl9UhaUWX7FElr0vZ1ktpz265K6ZslvT+lzZX0gKRNkjZK+t1c/uskPSNpQ3pddPSnaWY29u1NgaLZg9nDHk1SC3AzcAHQC6yX1BURP8pluxLYHRFnS+oEbgA+ImkB2bSo5wBnAvdKej3ZtKi/HxGPpLmzH5Z0T67MGyPiC8fqJM3MxoN9/dkjxovYolgI9ETElog4AKwGllTkWQKsSst3AYuUTYi9BFgdEf0R8RTQAyyMiB0R8QhARLwEbAJmH/3pmJmNX3sPdj0V766n2cC23Hovr/5QP5gnIgaBPcCMRvZN3VTnAetyycslPSpppaRTq1VK0jJJ3ZK6+/r6GjgNM7Ox7cVXBgCY/nOTm3rcRgJFtdkxosE8dfeVNBX4S+BTEfFiSv4q8DrgXGAH8MVqlYqIWyKiIyI62tra6p+Bmdk48MIrBwA45cTWph63kUDRC8zNrc8BttfKI2kSMB3YVW9fSZPJgsQ3I+Lb5QwRsTMiShExBNxK1vVlZjbh7d6XtShOKWCLYj0wX9I8Sa1kg9NdFXm6gKVp+VLg/oiIlN6Z7oqaB8wHHkrjF7cBmyLiS/mCJM3KrX4QeHykJ2VmNh698MoAEkxrcqAYdug8IgYlLQfuBlqAlRGxUdL1QHdEdJF96N8uqYesJdGZ9t0o6U7gR2R3On0yIkqS3gl8DHhM0oZ0qP8eEWuBz0s6l6yLaivwW8fwfM3MxqwXXj7AtNdMpqWJ82VDA4ECIH2Ar61Iuza3vB+4rMa+nwU+W5H2INXHL4iIjzVSJzOzieaFlwc45cTmtibAv8w2MxszXnhloOkD2eBAYWY2Zuza18+pblGYmVktO1/sZ+bJr2n6cR0ozMzGgIHSEM/t7WfmtClNP7YDhZnZGPDc3n4iYOZ0tyjMzKyKn+3ZD+CuJzMzq27ni1mg+Hm3KMzMrJqtz78MwNzTTmz6sR0ozMzGgKf69nH61NamPzkWHCjMzMaEp57fx7zTTxqVYztQmJkVXETQ8+xeXnv61FE5vgOFmVnBbdv1Crv2HeAtc6ePyvEdKMzMCu4H23YD8NY5p4zK8R0ozMwK7h9/0sf0n5vMG3/+5FE5vgOFmVmB9Q+WeODHz/KeN57BpJbR+ch2oDAzK7DvbtjO7pcHuOS82aNWh4YChaTFkjZL6pG0osr2KZLWpO3rJLXntl2V0jdLev9wZaYpV9dJeiKV2fyHr5uZFcDP9uzn83+/mTfPns67558+avUYNlBIagFuBi4EFgCXS1pQke1KYHdEnA3cCNyQ9l1ANi3qOcBi4E8ltQxT5g3AjRExH9idyjYzmxBKQ8G2XS9zx0M/5YN/+i/sHyjx+UvfgtTc6U/zGpkKdSHQExFbACStBpaQzYNdtgS4Li3fBdyk7KyWAKsjoh94Ks2pvTDle1WZkjYB7wH+fcqzKpX71SM6u2HcuX4bt/zzlobyRkRj+UZSgQYzN1pmo3UcWZkNF0k0WOqIyhzRBW20zGP/t2y0no1eo5GV2bjGr+do13MU/0bH4f/RSCq6f7DEQCnbYcGsadz68Q7eNGta4wUcB40EitnAttx6L/DLtfJExKCkPcCMlP79in3LHW3VypwBvBARg1XyH0bSMmAZwFlnndXAabzaqSe18oaZI7iLoMGAPpK43+i3hEbLHMmXjsbLbLzQhnOOqJ4NXqPjcu4jKfM41PM4XNBGyxzZ+7jRMke7nsf+W/mxPvcpk0/grNNO5M2zp3POmdNGtSVR1kigqFbLyvhYK0+t9GpdXvXyvzox4hbgFoCOjo4j+t55wYKZXLBg5pHsamY2YTQymN0LzM2tzwG218ojaRIwHdhVZ99a6c8Bp6Qyah3LzMyaqJFAsR6Yn+5GaiUbnO6qyNMFLE3LlwL3R9bR1wV0prui5gHzgYdqlZn2eSCVQSrzu0d+emZmdrSG7XpKYw7LgbuBFmBlRGyUdD3QHRFdwG3A7WmwehfZBz8p351kA9+DwCcjogRQrcx0yD8EVkv6DPCDVLaZmY0SjWSEv6g6Ojqiu7t7tKthZjamSHo4IjqGy+dfZpuZWV0OFGZmVpcDhZmZ1eVAYWZmdY2LwWxJfcDTR7j76WS/3yga12tkXK+RK2rdXK+ROZp6/UJEtA2XaVwEiqMhqbuRUf9mc71GxvUauaLWzfUamWbUy11PZmZWlwOFmZnV5UCRHixYQK7XyLheI1fUurleI3Pc6zXhxyjMzKw+tyjMzKwuBwozM6trQgUKSZdJ2ihpSFJHxbarJPVI2izp/bn0xSmtR9KKJtRxjaQN6bVV0oaU3i7pldy2rx3vulTU6zpJz+SOf1FuW9Vr16R6/S9JP5b0qKTvSDolpY/q9Up1aOp7p0495kp6QNKm9P7/3ZRe82/axLptlfRYOn53SjtN0j2Snkj/ntrkOr0hd002SHpR0qdG43pJWinpWUmP59KqXh9lvpLeb49KOv+YVSQiJswLeBPwBuAfgI5c+gLgh8AUYB7wJNnjz1vS8muB1pRnQRPr+0Xg2rTcDjw+itfuOuDTVdKrXrsm1ut9wKS0fANwQ0Gu16i+dyrqMgs4Py2fDPwk/d2q/k2bXLetwOkVaZ8HVqTlFeW/6Sj+HX8G/MJoXC/g3cD5+fdyresDXAT8HdlMoW8H1h2rekyoFkVEbIqIzVU2LQFWR0R/RDwF9AAL06snIrZExAFgdcp73CmbKPfDwB3NON5RqHXtmiIivheH5lj/PtmsiEUwau+dShGxIyIeScsvAZuoMRd9QSwBVqXlVcAlo1iXRcCTEXGkT344KhHxT2Rz/OTVuj5LgG9E5vtks4XOOhb1mFCBoo7ZwLbcem9Kq5XeDO8CdkbEE7m0eZJ+IOkfJb2rSfXIW56atCtz3QGjeY0q/Ueyb1Rlo3m9inRdDpIEE0lEAAACqUlEQVTUDpwHrEtJ1f6mzRTA9yQ9LGlZSpsZETsgC3LAGaNQr7JODv+yNtrXC2pfn+P2nht3gULSvZIer/Kq921OVdKiTnoz6ng5h79BdwBnRcR5wO8B35I07WjrMoJ6fRV4HXBuqssXy7tVKeqY3nPdyPWSdDXZLIrfTEnH/XoNV+0qaaN6L7qkqcBfAp+KiBep/TdtpndExPnAhcAnJb17FOpQlbJpmi8G/iIlFeF61XPc3nPDToU61kTEe49gt15gbm59DrA9LddKP2LD1VHSJODfAW/L7dMP9KflhyU9CbweOGZT+zV67STdCvxNWq137ZpSL0lLgV8HFkXqrG3G9RrGcb8uIyFpMlmQ+GZEfBsgInbmtuf/pk0TEdvTv89K+g5Zl91OSbMiYkfqOnm22fVKLgQeKV+nIlyvpNb1OW7vuXHXojhCXUCnpCmS5gHzgYeA9cB8SfPSt4vOlPd4ey/w44joLSdIapPUkpZfm+q4pQl1KR8/39f5QaB8F0ata9esei0mm2f94oh4OZc+qteL0XvvvEoa77oN2BQRX8ql1/qbNqteJ0k6ubxMdmPC42TXaWnKthT4bjPrlXNYq360r1dOrevTBXw83f30dmBPuYvqqDVzBH+0X2R/3F6yb5o7gbtz264mu0tlM3BhLv0isrtEngSublI9vw58oiLtQ8BGsrtnHgE+0ORrdzvwGPBoekPOGu7aNalePWT9shvS62tFuF6j9d6pUY93knVBPJq7ThfV+5s2qV6vTX+fH6a/1dUpfQZwH/BE+ve0UbhmJwLPA9NzaU2/XmSBagcwkD67rqx1fci6nm5O77fHyN3ZebQvP8LDzMzqcteTmZnV5UBhZmZ1OVCYmVldDhRmZlaXA4WZmdXlQGFmZnU5UJiZWV3/H8zxPpM17qvjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, softmax_i)\n",
    "plt.title('Softmax Function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "      return 1 / (1 + np.exp(-x))\n",
    "for i in x:\n",
    "    sigmoid_i = sigmoid(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Sigmoid Function')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGrtJREFUeJzt3XuQXOV95vHvo9HM6H4xGjDogoQtCIoTGzwmbHxjF7CBSiTHsbFUcYwTJ9RuBWdT9m4WB4clpFwb2+Vky2tsrxJjbAcbMHE2Wlte2E0gjp1AEBhkhJAtxEUDAg1C1xlNz3TPb//oM+LQdM/0jPoy58zzqZpS9+m3+/zm9DuP3n7P6XMUEZiZWb7MancBZmbWeA53M7MccribmeWQw93MLIcc7mZmOeRwNzPLIYe7tYSk35B093Rbr6R7Jf1OK2uaDElvl7Sr3XVY9jjcrWEkvU3SP0s6LOklST+S9BaAiLg1It7V6ppOZr2SbpA0IulY6ucPG11jxTpD0uvH7kfEP0XEOc1cp+XT7HYXYPkgaRHwXeA/AHcAXcDbgUI762qA2yPig+0uwmyyPHK3RjkbICK+FRGliDgeEXdHxHYASR+W9MOxxpLeJWlXMsr/oqR/HJseSdr+SNJfSDokaY+kX06W75W0X9JVqddaLOnrkvolPS3pk5Jm1VjvpZIeT9b7BUBT+WUlPSXpktT9GyT9dXJ7dTICv0rSM5JelHRdqm2HpD+S9ISko5IelLRS0g+SJo8knxI+IOkiSX2p556bTCUdkrRD0vrUY7dIuknS95LXvV/S66by+1n2OdytUX4KlCR9TdLlkpbWaihpGXAn8AngFGAX8MsVzX4J2J48/k3gNuAtwOuBDwJfkLQgafs/gMXAWcA7gQ8Bv1VjvX8DfBJYBjwBvHUqv2yd3gacA1wMXC/p3GT5x4BNwBXAIuC3gcGIeEfy+BsjYkFE3F5Rfyfwv4G7gVOBjwK3SkpP22wC/gRYCuwGPtWMX8ymP4e7NUREHKEcZgH8JdAvaYuk06o0vwLYERHfiYgi8Hng+Yo2T0bEVyOiBNwOrARujIhCRNwNDAOvl9QBfAD4REQcjYingM8Bv1ljvY9FxJ0RMQL89yrrrXRlMkoe+zlj4q1xwp8kn2AeAR4B3pgs/x3gkxGxK8oeiYgDdbzehcAC4M8iYjgi/oHyVNimVJvvRMS/Jtv1VuBNk6jXcsThbg0TETsj4sMRsQJ4A3AG5QCtdAawN/W8APoq2ryQun08aVe5bAHlEXgX8HTqsaeB5XWud2+Vdml3RMSS1M9zE7RPS//HMZjUC+X/qJ6YxOuMOQPYGxGjqWWVv2utddoM43C3poiIx4FbKId8pX3AirE7kpS+P0kvAiPAmallq4Bna6x3ZcV6V1ZpV48BYF7q/msn8dy9wFTmwp8DVo7tT0jU+l1thnO4W0NI+jlJH5e0Irm/kvJ0wX1Vmn8P+AVJ75E0G/g9JheOJyTTNncAn5K0UNKZlOe0/7rGen9e0nuT9f7+VNcLPAxslNQpqRd43ySe+1fAn0paq7JflHRK8tgLlPcdVHM/5f9U/jBZ70XAr1LeH2H2Cg53a5SjlHeC3i9pgHKoPwp8vLJhRLwIvB/4DHAAWAdsY+qHTX6UcujtAX5IeQfszeOs98+S9a4FfjTFdf4x5dH3Qco7ML85ief+OeX/kO4GjgBfAeYmj90AfC2Z37+yov5hYD1wOeVPLF8EPpR8SjJ7BfliHdZuyTRDH/AbEXFPu+sxywOP3K0tJL1b0hJJ3cAfUT7evNoUjplNgcPd2uXfUD5i5EXK88bviYjj7S3JLD88LWNmlkMeuZuZ5VDbThy2bNmyWL16dbtWb2aWSQ8++OCLEdEzUbu2hfvq1avZtm1bu1ZvZpZJkp6euJWnZczMcsnhbmaWQw53M7MccribmeWQw93MLIcmDHdJNyeXNXu0xuOS9HlJuyVtl3R+48s0M7PJqGfkfgtw2TiPX0757HprgauBL518WWZmdjImPM49In4gafU4TTYAX0+uanNfcjKo0yNiX4NqNJt2hkZKPPrsYfoOHufw8REKxRLDxVGKoy+fziN9Zo9XneTDp/2Y0S4+9zTeuHJJU9fRiC8xLeeVlyrrS5a9KtwlXU15dM+qVasasGqz1joyNMKnv/84dz7YR6E4OvETxiE1qCjLnFMXzclEuFfrolWHJRGxGdgM0Nvb66GLZcqxQpErv/wv7N5/jPe9eQWXnHsaq5fNZ+m8Tro7O+jqmEVnxyv/HOQEtzZpRLj38crrUK6gfK1Hs1z51Pd28tMXjnLzh9/CReec2u5yzMbViEMhtwAfSo6auRA47Pl2y5u+g4N8e9tefvPCMx3slgkTjtwlfQu4CFgmqQ/4r0AnQER8GdgKXAHsBgaB32pWsWbt8p2HnqUUwdXvfF27SzGrSz1Hy2ya4PGgfPV6s9z6P48+z5tXLWX5krkTNzabBvwNVbMJvDQwzGP7jvDvzvV0jGWHw91sAg/vPQjAm1ctbXMlZvVzuJtN4OFnDjFL8Ibli9tdilndHO5mE3hs31Fe17OA+d1tu3CZ2aQ53M0m8NSBAdYsm9/uMswmxeFuNo7SaPDMgUHW9DjcLVsc7mbjeO7QcYZLo6w5xeFu2eJwNxvHUwcGAFjtaRnLGIe72Tj2HR4C8JeXLHMc7mbj2H+kHO49C7vbXInZ5Djczcax/2iBJfM6mdPZ0e5SzCbF4W42jheODHGqR+2WQQ53s3HsP1rgtEVz2l2G2aQ53M3Gsf9IwfPtlkkOd7MaIoL+ow53yyaHu1kNg8MlhkujvGZeV7tLMZs0h7tZDYePjwCwZF5nmysxmzyHu1kNhwbL4b54rsPdssfhblbDoePDACye62kZyx6Hu1kNhwc9LWPZ5XA3q+HQcU/LWHY53M1q8A5VyzKHu1kNhwZH6OqYxVyfV8YyyOFuVsPh48MsmtuJpHaXYjZpDnezGg4NjnhKxjLL4W5Ww9GhIgvnzG53GWZT4nA3q+FYociCboe7ZZPD3ayGweEi87sc7pZNDnezGgYKJeZ75G4Z5XA3q+FYocj8bh8GadnkcDerYXC46JG7ZVZd4S7pMkm7JO2WdG2Vx1dJukfSjyVtl3RF40s1a51CscRIKbxD1TJrwnCX1AHcBFwOrAM2SVpX0eyTwB0RcR6wEfhiows1a6XBQgmAeV2elrFsqmfkfgGwOyL2RMQwcBuwoaJNAIuS24uB5xpXolnrHSsUATwtY5lVT7gvB/am7vcly9JuAD4oqQ/YCny02gtJulrSNknb+vv7p1CuWWsMDJfD3dMyllX1hHu1E2tExf1NwC0RsQK4AviGpFe9dkRsjojeiOjt6emZfLVmLTLgaRnLuHrCvQ9Ymbq/gldPu3wEuAMgIv4FmAMsa0SBZu0wUPDI3bKtnnB/AFgraY2kLso7TLdUtHkGuBhA0rmUw93zLpZZg8Oec7dsmzDcI6IIXAPcBeykfFTMDkk3SlqfNPs48LuSHgG+BXw4Iiqnbswy41gyLePTD1hW1dVzI2Ir5R2l6WXXp24/Bry1saWZtc/AiaNlPOdu2eRvqJpVMeBpGcs4h7tZFceHS0jQPdt/IpZN7rlmVRSKo8yZ3eFL7FlmOdzNqhgaKTGn038ell3uvWZVFEZG6Z7tnamWXQ53syqGih65W7a595pVMTRS8sjdMs3hblZFoTjqkbtlmnuvWRVDIyW6Oz1yt+xyuJtVUSiO+hh3yzT3XrMqhkZGmeORu2WYw92sikKx5JG7ZZp7r1kVBY/cLeMc7mZVeORuWefea1aF59wt6xzuZlWUv8TkPw/LLvdeswrF0ijF0fDI3TLN4W5WoVAcBXwud8s2916zCmPh7pG7ZZnD3azC0Ej54tg+t4xlmXuvWYWXp2U8crfscribVfDI3fLAvdesgkfulgcOd7MKYyP3bo/cLcPce80qnAh3j9wtwxzuZhVePhTSfx6WXe69ZhU8crc8cLibVfDI3fLAvdesQsEjd8sBh7tZBY/cLQ/q6r2SLpO0S9JuSdfWaHOlpMck7ZD0zcaWadY6nnO3PJg9UQNJHcBNwKVAH/CApC0R8ViqzVrgE8BbI+KgpFObVbBZsxWKo8wSdHao3aWYTVk9I/cLgN0RsScihoHbgA0VbX4XuCkiDgJExP7GlmnWOkMjJeZ0diA53C276gn35cDe1P2+ZFna2cDZkn4k6T5Jl1V7IUlXS9omaVt/f//UKjZrsqGRUZ/L3TKvnh5cbfgSFfdnA2uBi4BNwF9JWvKqJ0VsjojeiOjt6emZbK1mLVEolnwud8u8esK9D1iZur8CeK5Km7+LiJGIeBLYRTnszTLHI3fLg3p68APAWklrJHUBG4EtFW3+F/BvASQtozxNs6eRhZq1ikfulgcThntEFIFrgLuAncAdEbFD0o2S1ifN7gIOSHoMuAf4zxFxoFlFmzWTR+6WBxMeCgkQEVuBrRXLrk/dDuBjyY9ZphWKJbo9creM8/DErIJH7pYH7sFmFQrFUc+5W+Y53M0qFEZKHrlb5rkHm1UY+4aqWZY53M0qlKdl/Kdh2eYebFZhaKTkM0Ja5jnczSp45G554B5sllIsjVIcDY/cLfMc7mYpvgqT5YV7sFmKr8JkeeFwN0sZ8sjdcsI92Cyl4JG75YTD3SxlaMQjd8sH92CzlELRI3fLB4e7WcrYyL3bI3fLOPdgs5SxkbvPLWNZ53A3SzkxcvdZIS3j3IPNUjxyt7xwuJulFDxyt5xwDzZLGfLI3XLC4W6W4pG75YV7sFnK2LllPHK3rHO4m6UUiqPMEsyepXaXYnZSHO5mKWPXT5Uc7pZtDnezlEJx1PPtlgvuxWYpYyN3s6xzuJuleORueeFebJbikbvlhcPdLGWoOEq3w91ywOFullIYKXlaxnLBvdgsZag46mkZy4W6wl3SZZJ2Sdot6dpx2r1PUkjqbVyJZq3jkbvlxYS9WFIHcBNwObAO2CRpXZV2C4HfB+5vdJFmrVLwyN1yop4hygXA7ojYExHDwG3Ahirt/hT4DDDUwPrMWmrII3fLiXp68XJgb+p+X7LsBEnnASsj4rvjvZCkqyVtk7Stv79/0sWaNVv5UEiHu2VfPb242kk24sSD0izgL4CPT/RCEbE5Inojorenp6f+Ks1apFAcZc5sT8tY9tUT7n3AytT9FcBzqfsLgTcA90p6CrgQ2OKdqpY1EeEvMVlu1BPuDwBrJa2R1AVsBLaMPRgRhyNiWUSsjojVwH3A+ojY1pSKzZpkpBSMBp6WsVyYsBdHRBG4BrgL2AncERE7JN0oaX2zCzRrFV9iz/Jkdj2NImIrsLVi2fU12l508mWZtd7YVZh8+gHLA3/+NEuMXT91jg+FtBxwLzZL+PqplicOd7PEUDJy95eYLA/ci80S3qFqeeJwN0t4WsbyxOFulhiblvFx7pYH7sVmiYKnZSxHHO5miRMjd59bxnLA4W6WeHnO3X8Wln3uxWYJf0PV8sThbpYoFL1D1fLDvdgsMTRSQoKuDv9ZWPa5F5slhkZKzJndgVTt+jRm2eJwN0sMjYzS7SkZywn3ZLPE2MjdLA8c7maJoeKod6ZabrgnmyV8/VTLE4e7WWJopORj3C03HO5miUJx1FdhstxwTzZLeFrG8sThbpYYHC4xv9vhbvngcDdLHB8uMbdzdrvLMGsIh7tZYnC4yLwuj9wtHxzuZomB4ZLD3XLD4W4GlEaD4eIo87o8LWP54HA3ozwlA3jkbrnhcDejfKQMwFyHu+WEw92Ml8PdI3fLC4e7GelpGc+5Wz443M0oH+MOHrlbftQV7pIuk7RL0m5J11Z5/GOSHpO0XdLfSzqz8aWaNc+Aw91yZsJwl9QB3ARcDqwDNklaV9Hsx0BvRPwicCfwmUYXatZMx5NpGe9QtbyoZ+R+AbA7IvZExDBwG7Ah3SAi7omIweTufcCKxpZp1lxjO1Tne87dcqKecF8O7E3d70uW1fIR4PvVHpB0taRtkrb19/fXX6VZk/loGcubesK92qXgo2pD6YNAL/DZao9HxOaI6I2I3p6envqrNGuyQU/LWM7U8xm0D1iZur8CeK6ykaRLgOuAd0ZEoTHlmbXGyyN3T8tYPtQzcn8AWCtpjaQuYCOwJd1A0nnA/wTWR8T+xpdp1lwDhSJzOmfRMavaB1Wz7Jkw3COiCFwD3AXsBO6IiB2SbpS0Pmn2WWAB8G1JD0vaUuPlzKalo0NFFs7pbHcZZg1T12fQiNgKbK1Ydn3q9iUNrsuspcrh7ikZyw9/Q9UMODI04pG75YrD3YzyyH2RR+6WIw53M+Do0IinZSxXHO5mJHPu3Z6WsfxwuJvhHaqWPw53m/FGSqMcHyl5h6rlisPdZrxjQ+VTD3jkbnnicLcZ76jD3XLI4W4z3pGhEQAWzfW0jOWHw91mvJcGhgF4zfyuNldi1jgOd5vxDg463C1/HO424x04loT7PIe75YfD3Wa8g4PDzBIs9py75YjD3Wa8AwPDLJ3XxSyfy91yxOFuM97BgWGWer7dcsbhbjPeSwPDnm+33HG424zXf7TAsoUOd8sXh7vNaBHBvsNDnL54brtLMWsoh7vNaEeOFzk+UuL0xXPaXYpZQzncbUbbd+Q4AK91uFvOONxtRtt3aAjA0zKWOw53m9GePVQeuXtaxvLG4W4z2pMvDjCncxavXeRwt3xxuNuMtqf/GGctW+Bvp1ruONxtRnuif4Czeua3uwyzhnO424x1rFBk78FB1p66sN2lmDWcw91mrO17DxEBb1q1pN2lmDWcw91mrIeeOQjAm1Y43C1/HO42Y927q591py9i8Tyfx93yx+FuM9Lzh4d48JmDvPvnX9vuUsyawuFuM9JX//lJBLznvDPaXYpZU9QV7pIuk7RL0m5J11Z5vFvS7cnj90ta3ehCzRple98hvvrDp1j/xjM48xQfBmn5NHuiBpI6gJuAS4E+4AFJWyLisVSzjwAHI+L1kjYCnwY+0IyCzaZiuDjKMy8NcteO5/nSvU/Qs7CbP/6Vde0uy6xpJgx34AJgd0TsAZB0G7ABSIf7BuCG5PadwBckKSKigbUCcMcDe9n8T3tesazaaqquuMrCykX1vla13yyqtKzars6tUllLw+uo8/WqtaxsV/21TmJbNvA9BRgcLjKaPPCOs3v4b+/9BU5Z0F3tFc1yoZ5wXw7sTd3vA36pVpuIKEo6DJwCvJhuJOlq4GqAVatWTangpfO7OOe0Kl86qfLt8WpfKJdevbRySZUmU36t2rVVeW4d663eps7XqrO4+n//iWs5mW1ZzVR///ldHaw6ZT5vWb3UUzE2I9QT7tX+6ioHR/W0ISI2A5sBent7pzSqv3TdaVy67rSpPNXMbMaoZ4dqH7AydX8F8FytNpJmA4uBlxpRoJmZTV494f4AsFbSGkldwEZgS0WbLcBVye33Af/QjPl2MzOrz4TTMskc+jXAXUAHcHNE7JB0I7AtIrYAXwG+IWk35RH7xmYWbWZm46tnzp2I2ApsrVh2fer2EPD+xpZmZmZT5W+ompnlkMPdzCyHHO5mZjnkcDczyyG164hFSf3A01N8+jIqvv06TbiuyXFdkzdda3Ndk3MydZ0ZET0TNWpbuJ8MSdsiorfddVRyXZPjuiZvutbmuianFXV5WsbMLIcc7mZmOZTVcN/c7gJqcF2T47omb7rW5romp+l1ZXLO3czMxpfVkbuZmY3D4W5mlkPTPtwlvV/SDkmjknorHvtEclHuXZLenVo+7gW9m1Dj7ZIeTn6ekvRwsny1pOOpx77c7Foq6rpB0rOp9V+ReqzqtmtRXZ+V9Lik7ZL+VtKSZHlbt1dSQ0v7zjh1rJR0j6SdSf//j8nymu9pC2t7StJPkvVvS5a9RtL/lfSz5N+lLa7pnNQ2eVjSEUl/0I7tJelmSfslPZpaVnX7qOzzSX/bLun8hhUSEdP6BzgXOAe4F+hNLV8HPAJ0A2uAJyifkrgjuX0W0JW0WdfCej8HXJ/cXg082sZtdwPwn6osr7rtWljXu4DZye1PA5+eJturrX2nopbTgfOT2wuBnybvW9X3tMW1PQUsq1j2GeDa5Pa1Y+9pG9/H54Ez27G9gHcA56f7cq3tA1wBfJ/y1ewuBO5vVB3TfuQeETsjYleVhzYAt0VEISKeBHZTvpj3iQt6R8QwMHZB76ZT+UKgVwLfasX6TkKtbdcSEXF3RBSTu/dRvrrXdNC2vlMpIvZFxEPJ7aPATsrXKp6uNgBfS25/DXhPG2u5GHgiIqb6DfiTEhE/4NVXoqu1fTYAX4+y+4Alkk5vRB3TPtzHUe3C3cvHWd4KbwdeiIifpZatkfRjSf8o6e0tqiPtmuTj3s2pj8rt3EaVfpvyyGVMO7fXdNouJ0haDZwH3J8sqvaetlIAd0t6UOWL3gOcFhH7oPwfE3BqG+oas5FXDrDavb2g9vZpWp+bFuEu6f9JerTKz3ijploX5a7rYt1NqnETr+xU+4BVEXEe8DHgm5IWnWwtk6jrS8DrgDcltXxu7GlVXqqhx8TWs70kXQcUgVuTRU3fXhOVXWVZW48VlrQA+BvgDyLiCLXf01Z6a0ScD1wO/J6kd7ShhqpUvhToeuDbyaLpsL3G07Q+V9eVmJotIi6ZwtPGu3D3RBf0nrSJalT5wuDvBd6cek4BKCS3H5T0BHA2sO1k66m3rlR9fwl8N7lbz0XPm1qXpKuAXwEujmTysRXbawJN3y6TIamTcrDfGhHfAYiIF1KPp9/TlomI55J/90v6W8rTWS9IOj0i9iXTCvtbXVficuChse00HbZXotb2aVqfmxYj9ynaAmyU1C1pDbAW+Ffqu6B3M1wCPB4RfWMLJPVI6khun5XUuKcFtYytPz1392vA2N77WtuuVXVdBvwXYH1EDKaWt3V70b6+8yrJ/puvADsj4s9Ty2u9p62qa76khWO3Ke8cf5TydroqaXYV8HetrCvlFZ+e2729Umptny3Ah5KjZi4EDo9N35y0Vu5FnuKe51+j/L9bAXgBuCv12HWUj27YBVyeWn4F5aMLngCua1GdtwD/vmLZrwM7KB918RDwqy3edt8AfgJsTzrR6RNtuxbVtZvyPOPDyc+Xp8P2alffqVHH2yh/PN+e2k5XjPeetqius5L355HkvbouWX4K8PfAz5J/X9OGbTYPOAAsTi1r+fai/J/LPmAkya6P1No+lKdlbkr6209IHRF4sj8+/YCZWQ5leVrGzMxqcLibmeWQw93MLIcc7mZmOeRwNzPLIYe7mVkOOdzNzHLo/wOWNp3aEUdcXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, sigmoid_i)\n",
    "plt.title('Sigmoid Function')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "    log\\,softmax(z)_i \\, = \\, z_i \\, - \\, log \\sum_{j}^{} exp(z_j)\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First term cannot saturate, and the learning can proceed even when the second term becomes very small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When maximizing Log-Likelihood, first term encourages $z_i$ to be pushed up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the second term encourages all of $z$ to be pushed down"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$log \\sum_{j}^{} exp(z_j)$ can be roughly approximated by $max_j\\,z_j$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " objective functions that do not use a log to\n",
    "undo the exp of the softmax fail to learn when the argument to the exp becomes\n",
    "very negative, causing the gradient to vanish. In particular, squared error is a\n",
    "poor loss function for softmax units, and can fail to train the model to change its\n",
    "output, even when the model makes highly confident incorrect predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Softmax: At the\n",
    "extreme (when the difference between the maximal a i and the others is large in\n",
    "magnitude) it becomes a form of winner-take-all (one of the outputs is nearly 1\n",
    "and the others are nearly 0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Name:  The softmax function thus provides a “softened” version of the\n",
    "argmax "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, if we define a conditional distribution $p(y\\,|\\,x;\\,\\theta)$, the principle of maximum likelihood suggests we use $-log\\,p(y\\,|\\,x;\\,\\theta)$ as our cost function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, we can think of the neural network as representing a function $f(x;\\theta)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$f(x;\\theta)\\,=\\,w$ provides the parameters for a distribution over $y$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our loss function can then be interpreted as $-log\\,p(y;w(x))$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.1 Universal Approximation Properties and Depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Universal Approximation Theorem stats that:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A FFNN with a Linear output layer and at least one Hidden Layer with any 'squashing' activation function can approximate any Borel measurable function from one finite-dimensional space to another with any desired non-zero amount of error, provided that the network is given enough hidden units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Universal approximation theorems have been proven for a wide class of activation functions, which includes ReLU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "UAT guarantees that a large MLP will be able to <em>represent</em> this function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we are not guaranteed that the training algorithm will be able to <em>learn</em> that function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning can fail for two different reasons\n",
    "1. optimization algorithm may not be able to find the values of parameters that corresponds to the desired function.\n",
    "1. training algorithm might choose the wrong function due to overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.5 Back-Propagation and Other Differentiation Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Backprop refers only to the Method of computing the gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actual learning is performed by SGD, which uses the gradients computed by backprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.1 Computational Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Node = Variable (scalar, vector, matrix, tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operation = Function of one or more Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.2 Chain Rule of Calculus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "backprop is just multiplying Jacobians by gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.5 Symbol-to-Symbol Derivatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong><em>Symbol-to-Number</em> differentiation: </strong> <br>\n",
    "Input: computational graph + set of numerical values<br>\n",
    "Output: set of numerical values describing the gradient at those input values <br>\n",
    "used by: Torch, Caffe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong> <em> Symbol-to-Symbol </em> differentiation: </strong><br>\n",
    "Input: computational graph + additional nodes to the graph <br>\n",
    "Output: Symbolic description of the desired derivatives <br>\n",
    "Used by: Theano, TensorFlow <br>\n",
    "Advantage: <br>\n",
    "- derivatives are described in the same language as the original expression<br>\n",
    "- Any subset of the graph may then be evaluated using specific numerical values at a later time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.6 General Back-Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing a gradient in a graph with n nodes will never execute more than $ O ( n^2 )$ operations or store the output of more than $O ( n^2 )$ operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The back-propagation algorithm adds one Jacobian-vector product, which should be expressed with $O (1)$ nodes, per edge in the original graph. Because the computational graph is a directed acyclic graph it has at most $O ( n^2 )$ edges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most neural network cost functions are roughly chain-structured, causing back-propagation to have $O ( n )$ cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<em><strong> dynamic programming </strong></em> <br>\n",
    "Each node in the graph has a corresponding slot in a table to store the gradient for that node. <br>\n",
    "By filling in these table entries in order, back-propagation avoids repeating many common subexpressions. This table-filling strategy is sometimes called <strong><em> dynamic programming </em></strong>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.8 Compications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Ouput may be more than a single Tensor\n",
    "1. Memory consumption for backprop - need to maintain a buffer\n",
    "1. various data types - float32, float64, int, etc.\n",
    "1. undefined gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.9 Differentiation outside the Deep Learning Community"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatic Differentiation - computing derivatives algorithmically <br>\n",
    "backprop - a special case of a broader class of techniques called <strong> reverse mode accumulation </strong> <br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When forward graph $G$ has a single output node <br>\n",
    "and each partial derivative $\\frac{\\partial u^{(i)}}{\\partial u^{(j)}}$ can be computed with a constant amount of computation, <br>\n",
    "backprop guarantees that # of computations for backprop is same as for forward prop <br>\n",
    "Overall computation is therefore $ O(# edges)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Theano and TensorFlow use heuristics based on matching known simplification patterns in order to simplify the graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the # of outputs of graph is LARGER than the number of inputs, it may be preferable to use <strong>forward mode accumulation</strong> <br>\n",
    "Relationship between forward mode and backward mode is analogous to the relationship between left-multiplying vs. right-multiplying a sequence of matrices such as <br>\n",
    "<strong><center>ABCD</center></strong>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.10 Higher-Order Derivatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we have a function $f : {\\mathbb{R}^{n}} \\rightarrow \\mathbb{R} $, then the Hessian matrix is of size $n$ x $n$ <br>\n",
    "In DL, $n$ will be # of parameters, which could easily number in the Billions. <br>\n",
    "The entire Hessian matrix is thus infeasible to even represent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of explicitly computing the Hessian, DL approach is to use <strong>Krylov methods</strong>. <br>\n",
    "It approximately inverts a matrix or approximates eigendecomposition without using any operation other than matrix-vector products <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.6 Historical Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>Chain Rule</strong> - 17c (Leibniz, L'Hopital)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>Gradient Descent</strong> - 19c (Cauchy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>Chain rule based on DP, ANNs, Connectionism</strong> - 20c (Hinton, Rumelhart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>DL Renaissance</strong> - 2006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Larger datasets\n",
    "1. Powerful computers, better S/W infrastructure\n",
    "1. MSE $\\rightarrow$ cross-entropy loss + sigmoid/softmax outputs\n",
    "1. Sigmoid $\\rightarrow$ ReLU hidden units\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>ReLU</strong> - motivated from biological considerations \n",
    "1. For some inputs, biological neurons are completely active\n",
    "1. For some inputs, a biological neuron's output is proportional to its input\n",
    "1. Most of the time, biological neurons operate in the regime where they are inactive (ie. they should have <strong>sparse activations</strong>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}