Part 6 - Saving and Loading Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and Loading Models\n",
    "\n",
    "In this notebook, I'll show you how to save and load models with PyTorch. This is important because you'll often want to load previously trained models to use in making predictions or to continue training on new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "import helper\n",
    "import fc_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a transform to normalize the data\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "# Download and load the training data\n",
    "trainset = datasets.FashionMNIST('F_MNIST_data/', download=True, train=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)\n",
    "\n",
    "# Download and load the test data\n",
    "testset = datasets.FashionMNIST('F_MNIST_data/', download=True, train=False, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see one of the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAHTCAYAAAB8/vKtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAEWhJREFUeJzt3UuP5PV1x+FfVXVdu3t6+jIziAFscCQWxrIjBUFkVl7HeysvLMl7ibFiybtYKGwCYxib6wCGufRluqrrmkX8Cs6XUGn18+zPnJrq6v7Uf3U6m82mAQB13W2/AAC47sQUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAENpJ/4FfvfNzB1FvkJOTk2j+/osvlmfvnNyJdn/w4Qfl2UdffRXtTu4G93q9aPfBwUE0/9abb5ZnHz95Eu1++PDh1nZzs/zuD+93knlPpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJAKL5nep11OvXzdcl9ytTP3ngjmn/5pZfKs71e9pGZzqbl2fFoHO3+59/8pjz7/PIy2r07mZRnLy6eR7v39naj+S8fPSrPPjs9jXa/88tflmens1m0+7PPPivPfvjgQbQ7kfxda227f9uuM0+mABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJATbEXpmaK//8UvyrMv3LsX7X7+vH7Sq9/vR7t3duofuc8+r5/Eaq210WhYnr175260+/TsrDzb6/ai3d/89a/R/H9/8EF59uDWQbT79LT+vnW72Smy119/vTy7v78f7f7PP/4xmueH58kUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAjd6Hum6/V6a7tPjk/Ks6tV9rqHw1F5NrlH2lpri8WiPLu3txft/ujjj8uzR4dH0e5Ep5t95/3000+j+dGo/nlp2UnRNtmdlGdXq1W0++Liojz7o1deiXYn90zTW8vbvPN8nXkyBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAISu9Qm25FRQa9m5oNFwGO0eBvPdbvb/ns2uyrPJCbXWWpvP5+XZbif77jef11/7t999G+2+/+L98uzp2Wm0+/GTJ9H88fFxeXa1zM6gbdNkPC7PXl5Oo92Ht2+XZ58+exbtpsaTKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQut73TMP5+jXT1o6OjsLtdf1+P5pPbopOp/VbqK21tlwty7Pr9Travbs7Kc9+9vnn0e7XXn2tPPvxw4fR7l6vF82vVsFN0vCXdH5V/6wOBtnvSfJ71ulkd1zv3LlTnk3vmSZ3opMb0dedJ1MACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJA6FqfYFtv8dzPwcFBNH81r58yS06JtdZat1v/DpW87tayM2rpeaedne193D/59JPy7Gw6i3YPh8Nofrmsn81Lznm11tp6U/+8pLuT03Xz+SLafXJyUp7900cfRbvTU4c3lSdTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASB0re+ZbtP+/n40v1nXb3MOBoNodyK9KTqZ1G+xJnc1W8vuuA762Xv++PGT8mx6O3exzG5rjoJ7qOnN4eQmaXrPdJv3b4+Pjra2mxpPpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQE2xFe7u74b9QP02VnoZKrmL1er1o93g0Ks/OF9kpseS1r1fraHfyM0vP3g2DE2qttdbtJqfMsjNoi+Bnnr1rrXU69WeN9PzbZDKO5vnheTIFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAELumRalNyIXy2V5ttfNboomtxbX6+yu5yq4C9oJb2P2uvXvjt3wPuV6Xb+u2etl33n39rLbu9PZrDy7Cj7nrbW2s1P/rG/Cz2pySzW9Z5rc3h2Ff5tmV1fl2fT/nd7u3SZPpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQE2xFvV721i2Xq/JsJzgl1lp2Rq0b7k5OLCXnuFprbbmqv+eTySTafXF+Xp7d2RlFu9PP6qDfL89ehifYks/bJjh711p2Tiw9RTYYDMqzh4eH0e6vvv66PHudT6ilPJkCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKAKEbfc90Z6f+39/pZ2/d1fyqPhzeDNxs6vdMe93spmhy77DXy3YvFovy7KBfvy/ZWmvr4LZmp5vdxkz+3397BeXJYXCXs7Xs9u58nd1SjW5zZj+ydnFxUZ7d3d3NllPiyRQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQutEn2A4ODra2u9etf4+JTkO11pbL+mmq9BxYcv6t08m++3U69de+WmXnvPb398uzs6tZtHsyHkfz02l9f3oO7OzsrDwbXkFrrdV/z7rhZzUxmUy2tvsm82QKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAIRu9D3TW8GNyfSmaK/XK88mtzFba221Sm6K5lci67L3fDAYlGdns6to9+3b9du50+k02j0cDqP5i4vn0XyiG9zPXS6zz0vyO76zk/1pTe7+3t7ineabzJMpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBIHSjT7AlJ7lS3W79BNs2dbvZ969Opz6/XC6j3cPhqDx7Ob2Mdien63Z2ss/KarWK5vf2dsuzZ2dn0e7kd/RqPo92Jz+zfr8f7Z7P6yf/RqNxtJsaT6YAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQOhG3zM9OT4pzy4Wi2j3KLitmd6IvLqalWcnu/Xblq211u3Vv7+t1utod3QXdBOtbldX9fuUh7cPo93fPX4czf/dT35Snj07P492J79nvfD27s5O/c9jL/ictxbev+3f6D/rW+PJFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAI3ejDd/fvv1iePTvL7jTu7k7Ks188+jLavVytyrPJjcfWWmub+mHQTnifMrmHOh6Po91Pnz0rz77y8svR7uiOa8vuob76ox9Hu//yyV/Ks8vlMtrd69Xft2T2f9XvmfbT31FKPJkCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQjf6Vs/u7m559vw8O8G2Ds6BHR8dRbvPTk/Ls91O9v1r3er/726nfpaqtdbmV/Py7N5e/bPSWmuXl5fl2adPn0a77969F81/8cXn5dlbt/aj3T8OTrh98ODDaPfl5bQ8e3h4O9rdCT7rw+Ew2k2NJ1MACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBIHSt75kmN/9aa63f7yfbo92LxbI8e3g7vJXYrX+H2mzq90hba60X7O52e9Hu1ar+nq9W2f/74OCgPHt6+izaPRgMovnj4+Py7Jdffhntfun+S+XZu3fuRLuTvy+9XvpZXZVn1+v6bGutjUaj8uxsNot2X2eeTAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgCha32CLdXt1L9LDIbZWatb+/vl2bPzs2h38v/uBLN/+xfKk72d7KzVel0/o7Zc1s+3tdbacDgsz45G42h3ehbr6PCoPLtYLKLdX3/zdXk2/ZklZ9Tu3bsX7R4GZ/MWy+w9Hw7qn1Un2ACAMjEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhK71PdNb+7ei+SdPn5Rn33vvvWj3r//p1+XZ9J5pcou106nfI22ttc1mE80n+oN+efZqdhXtHo9H5dlheDs3vTGZfN6Ojuq3UFtr7ezsvDx7dZX9vy8vp+XZ1XIV7X5+eVmeXYb3TN988x/Ks799991o93XmyRQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQutYn2N5446fR/GQyKc9+/sUX0e5/+bd/Lc++/dZb0e4X7t0rz07Dc147vV55Nj1rNRoNy7OLXnbW6vnz5+XZ8bj+OW0t+5y3lr32R4++inaPRvXTdfv7+9Huh3/+c3n239/9bbQ78Y9vvx3Nv/bqa9/TK7lZPJkCQEhMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKAKFrfc+0F9zGbK218/OL7+mV/LAm43E03+l0yrODfj/avd5syrPJ626tteVyWZ4dDeu3UFtrbTqdlmfX6+yO685O9mt+cHBQnk3v387nV+XZ4XAQ7b5161Y0vy3JDdjWWptOL7+nV3KzeDIFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhK71CbZ+eFpqMKifE0vPv61W9bNai+CUWGutLRaL8myvl73nnc26PDufz7PdnfoZtdUqe88HwQm3WXjGrLXsdF1ygm0ynkS7Z5366bqrq/r5ttbys3vbkvxtaa21/f397+mV3CyeTAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWA0LW+ZzoYZPcGnz59Vp49OjyMdn/73Xfl2WV4z3Qy2S3PdrvZbcxut34Hdjar37ZsrbXNpj7b7WbfOzvB25be1Uw/LxfPL8qzg/4g2p3cDe4kb3pr7e7du9H8tqQ/7/Qe6k3lyRQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQutYn2Lq97LvAN998U57d29uLdicn2P7r/fej3cPgpNf+3n60OznvtFgsot39Qb88Oxxs75RYehJrOBxF84PNujx7fl4/39Zaa/1+/WeWfl4+fPBhNL8tDx78KZp/6f798uzx0VG0+/GTJ9H8NnkyBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWAkJgCQEhMASAkpgAQElMACIkpAITEFABC1/qe6Wa9ieaTe6aTySTanZhOp9H8f/z+99/TKwH+v3n85HE0n9zefeGFF6Ld7pkCwA0mpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKAKHOZpOdMfvVOz/P/oFrajweR/PpGTXg/1a30ynPrsO/q9s0GAzKs8vlMtq9Xq+j+cTv/vB+/QfePJkCQExMASAkpgAQElMACIkpAITEFABCYgoAITEFgJCYAkBITAEgJKYAEBJTAAiJKQCExBQAQmIKAKH4nikA3HSeTAEgJKYAEBJTAAiJKQCExBQAQmIKACExBYCQmAJASEwBICSmABASUwAIiSkAhMQUAEJiCgAhMQWA0P8ArHb5DfW3THoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1030f1518>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 233,
       "width": 233
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "image, label = next(iter(trainloader))\n",
    "helper.imshow(image[0,:]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a network\n",
    "\n",
    "To make things more concise here, I moved the model architecture and training code from the last part to a file called `fc_model`. Importing this, we can easily create a fully-connected network with `fc_model.Network`, and train the network using `fc_model.train`. I'll use this model (once it's trained) to demonstrate how we can save and load models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the network, define the criterion and optimizer\n",
    "model = fc_model.Network(784, 10, [512, 256, 128])\n",
    "criterion = nn.NLLLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc_model.train(model, trainloader, testloader, criterion, optimizer, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and loading networks\n",
    "\n",
    "As you can imagine, it's impractical to train a network every time you need to use it. Instead, we can save trained networks then load them later to train more or use them for predictions.\n",
    "\n",
    "The parameters for PyTorch networks are stored in a model's `state_dict`. We can see the state dict contains the weight and bias matrices for each of our layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Our model: \\n\\n\", model, '\\n')\n",
    "print(\"The state dict keys: \\n\\n\", model.state_dict().keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest thing to do is simply save the state dict with `torch.save`. For example, we can save it to a file `'checkpoint.pth'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), 'checkpoint.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can load the state dict with `torch.load`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "state_dict = torch.load('checkpoint.pth')\n",
    "print(state_dict.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to load the state dict in to the network, you do `model.load_state_dict(state_dict)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_state_dict(state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems pretty straightforward, but as usual it's a bit more complicated. Loading the state dict works only if the model architecture is exactly the same as the checkpoint architecture. If I create a model with a different architecture, this fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try this\n",
    "model = fc_model.Network(784, 10, [400, 200, 100])\n",
    "# This will throw an error because the tensor sizes are wrong!\n",
    "model.load_state_dict(state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means we need to rebuild the model exactly as it was when trained. Information about the model architecture needs to be saved in the checkpoint, along with the state dict. To do this, you build a dictionary with all the information you need to compeletely rebuild the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = {'input_size': 784,\n",
    "              'output_size': 10,\n",
    "              'hidden_layers': [each.out_features for each in model.hidden_layers],\n",
    "              'state_dict': model.state_dict()}\n",
    "\n",
    "torch.save(checkpoint, 'checkpoint.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the checkpoint has all the necessary information to rebuild the trained model. You can easily make that a function if you want. Similarly, we can write a function to load checkpoints. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_checkpoint(filepath):\n",
    "    checkpoint = torch.load(filepath)\n",
    "    model = fc_model.Network(checkpoint['input_size'],\n",
    "                             checkpoint['output_size'],\n",
    "                             checkpoint['hidden_layers'])\n",
    "    model.load_state_dict(checkpoint['state_dict'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = load_checkpoint('checkpoint.pth')\n",
    "print(model)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}