task-2-nn/IMT2020553.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "daP5P95xedI7"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fuKM2YIedI_"
      },
      "source": [
        "\n",
        "[Learn the Basics](intro.html) ||\n",
        "[Quickstart](quickstart_tutorial.html) ||\n",
        "[Tensors](tensorqs_tutorial.html) ||\n",
        "[Datasets & DataLoaders](data_tutorial.html) ||\n",
        "[Transforms](transforms_tutorial.html) ||\n",
        "**Build Model** ||\n",
        "[Autograd](autogradqs_tutorial.html) ||\n",
        "[Optimization](optimization_tutorial.html) ||\n",
        "[Save & Load Model](saveloadrun_tutorial.html)\n",
        "\n",
        "# Build the Neural Network\n",
        "\n",
        "Neural networks comprise of layers/modules that perform operations on data.\n",
        "The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace provides all the building blocks you need to\n",
        "build your own neural network. Every module in PyTorch subclasses the [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html).\n",
        "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
        "building and managing complex architectures easily.\n",
        "\n",
        "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "t5TuAX-NedJB"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision import datasets, transforms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H0Tday1bedJC"
      },
      "source": [
        "## Get Device for Training\n",
        "We want to be able to train our model on a hardware accelerator like the GPU,\n",
        "if it is available. Let's check to see if\n",
        "[torch.cuda](https://pytorch.org/docs/stable/notes/cuda.html) is available, else we\n",
        "continue to use the CPU.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "X5qEP-YredJD",
        "outputId": "a42deaf8-e1f7-470d-bd66-67671c0a0b44",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n"
          ]
        }
      ],
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(f\"Using {device} device\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fznp094pedJD"
      },
      "source": [
        "## Define the Class\n",
        "We define our neural network by subclassing ``nn.Module``, and\n",
        "initialize the neural network layers in ``__init__``. Every ``nn.Module`` subclass implements\n",
        "the operations on input data in the ``forward`` method.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "0Z7kMNBredJE"
      },
      "outputs": [],
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NeuralNetwork, self).__init__()\n",
        "        self.flatten = nn.Flatten()\n",
        "        self.linear_relu_stack = nn.Sequential(\n",
        "            nn.Linear(28*28, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 256),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(256, 128),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(128, 10),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.flatten(x)\n",
        "        logits = self.linear_relu_stack(x)\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MX417W1IedJF"
      },
      "source": [
        "We create an instance of ``NeuralNetwork``, and move it to the ``device``, and print\n",
        "its structure.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "EtilI6WWedJG",
        "outputId": "517110e8-e468-406a-bd7a-771a5e77abff",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=512, out_features=256, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=256, out_features=128, bias=True)\n",
            "    (7): ReLU()\n",
            "    (8): Linear(in_features=128, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "model = NeuralNetwork().to(device)\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SgWqORPMedJH"
      },
      "source": [
        "To use the model, we pass it the input data. This executes the model's ``forward``,\n",
        "along with some [background operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866).\n",
        "Do not call ``model.forward()`` directly!\n",
        "\n",
        "Calling the model on the input returns a 2-dimensional tensor with dim=0 corresponding to each output of 10 raw predicted values for each class, and dim=1 corresponding to the individual values of each output.  .\n",
        "We get the prediction probabilities by passing it through an instance of the ``nn.Softmax`` module.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "bL7Fd-jPedJH",
        "outputId": "3c7d26ec-1469-4a9f-a08c-24d8e0ffee96",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predicted class: tensor([2], device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "X = torch.rand(1, 28, 28, device=device)\n",
        "logits = model(X)\n",
        "pred_probab = nn.Softmax(dim=1)(logits)\n",
        "y_pred = pred_probab.argmax(1)\n",
        "print(f\"Predicted class: {y_pred}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OqfOibMwedJI"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m6xK6kOUedJI"
      },
      "source": [
        "## Model Layers\n",
        "\n",
        "Let's break down the layers in the FashionMNIST model. To illustrate it, we\n",
        "will take a sample minibatch of 3 images of size 28x28 and see what happens to it as\n",
        "we pass it through the network.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "Ll53Nx-FedJJ",
        "outputId": "088e1397-d5bc-42f6-ec21-fcccf186923c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "input_image = torch.rand(3,28,28)\n",
        "print(input_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHpcf0RyedJJ"
      },
      "source": [
        "### nn.Flatten\n",
        "We initialize the [nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html)\n",
        "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (\n",
        "the minibatch dimension (at dim=0) is maintained).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "K9YB00zTedJJ",
        "outputId": "c588d6e2-d54a-48cc-b4f4-bfd7ec136947",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 784])\n"
          ]
        }
      ],
      "source": [
        "flatten = nn.Flatten()\n",
        "flat_image = flatten(input_image)\n",
        "print(flat_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j_JI6cf3edJK"
      },
      "source": [
        "### nn.Linear\n",
        "The [linear layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
        "is a module that applies a linear transformation on the input using its stored weights and biases.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "NTjtAxGuedJK",
        "outputId": "0636e8a8-00b8-4461-ea5e-3d4a970de171",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 20])\n"
          ]
        }
      ],
      "source": [
        "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
        "hidden1 = layer1(flat_image)\n",
        "print(hidden1.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vP59RkcledJK"
      },
      "source": [
        "### nn.ReLU\n",
        "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
        "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
        "learn a wide variety of phenomena.\n",
        "\n",
        "In this model, we use [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) between our\n",
        "linear layers, but there's other activations to introduce non-linearity in your model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "57XADFT1edJL",
        "outputId": "811d775f-e846-4b79-d8ca-0f54e17e3033",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before ReLU: tensor([[-0.3498, -0.1254, -0.4681,  0.3887,  0.4149,  0.0081, -0.4632,  0.1911,\n",
            "         -0.0842,  0.0370,  0.4349, -0.1934,  0.5443, -0.1626,  0.1376,  0.4881,\n",
            "          0.2000,  0.0248, -0.0703, -0.4049],\n",
            "        [-0.6048,  0.1332, -0.5576,  0.4004,  0.1000,  0.0750, -0.0309, -0.2407,\n",
            "         -0.3944, -0.2268,  0.1850,  0.0445,  0.7433, -0.2390,  0.3930,  0.4572,\n",
            "          0.3350,  0.1396, -0.1456, -0.6955],\n",
            "        [-0.2510, -0.1905, -0.4218,  0.1347,  0.4308,  0.0014, -0.3952,  0.0819,\n",
            "         -0.5256, -0.1620,  0.3747, -0.1405,  0.7670, -0.3241,  0.2693,  0.3087,\n",
            "          0.0898, -0.2047,  0.3649, -0.5633]], grad_fn=<AddmmBackward0>)\n",
            "\n",
            "\n",
            "After ReLU: tensor([[0.0000, 0.0000, 0.0000, 0.3887, 0.4149, 0.0081, 0.0000, 0.1911, 0.0000,\n",
            "         0.0370, 0.4349, 0.0000, 0.5443, 0.0000, 0.1376, 0.4881, 0.2000, 0.0248,\n",
            "         0.0000, 0.0000],\n",
            "        [0.0000, 0.1332, 0.0000, 0.4004, 0.1000, 0.0750, 0.0000, 0.0000, 0.0000,\n",
            "         0.0000, 0.1850, 0.0445, 0.7433, 0.0000, 0.3930, 0.4572, 0.3350, 0.1396,\n",
            "         0.0000, 0.0000],\n",
            "        [0.0000, 0.0000, 0.0000, 0.1347, 0.4308, 0.0014, 0.0000, 0.0819, 0.0000,\n",
            "         0.0000, 0.3747, 0.0000, 0.7670, 0.0000, 0.2693, 0.3087, 0.0898, 0.0000,\n",
            "         0.3649, 0.0000]], grad_fn=<ReluBackward0>)\n"
          ]
        }
      ],
      "source": [
        "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
        "hidden1 = nn.ReLU()(hidden1)\n",
        "print(f\"After ReLU: {hidden1}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XyR5r7jqedJL"
      },
      "source": [
        "### nn.Sequential\n",
        "[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) is an ordered\n",
        "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
        "sequential containers to put together a quick network like ``seq_modules``.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "q1qPUD9tedJL"
      },
      "outputs": [],
      "source": [
        "seq_modules = nn.Sequential(\n",
        "    flatten,\n",
        "    layer1,\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(20, 10)\n",
        ")\n",
        "input_image = torch.rand(3,28,28)\n",
        "logits = seq_modules(input_image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6sAuEThSedJM"
      },
      "source": [
        "### nn.Softmax\n",
        "The last linear layer of the neural network returns `logits` - raw values in [-\\infty, \\infty] - which are passed to the\n",
        "[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) module. The logits are scaled to values\n",
        "[0, 1] representing the model's predicted probabilities for each class. ``dim`` parameter indicates the dimension along\n",
        "which the values must sum to 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "YGFwy0udedJM"
      },
      "outputs": [],
      "source": [
        "softmax = nn.Softmax(dim=1)\n",
        "pred_probab = softmax(logits)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyDhNYCmedJM"
      },
      "source": [
        "## Model Parameters\n",
        "Many layers inside a neural network are *parameterized*, i.e. have associated weights\n",
        "and biases that are optimized during training. Subclassing ``nn.Module`` automatically\n",
        "tracks all fields defined inside your model object, and makes all parameters\n",
        "accessible using your model's ``parameters()`` or ``named_parameters()`` methods.\n",
        "\n",
        "In this example, we iterate over each parameter, and print its size and a preview of its values.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "4xIX9LEOedJN",
        "outputId": "3f3c96c9-664d-4d04-b8ea-29b8816131fd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model structure: NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=512, out_features=256, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=256, out_features=128, bias=True)\n",
            "    (7): ReLU()\n",
            "    (8): Linear(in_features=128, out_features=10, bias=True)\n",
            "  )\n",
            ")\n",
            "\n",
            "\n",
            "Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[ 0.0278, -0.0129, -0.0248,  ..., -0.0124,  0.0336, -0.0132],\n",
            "        [-0.0072, -0.0081, -0.0110,  ...,  0.0126,  0.0263,  0.0319]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([-0.0027,  0.0271], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[-0.0412,  0.0019,  0.0014,  ...,  0.0143,  0.0306,  0.0187],\n",
            "        [ 0.0313, -0.0215,  0.0211,  ...,  0.0232, -0.0217, -0.0260]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([0.0087, 0.0350], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.weight | Size: torch.Size([256, 512]) | Values : tensor([[ 0.0365,  0.0256,  0.0313,  ...,  0.0166,  0.0124, -0.0308],\n",
            "        [-0.0005, -0.0287,  0.0248,  ...,  0.0297, -0.0201, -0.0130]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.bias | Size: torch.Size([256]) | Values : tensor([-0.0357,  0.0254], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.weight | Size: torch.Size([128, 256]) | Values : tensor([[-2.9274e-02,  1.3780e-02,  5.2827e-02,  5.0875e-02, -1.9230e-02,\n",
            "         -1.0163e-02,  1.2457e-02,  5.5698e-02, -2.9279e-02,  4.8277e-02,\n",
            "          3.6268e-02,  3.1603e-02, -4.2653e-02,  4.2452e-02, -1.9135e-03,\n",
            "         -2.9683e-02, -4.9169e-03,  5.5975e-02, -3.4191e-02, -1.9560e-02,\n",
            "          4.8433e-02, -3.5286e-02,  4.1537e-03,  4.6751e-02, -3.0345e-02,\n",
            "          5.4942e-02, -3.7154e-02,  5.1121e-02, -2.7289e-02,  4.4685e-02,\n",
            "         -5.1654e-02,  8.1628e-03,  8.7597e-03, -6.1067e-03, -2.8172e-03,\n",
            "         -9.5372e-03,  1.4827e-02, -3.4135e-03,  3.2737e-02, -5.9481e-02,\n",
            "         -1.1024e-02,  1.4485e-02,  1.4216e-02,  3.3212e-02,  2.8562e-02,\n",
            "         -4.8549e-02, -3.2974e-02, -1.8936e-02,  4.5814e-02, -3.4675e-02,\n",
            "          4.9272e-02, -4.9103e-02,  5.2506e-02, -6.4582e-03,  5.9780e-02,\n",
            "         -5.0944e-02, -1.2870e-02, -3.5189e-02,  2.2121e-03,  7.0902e-03,\n",
            "          5.3237e-02,  3.3037e-02, -1.7832e-02, -2.7124e-03,  2.5811e-02,\n",
            "         -3.2770e-02, -5.8041e-03, -1.3468e-02,  3.7336e-02, -3.7071e-02,\n",
            "         -4.3362e-02, -3.5773e-02,  1.1054e-02,  6.0503e-02, -1.1921e-03,\n",
            "         -1.5339e-02,  5.8428e-02,  3.7894e-02, -3.7758e-02,  5.8452e-02,\n",
            "          5.7457e-02, -1.0856e-02,  5.5541e-02,  5.6190e-02, -1.3842e-03,\n",
            "          5.4202e-02,  4.7140e-02,  3.0846e-02,  2.4284e-03,  4.2035e-02,\n",
            "         -1.6498e-02,  5.0308e-02, -1.5604e-03, -3.1787e-02, -3.6861e-02,\n",
            "          1.1256e-02, -3.2103e-02, -5.3838e-02,  3.2332e-02, -5.8874e-02,\n",
            "         -4.9765e-02,  2.5932e-02,  4.2631e-03,  2.8090e-02,  3.0358e-03,\n",
            "         -3.8927e-02, -4.6500e-02, -2.8241e-02, -1.8134e-02, -3.3381e-02,\n",
            "         -3.2126e-02, -9.7703e-03, -8.1560e-03,  3.2159e-02,  3.6267e-02,\n",
            "          5.5819e-02, -1.5827e-02, -2.0149e-02,  5.7407e-03, -5.3170e-02,\n",
            "         -5.2989e-03,  5.1744e-02,  1.0009e-03, -5.0286e-03, -1.9902e-02,\n",
            "         -1.1829e-02,  9.0933e-03, -3.7852e-02, -5.6587e-02, -4.6545e-02,\n",
            "          1.0241e-02, -2.7218e-02, -3.7415e-02,  1.5122e-02,  2.5628e-02,\n",
            "          2.3422e-02,  4.4085e-02,  7.6849e-03, -1.5089e-02, -5.2455e-02,\n",
            "          3.6958e-02, -3.3057e-02, -2.1343e-02,  3.2202e-03,  5.0764e-02,\n",
            "         -5.0232e-02,  7.7291e-03,  5.2849e-02,  1.8218e-02, -5.5173e-02,\n",
            "         -2.3592e-02,  5.9666e-03, -3.2928e-02,  4.0701e-02, -1.5628e-02,\n",
            "         -1.9816e-02,  2.2158e-03,  5.0470e-02,  1.6835e-02,  4.2945e-02,\n",
            "         -2.6378e-02, -2.5364e-02,  1.1330e-02, -1.4518e-03,  1.7837e-02,\n",
            "          4.6717e-02, -6.5578e-03, -8.7414e-03, -1.6265e-02,  2.0827e-02,\n",
            "          6.0010e-02, -1.1735e-02, -2.9801e-02,  3.2120e-02, -1.6832e-03,\n",
            "         -3.4220e-02,  3.2761e-02,  2.6373e-02,  3.0891e-02, -1.3696e-02,\n",
            "          1.2604e-02, -2.2871e-02, -1.2686e-02, -3.5135e-04,  2.7539e-02,\n",
            "         -4.1180e-02,  1.1250e-02,  1.9896e-02,  5.8773e-02, -5.7396e-02,\n",
            "          3.6041e-03, -1.1069e-02, -4.4409e-02, -1.1794e-02,  1.7444e-02,\n",
            "         -5.1225e-02,  1.4974e-02, -3.9397e-03,  5.1788e-02, -6.2161e-02,\n",
            "          4.7372e-02, -5.2343e-02, -4.6392e-02, -1.0016e-02,  5.0793e-02,\n",
            "         -3.0040e-02,  3.4501e-02, -1.1309e-02, -2.5879e-02,  5.3696e-02,\n",
            "          5.5145e-02,  5.6759e-02,  5.6807e-02, -5.2119e-02,  4.4108e-02,\n",
            "          1.0007e-02, -4.8643e-02,  5.4839e-03,  1.5474e-02, -3.9767e-02,\n",
            "          2.7144e-02,  5.6302e-02, -3.6488e-02, -3.5802e-03,  4.3924e-02,\n",
            "         -1.2289e-02, -4.6071e-03,  5.5746e-03, -1.3775e-02,  5.0893e-02,\n",
            "         -4.5548e-02, -2.0046e-02,  1.6005e-02,  4.0596e-02, -3.9100e-02,\n",
            "         -1.9106e-03,  2.9165e-02, -5.0274e-02,  1.2323e-02,  1.6975e-02,\n",
            "         -1.0270e-03,  4.1560e-03, -4.2874e-02, -1.9402e-02,  3.6311e-02,\n",
            "          3.4060e-02, -4.5780e-02, -5.5382e-02, -5.5350e-02, -3.3191e-02,\n",
            "          3.9039e-02, -3.0044e-02, -2.3972e-02, -5.4048e-03,  2.2247e-02,\n",
            "          3.6889e-02],\n",
            "        [ 2.1558e-02,  2.9044e-02,  3.2041e-02, -3.5355e-03,  4.5457e-02,\n",
            "         -4.6631e-03,  1.2817e-02, -4.3786e-02, -2.2908e-02, -1.7820e-02,\n",
            "         -4.5877e-02, -5.5790e-02, -1.9240e-02, -5.0121e-02, -1.2478e-02,\n",
            "          3.4865e-02,  4.9011e-02,  1.1026e-02,  4.6168e-02,  4.1345e-02,\n",
            "          4.0911e-02, -4.6468e-02,  8.6337e-04,  2.3260e-02,  7.5517e-03,\n",
            "         -2.5782e-02, -2.2014e-02,  9.8780e-03,  1.2887e-03, -4.8195e-02,\n",
            "         -3.0667e-02,  7.5565e-03,  1.5519e-02, -1.9429e-03, -7.2959e-03,\n",
            "         -3.0348e-02,  2.7293e-02,  5.3291e-02,  6.0465e-02, -3.0348e-02,\n",
            "         -6.2179e-02, -5.0287e-02,  3.3160e-02,  4.1224e-02, -5.8046e-02,\n",
            "          2.7588e-02, -3.6547e-02, -4.1995e-02,  5.2761e-02, -1.2340e-02,\n",
            "          5.9345e-02, -3.7875e-02, -2.6183e-02, -1.0054e-02, -5.6655e-02,\n",
            "         -1.7532e-02,  2.2693e-02,  4.7242e-02,  3.1101e-02, -1.0183e-02,\n",
            "          4.2121e-02, -4.0649e-02, -3.6632e-02,  3.9615e-02,  3.1088e-02,\n",
            "         -2.0292e-02,  3.6358e-03, -2.0441e-02, -3.5945e-02,  5.4830e-02,\n",
            "         -5.2953e-02, -1.7978e-02, -2.1878e-02,  3.3330e-02, -1.5450e-02,\n",
            "          5.4860e-02,  3.4061e-02,  6.8179e-03,  5.0682e-02, -2.0791e-02,\n",
            "         -5.4096e-03,  5.6125e-02,  1.8441e-02, -3.0411e-02,  6.8333e-03,\n",
            "         -1.1072e-03,  5.1519e-03, -1.7936e-02, -4.1649e-02,  6.0051e-03,\n",
            "          3.3213e-02,  4.9440e-02, -5.4141e-02, -2.5814e-02, -4.9514e-02,\n",
            "         -2.9088e-02,  8.5302e-05,  3.1101e-02,  3.8543e-02, -1.0130e-02,\n",
            "          3.8384e-03, -3.3516e-02,  7.7214e-03,  1.2785e-02, -6.0468e-02,\n",
            "         -8.6471e-03, -4.4801e-03, -5.0835e-02,  3.0613e-02, -2.9293e-02,\n",
            "         -3.5621e-02, -2.5863e-04,  4.6484e-02,  3.9474e-02,  4.7867e-02,\n",
            "         -2.2605e-02, -3.6656e-02,  5.0695e-02,  5.8794e-02, -1.5402e-02,\n",
            "         -1.3983e-02, -5.3811e-02,  5.0337e-02,  2.6725e-02, -4.1369e-02,\n",
            "         -4.7665e-02,  3.2171e-02, -4.4517e-02, -4.5593e-02, -2.8513e-02,\n",
            "         -3.8268e-03, -8.8652e-03,  5.0866e-02,  1.2241e-02,  1.1003e-02,\n",
            "          5.8534e-02, -1.0613e-03,  6.2220e-02, -4.5403e-02,  4.5186e-02,\n",
            "          4.4024e-02,  4.0310e-02, -1.1960e-02,  5.7525e-02,  1.3867e-02,\n",
            "         -2.8943e-03,  3.6153e-02, -5.0853e-02,  3.1561e-02, -5.2996e-02,\n",
            "         -5.3160e-02, -5.7494e-02, -4.0073e-02, -1.6324e-03,  2.2703e-03,\n",
            "          5.9139e-02, -6.0087e-02, -5.6316e-03,  5.5471e-02, -2.2258e-02,\n",
            "         -3.7783e-02, -3.9622e-02, -6.2126e-02,  2.0316e-02,  5.5294e-02,\n",
            "          3.4916e-02, -4.9245e-02,  2.3218e-02, -4.1774e-02,  2.1464e-02,\n",
            "          3.9853e-02, -5.7861e-02,  1.8125e-02, -3.2349e-02, -2.7170e-02,\n",
            "          3.4672e-02,  2.9171e-02, -4.9255e-02, -8.6152e-03, -2.0139e-02,\n",
            "          4.0586e-02,  8.1711e-03, -4.5513e-02, -6.3710e-03,  2.4451e-02,\n",
            "          4.9702e-02, -5.0016e-02, -5.5351e-02,  2.7315e-02, -4.7136e-02,\n",
            "          2.4120e-03, -6.5553e-03, -5.3093e-02, -5.8387e-02, -3.5005e-02,\n",
            "          3.6504e-02, -5.5233e-02,  5.3821e-02, -3.8142e-03, -1.1397e-02,\n",
            "          7.9970e-03,  6.7878e-03,  2.9160e-02, -1.4089e-02,  3.7059e-02,\n",
            "          2.2300e-02, -5.4698e-02, -1.2852e-02,  3.3871e-02, -5.3337e-02,\n",
            "         -1.4481e-02,  3.1781e-02, -1.8053e-02, -1.9491e-02,  2.8219e-03,\n",
            "         -4.8260e-02,  6.5680e-03,  1.8207e-02, -2.1046e-02,  1.5498e-02,\n",
            "          6.0074e-02, -5.3173e-02,  4.7691e-05,  1.2600e-02,  4.5374e-02,\n",
            "          4.6579e-02,  5.3986e-02,  2.7116e-04,  7.0559e-03, -3.6273e-02,\n",
            "          2.6173e-02, -4.5059e-02,  3.0662e-03, -4.4840e-02, -4.0980e-02,\n",
            "          1.4239e-02, -4.5791e-02,  1.0876e-02,  5.0296e-02,  1.4070e-02,\n",
            "         -6.1664e-02,  1.4520e-02,  4.8623e-02,  5.7482e-02,  1.8529e-02,\n",
            "          2.9572e-02, -2.9499e-02,  3.4958e-02,  1.6271e-02,  4.3489e-03,\n",
            "          3.2030e-02, -3.6334e-02,  3.6179e-02,  6.2277e-02,  4.2532e-02,\n",
            "         -4.7089e-02]], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.bias | Size: torch.Size([128]) | Values : tensor([-0.0138,  0.0324], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.8.weight | Size: torch.Size([10, 128]) | Values : tensor([[-0.0253, -0.0521,  0.0292, -0.0640, -0.0835, -0.0660, -0.0755, -0.0868,\n",
            "          0.0269,  0.0285,  0.0283, -0.0784, -0.0875, -0.0257, -0.0324,  0.0479,\n",
            "         -0.0075, -0.0741, -0.0322, -0.0627, -0.0356, -0.0765, -0.0209, -0.0851,\n",
            "          0.0001, -0.0703, -0.0634,  0.0539, -0.0122, -0.0865,  0.0423,  0.0400,\n",
            "         -0.0091, -0.0310,  0.0546, -0.0796,  0.0326, -0.0872,  0.0806, -0.0120,\n",
            "         -0.0347,  0.0853, -0.0697, -0.0412, -0.0060, -0.0159, -0.0560, -0.0749,\n",
            "          0.0488, -0.0336, -0.0310,  0.0471,  0.0349,  0.0451,  0.0153, -0.0629,\n",
            "          0.0510, -0.0428, -0.0487,  0.0181,  0.0412, -0.0750, -0.0040,  0.0824,\n",
            "         -0.0598, -0.0078, -0.0324, -0.0464,  0.0778, -0.0142, -0.0153,  0.0837,\n",
            "          0.0295, -0.0500, -0.0078, -0.0424,  0.0282, -0.0796, -0.0873, -0.0092,\n",
            "          0.0493,  0.0406,  0.0407, -0.0602, -0.0189,  0.0388,  0.0663, -0.0261,\n",
            "          0.0706, -0.0595,  0.0158,  0.0344,  0.0514, -0.0463,  0.0734, -0.0307,\n",
            "         -0.0446, -0.0337,  0.0774,  0.0813, -0.0582, -0.0717, -0.0245, -0.0699,\n",
            "         -0.0781,  0.0605,  0.0112, -0.0008,  0.0123,  0.0178, -0.0427, -0.0591,\n",
            "         -0.0756, -0.0835,  0.0052, -0.0435,  0.0107,  0.0473,  0.0673, -0.0315,\n",
            "          0.0670, -0.0696,  0.0304, -0.0614, -0.0416, -0.0668,  0.0250,  0.0623],\n",
            "        [-0.0687, -0.0591, -0.0392,  0.0445, -0.0791, -0.0818,  0.0251,  0.0794,\n",
            "          0.0347,  0.0697,  0.0704, -0.0767,  0.0032, -0.0598, -0.0352, -0.0726,\n",
            "          0.0121,  0.0154,  0.0504, -0.0778, -0.0460, -0.0170, -0.0289, -0.0355,\n",
            "         -0.0020,  0.0165, -0.0226,  0.0027,  0.0212, -0.0518,  0.0436,  0.0127,\n",
            "         -0.0844, -0.0676,  0.0720,  0.0307, -0.0639,  0.0807,  0.0263, -0.0819,\n",
            "         -0.0476, -0.0626,  0.0870,  0.0500,  0.0213,  0.0551,  0.0689, -0.0851,\n",
            "          0.0095, -0.0436, -0.0339,  0.0730, -0.0441,  0.0781, -0.0155, -0.0425,\n",
            "          0.0510,  0.0305, -0.0859, -0.0093,  0.0186, -0.0794,  0.0728, -0.0035,\n",
            "         -0.0046,  0.0223, -0.0246,  0.0563,  0.0482, -0.0276, -0.0337,  0.0584,\n",
            "         -0.0300,  0.0128,  0.0200,  0.0161,  0.0375,  0.0314, -0.0028, -0.0528,\n",
            "          0.0123, -0.0662, -0.0861, -0.0733,  0.0374,  0.0763,  0.0299, -0.0254,\n",
            "          0.0686,  0.0447, -0.0709, -0.0086,  0.0830, -0.0219, -0.0300, -0.0128,\n",
            "          0.0465,  0.0258, -0.0556,  0.0599, -0.0695,  0.0567, -0.0676,  0.0232,\n",
            "          0.0019,  0.0436, -0.0536, -0.0030,  0.0647,  0.0517,  0.0253,  0.0050,\n",
            "          0.0060, -0.0327, -0.0781, -0.0422, -0.0400, -0.0863,  0.0737,  0.0579,\n",
            "         -0.0171,  0.0133, -0.0288,  0.0248, -0.0801, -0.0186,  0.0091, -0.0864]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.8.bias | Size: torch.Size([10]) | Values : tensor([-0.0523,  0.0833], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(f\"Model structure: {model}\\n\\n\")\n",
        "\n",
        "for name, param in model.named_parameters():\n",
        "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4QwdrYFedJN"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoXPbcy7edJN"
      },
      "source": [
        "## Further Reading\n",
        "- [torch.nn API](https://pytorch.org/docs/stable/nn.html)\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    },
    "colab": {
      "provenance": []
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}