task-2-nn/IMT2020100.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "eusTinvWelOk"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U7fEXC_MelOn"
      },
      "source": [
        "\n",
        "[Learn the Basics](intro.html) ||\n",
        "[Quickstart](quickstart_tutorial.html) ||\n",
        "[Tensors](tensorqs_tutorial.html) ||\n",
        "[Datasets & DataLoaders](data_tutorial.html) ||\n",
        "[Transforms](transforms_tutorial.html) ||\n",
        "**Build Model** ||\n",
        "[Autograd](autogradqs_tutorial.html) ||\n",
        "[Optimization](optimization_tutorial.html) ||\n",
        "[Save & Load Model](saveloadrun_tutorial.html)\n",
        "\n",
        "# Build the Neural Network\n",
        "\n",
        "Neural networks comprise of layers/modules that perform operations on data.\n",
        "The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace provides all the building blocks you need to\n",
        "build your own neural network. Every module in PyTorch subclasses the [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html).\n",
        "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
        "building and managing complex architectures easily.\n",
        "\n",
        "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "-zpWEvXPelOr"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision import datasets, transforms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wDLZdSTVelOs"
      },
      "source": [
        "## Get Device for Training\n",
        "We want to be able to train our model on a hardware accelerator like the GPU,\n",
        "if it is available. Let's check to see if\n",
        "[torch.cuda](https://pytorch.org/docs/stable/notes/cuda.html) is available, else we\n",
        "continue to use the CPU.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "yHVtMXdqelOu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6294c20a-5a17-4e99-ac25-cd8c0c3c6fc0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n"
          ]
        }
      ],
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(f\"Using {device} device\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZoq7DTLelOv"
      },
      "source": [
        "## Define the Class\n",
        "We define our neural network by subclassing ``nn.Module``, and\n",
        "initialize the neural network layers in ``__init__``. Every ``nn.Module`` subclass implements\n",
        "the operations on input data in the ``forward`` method.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "yrQjN-d_elOw"
      },
      "outputs": [],
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NeuralNetwork, self).__init__()\n",
        "        self.flatten = nn.Flatten()\n",
        "        self.linear_relu_stack = nn.Sequential(\n",
        "            nn.Linear(28*28, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 456),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(456,512),\n",
        "            nn.GELU(),\n",
        "            nn.Linear(512, 10),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.flatten(x)\n",
        "        logits = self.linear_relu_stack(x)\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfsNCM_GelOx"
      },
      "source": [
        "We create an instance of ``NeuralNetwork``, and move it to the ``device``, and print\n",
        "its structure.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "ibTydPJ2elOy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6a6b76b6-2b13-4e3f-9649-a0d71ebbe3da"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=456, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=456, out_features=512, bias=True)\n",
            "    (5): GELU(approximate=none)\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "model = NeuralNetwork().to(device)\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvQbL6GqelO0"
      },
      "source": [
        "To use the model, we pass it the input data. This executes the model's ``forward``,\n",
        "along with some [background operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866).\n",
        "Do not call ``model.forward()`` directly!\n",
        "\n",
        "Calling the model on the input returns a 2-dimensional tensor with dim=0 corresponding to each output of 10 raw predicted values for each class, and dim=1 corresponding to the individual values of each output.  .\n",
        "We get the prediction probabilities by passing it through an instance of the ``nn.Softmax`` module.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "6AhtctV2elO1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fd951e53-df12-4ba1-c26d-3c5f336459de"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predicted class: tensor([3], device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "X = torch.rand(1, 28, 28, device=device)\n",
        "logits = model(X)\n",
        "pred_probab = nn.Softmax(dim=1)(logits)\n",
        "y_pred = pred_probab.argmax(1)\n",
        "print(f\"Predicted class: {y_pred}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NRfDlRwDelO3"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoDHuMoAelO3"
      },
      "source": [
        "## Model Layers\n",
        "\n",
        "Let's break down the layers in the FashionMNIST model. To illustrate it, we\n",
        "will take a sample minibatch of 3 images of size 28x28 and see what happens to it as\n",
        "we pass it through the network.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "KoBrTlkJelO3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "17cf2cdb-f590-4aba-de9c-efb4226bbd69"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "input_image = torch.rand(3,28,28)\n",
        "print(input_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c2XYnKMDelO4"
      },
      "source": [
        "### nn.Flatten\n",
        "We initialize the [nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html)\n",
        "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (\n",
        "the minibatch dimension (at dim=0) is maintained).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "s5QW3lyUelO5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0319ee3e-f323-4e74-a7e4-5591e0be8963"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 784])\n"
          ]
        }
      ],
      "source": [
        "flatten = nn.Flatten()\n",
        "flat_image = flatten(input_image)\n",
        "print(flat_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i9Hy3eEGelO5"
      },
      "source": [
        "### nn.Linear\n",
        "The [linear layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
        "is a module that applies a linear transformation on the input using its stored weights and biases.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "NOCoBaf3elO6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "200f46d4-23ec-45f6-cd1c-d97528dfec04"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 20])\n"
          ]
        }
      ],
      "source": [
        "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
        "hidden1 = layer1(flat_image)\n",
        "print(hidden1.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_3tA3K1elO7"
      },
      "source": [
        "### nn.ReLU\n",
        "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
        "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
        "learn a wide variety of phenomena.\n",
        "\n",
        "In this model, we use [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) between our\n",
        "linear layers, but there's other activations to introduce non-linearity in your model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "K0nj2xraelO7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c4c1d61b-d11a-4561-e159-c68a57e679b0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before ReLU: tensor([[ 0.4169,  0.3640, -0.1140, -0.0493, -0.6425,  0.0570,  0.1749, -0.2540,\n",
            "         -0.6943, -0.0534, -0.3428,  0.3055, -0.3597, -0.2085,  0.3280,  0.1757,\n",
            "          0.1974,  0.5915, -0.0408, -0.1036],\n",
            "        [ 0.1949,  0.1970,  0.3231,  0.2623, -0.5391,  0.3257,  0.2625, -0.2868,\n",
            "         -0.0309, -0.0491, -0.1200,  0.1136, -0.4560, -0.3390,  0.3957,  0.1215,\n",
            "          0.0180,  0.1229, -0.3354,  0.0349],\n",
            "        [ 0.2808, -0.0418, -0.1534,  0.3812, -0.1064, -0.0023,  0.0862, -0.0105,\n",
            "         -0.1595, -0.1115, -0.5014,  0.4094, -0.2553, -0.1795,  0.1197,  0.1216,\n",
            "         -0.0010,  0.2027,  0.0143, -0.5300]], grad_fn=<AddmmBackward0>)\n",
            "\n",
            "\n",
            "After ReLU: tensor([[0.4169, 0.3640, 0.0000, 0.0000, 0.0000, 0.0570, 0.1749, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.3055, 0.0000, 0.0000, 0.3280, 0.1757, 0.1974, 0.5915,\n",
            "         0.0000, 0.0000],\n",
            "        [0.1949, 0.1970, 0.3231, 0.2623, 0.0000, 0.3257, 0.2625, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.1136, 0.0000, 0.0000, 0.3957, 0.1215, 0.0180, 0.1229,\n",
            "         0.0000, 0.0349],\n",
            "        [0.2808, 0.0000, 0.0000, 0.3812, 0.0000, 0.0000, 0.0862, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.4094, 0.0000, 0.0000, 0.1197, 0.1216, 0.0000, 0.2027,\n",
            "         0.0143, 0.0000]], grad_fn=<ReluBackward0>)\n"
          ]
        }
      ],
      "source": [
        "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
        "hidden1 = nn.ReLU()(hidden1)\n",
        "print(f\"After ReLU: {hidden1}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bug20ufXelO7"
      },
      "source": [
        "### nn.Sequential\n",
        "[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) is an ordered\n",
        "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
        "sequential containers to put together a quick network like ``seq_modules``.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "p-RmfGDtelO8"
      },
      "outputs": [],
      "source": [
        "seq_modules = nn.Sequential(\n",
        "    flatten,\n",
        "    layer1,\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(20, 10)\n",
        ")\n",
        "input_image = torch.rand(3,28,28)\n",
        "logits = seq_modules(input_image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NchtohHielO8"
      },
      "source": [
        "### nn.Softmax\n",
        "The last linear layer of the neural network returns `logits` - raw values in [-\\infty, \\infty] - which are passed to the\n",
        "[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) module. The logits are scaled to values\n",
        "[0, 1] representing the model's predicted probabilities for each class. ``dim`` parameter indicates the dimension along\n",
        "which the values must sum to 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "15R_89skelO9"
      },
      "outputs": [],
      "source": [
        "softmax = nn.Softmax(dim=1)\n",
        "pred_probab = softmax(logits)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMwKPM07elO9"
      },
      "source": [
        "## Model Parameters\n",
        "Many layers inside a neural network are *parameterized*, i.e. have associated weights\n",
        "and biases that are optimized during training. Subclassing ``nn.Module`` automatically\n",
        "tracks all fields defined inside your model object, and makes all parameters\n",
        "accessible using your model's ``parameters()`` or ``named_parameters()`` methods.\n",
        "\n",
        "In this example, we iterate over each parameter, and print its size and a preview of its values.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "0dJf0P96elO-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95c1d52b-a16b-497a-a729-d77fe513ad0c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model structure: NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=456, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=456, out_features=512, bias=True)\n",
            "    (5): GELU(approximate=none)\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n",
            "\n",
            "\n",
            "Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[ 0.0002, -0.0026,  0.0311,  ...,  0.0201, -0.0192,  0.0189],\n",
            "        [ 0.0251, -0.0154,  0.0235,  ...,  0.0055,  0.0277, -0.0224]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([-0.0337, -0.0305], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.weight | Size: torch.Size([456, 512]) | Values : tensor([[-0.0218,  0.0352, -0.0112,  ...,  0.0404,  0.0015, -0.0341],\n",
            "        [ 0.0204, -0.0388, -0.0143,  ..., -0.0294,  0.0168,  0.0260]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.bias | Size: torch.Size([456]) | Values : tensor([ 0.0402, -0.0005], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.weight | Size: torch.Size([512, 456]) | Values : tensor([[ 1.1096e-02,  1.8968e-02,  2.8500e-02,  4.3018e-02, -4.0054e-02,\n",
            "         -3.1913e-03, -2.9795e-03, -1.9748e-02,  4.1305e-02,  2.1585e-02,\n",
            "          3.8273e-02,  2.5143e-02, -4.0029e-02,  3.8317e-02,  2.9308e-02,\n",
            "         -4.4441e-02,  2.0887e-03, -3.3574e-02, -4.5150e-02,  1.2592e-02,\n",
            "         -4.6039e-02,  2.5957e-02,  1.4792e-03,  1.8366e-02, -1.5102e-02,\n",
            "         -4.6358e-02,  2.4909e-02,  4.0347e-02,  4.5707e-02,  6.4143e-03,\n",
            "          3.9613e-02, -2.3806e-03,  3.7308e-02, -1.8185e-02, -1.1575e-02,\n",
            "          4.1879e-02, -3.4669e-02, -1.9768e-02, -2.1376e-02, -2.2513e-02,\n",
            "         -3.2678e-02, -4.3773e-03, -2.4162e-02, -1.1563e-02, -1.9843e-02,\n",
            "          2.3732e-02,  2.6624e-02, -2.1662e-02, -6.4483e-03, -7.6930e-03,\n",
            "          2.5574e-02, -1.7188e-03, -4.6343e-02, -3.8041e-02, -1.6744e-02,\n",
            "          3.0863e-02,  3.6978e-02,  2.4062e-02,  3.1214e-02,  3.0067e-02,\n",
            "          4.1017e-02, -4.4436e-03, -1.1267e-02, -1.4431e-02,  9.1454e-03,\n",
            "         -1.7568e-03, -3.4916e-02,  2.5752e-02, -2.3192e-02, -3.1753e-03,\n",
            "          3.0609e-02,  3.5740e-02, -1.4843e-03, -1.8362e-02, -1.6383e-02,\n",
            "         -1.1577e-02,  3.2298e-02, -1.9821e-02,  9.9172e-03,  7.0465e-03,\n",
            "         -1.9392e-03, -2.2520e-02, -5.8266e-03,  4.3648e-03, -7.6624e-03,\n",
            "         -3.8547e-02, -3.8572e-02,  3.8821e-05, -1.7586e-02, -3.9733e-02,\n",
            "          3.4766e-02, -2.9329e-02, -4.0989e-02,  2.1525e-02,  3.6902e-02,\n",
            "         -7.5727e-03,  1.1880e-03,  1.4299e-03, -3.4268e-02,  5.2901e-03,\n",
            "          1.8690e-02, -4.4536e-02, -3.7697e-02, -1.6228e-02, -1.2350e-02,\n",
            "          1.2451e-02,  1.9263e-02, -3.7164e-02,  7.1927e-03,  3.2733e-02,\n",
            "         -1.1513e-03, -1.0395e-02,  3.1008e-02, -2.8715e-02, -1.6242e-02,\n",
            "         -4.3847e-02, -3.3340e-02, -4.5106e-02, -1.8782e-02, -1.9632e-02,\n",
            "          1.8932e-03,  4.4863e-02, -8.2047e-03,  1.2639e-02,  2.0646e-02,\n",
            "          3.9238e-02,  1.1189e-02, -9.1440e-03, -1.7406e-02, -2.9726e-02,\n",
            "          2.3356e-03,  4.3311e-03,  2.8472e-03,  3.0007e-02, -1.1045e-02,\n",
            "         -2.8036e-02,  3.1987e-02, -1.2278e-02,  1.4997e-02, -2.7286e-02,\n",
            "          3.3779e-02, -3.4471e-02,  2.1963e-02, -1.9085e-02, -4.2988e-02,\n",
            "         -3.8802e-02, -6.3566e-03,  6.5394e-03, -2.8124e-02,  1.5263e-02,\n",
            "         -1.9274e-02,  3.2544e-02, -2.1240e-02, -3.7202e-02,  3.3355e-02,\n",
            "          1.3480e-02, -3.0638e-02,  2.0776e-02,  3.9235e-02,  1.8996e-02,\n",
            "         -3.1719e-02,  9.1966e-03,  1.3053e-02, -2.2879e-03,  2.3062e-02,\n",
            "         -4.0633e-02,  4.1495e-02, -4.5913e-02, -2.3858e-02, -1.7350e-02,\n",
            "          2.9827e-02, -2.3319e-02, -1.7193e-02, -2.3586e-02,  3.5325e-02,\n",
            "          2.5808e-02, -1.2511e-02,  3.5749e-02,  1.5629e-03,  3.0026e-02,\n",
            "          4.0861e-02, -2.0958e-02, -4.5966e-02,  6.3889e-03,  2.5028e-02,\n",
            "         -3.1202e-02, -2.9147e-03, -3.0023e-02, -2.6346e-02,  4.0465e-02,\n",
            "         -2.9747e-02,  3.1073e-02,  2.5605e-02, -5.9870e-03,  2.1699e-02,\n",
            "         -5.5559e-03, -2.8625e-02,  4.2747e-02, -5.9783e-03, -1.9121e-02,\n",
            "         -4.0236e-02,  2.8840e-02,  1.7262e-02,  4.6128e-02, -5.3016e-04,\n",
            "          3.5823e-03, -4.6109e-03, -3.6935e-02, -5.8467e-03,  1.3576e-02,\n",
            "         -3.7429e-02,  8.3813e-03, -1.5591e-03, -1.6626e-02, -4.0336e-02,\n",
            "          3.9411e-02, -3.9393e-02, -1.1562e-02,  4.0753e-03, -1.2465e-02,\n",
            "          2.6864e-02,  2.9298e-02, -1.1631e-02, -5.9988e-03,  1.1320e-02,\n",
            "          3.5696e-02, -1.2555e-02,  5.9147e-03,  3.5220e-04,  2.6457e-02,\n",
            "          2.6988e-02, -8.4861e-03,  1.1088e-02, -4.2341e-02,  4.7076e-03,\n",
            "          2.2456e-02,  1.4970e-02,  4.2415e-02,  3.1449e-02, -3.9898e-02,\n",
            "          2.3712e-02, -4.4689e-02,  1.4750e-02,  9.4631e-03, -3.7452e-03,\n",
            "          3.0042e-02,  5.6843e-04, -3.4353e-02,  2.3645e-02,  2.3738e-02,\n",
            "          3.5314e-02,  1.4620e-02,  8.6797e-03,  2.5334e-02, -4.6649e-02,\n",
            "          4.1336e-03, -8.4711e-03,  5.8950e-03, -2.4546e-02, -2.2321e-02,\n",
            "          3.9851e-02, -2.1053e-03, -7.4224e-04, -3.1696e-02, -7.9293e-03,\n",
            "          2.1903e-04,  3.0632e-02,  2.0258e-03, -2.5828e-02,  3.3457e-02,\n",
            "          7.6186e-03,  2.6239e-02, -2.4819e-02, -2.9373e-02, -4.3950e-02,\n",
            "         -4.1456e-02, -4.0717e-02,  7.5443e-04, -6.5857e-03,  4.0613e-02,\n",
            "          8.8676e-03, -2.2142e-02, -2.5477e-02, -4.6592e-02,  3.3911e-02,\n",
            "         -1.2784e-02, -3.1280e-02,  1.6962e-02,  1.2442e-02,  2.8168e-02,\n",
            "         -8.6806e-03, -9.2620e-03, -3.5964e-02, -4.5535e-02,  6.7479e-03,\n",
            "         -1.4543e-03, -3.0308e-03, -2.9986e-02, -1.0903e-02, -3.7916e-02,\n",
            "         -2.7668e-02, -3.1442e-02, -2.8655e-02,  9.1693e-03, -1.8858e-02,\n",
            "          4.2907e-02,  4.5407e-02,  5.2912e-03, -2.7208e-02, -2.4317e-02,\n",
            "         -1.0974e-02, -4.3997e-02,  4.3543e-02,  2.9806e-02, -2.0251e-03,\n",
            "         -3.4835e-02,  1.2078e-02, -2.5297e-02, -3.7073e-02,  2.4250e-03,\n",
            "          3.6698e-02,  3.5896e-02,  3.7915e-02, -8.7892e-04,  2.9294e-02,\n",
            "         -1.5874e-02,  4.4181e-02, -1.2709e-02, -1.1565e-03, -7.3424e-03,\n",
            "          1.8049e-02, -1.9279e-02, -3.6630e-03, -5.4018e-03,  1.4353e-02,\n",
            "          2.2199e-02, -2.6759e-02,  2.8730e-02,  3.4716e-02,  3.4396e-02,\n",
            "         -2.4412e-02, -1.2748e-02,  7.1584e-03, -1.7894e-02, -8.2034e-03,\n",
            "         -2.1660e-02,  4.3492e-02,  3.3954e-02,  3.9654e-02, -2.9993e-03,\n",
            "          3.2111e-03, -1.9636e-02, -3.1328e-02,  1.4834e-02, -4.4376e-04,\n",
            "         -4.4319e-02,  4.1632e-02, -4.0854e-02, -3.2716e-03, -4.4777e-02,\n",
            "         -2.4594e-02,  3.2452e-02, -6.5283e-03, -1.4107e-03, -3.8902e-02,\n",
            "         -4.4592e-02, -2.2355e-02, -1.2801e-02, -5.6531e-03, -1.6907e-02,\n",
            "         -4.6386e-02,  2.7449e-03, -2.8267e-02, -7.9526e-03,  3.6287e-02,\n",
            "         -1.6875e-02,  4.4763e-02, -5.9143e-03,  1.7773e-02, -1.9996e-03,\n",
            "         -7.5820e-03,  3.3438e-02,  7.8160e-03,  3.4414e-04, -1.4730e-02,\n",
            "          1.5281e-02,  3.7988e-02,  6.3727e-03, -1.0993e-02,  2.9438e-02,\n",
            "         -1.9933e-02,  3.8970e-02, -3.4968e-02,  2.4677e-03, -4.3410e-02,\n",
            "          1.5677e-03,  2.2760e-02, -4.0766e-02,  4.6647e-02, -2.8536e-02,\n",
            "          3.8074e-02,  9.4052e-04,  4.2010e-02,  2.1673e-02,  3.5033e-02,\n",
            "         -3.9132e-02,  3.0024e-02,  8.2803e-03, -2.7799e-02, -1.7924e-03,\n",
            "         -1.0239e-04, -2.9290e-02,  4.6831e-03,  3.2684e-02,  2.0972e-04,\n",
            "          1.1126e-02,  1.3863e-02, -1.9791e-02, -1.4481e-02, -1.5580e-02,\n",
            "         -4.4247e-02,  1.5637e-02, -1.3204e-02,  4.4822e-02,  4.1515e-02,\n",
            "          9.4136e-03, -1.7548e-02,  1.0439e-02,  1.9944e-02,  1.2742e-03,\n",
            "         -4.5057e-02, -2.5316e-02,  1.1079e-03, -1.1013e-02, -2.1042e-02,\n",
            "          1.6692e-02,  3.2005e-02, -1.8674e-02,  3.8628e-02, -2.2422e-02,\n",
            "         -1.1255e-02,  3.2560e-02,  3.1902e-02, -2.0989e-02, -5.3120e-03,\n",
            "         -4.1164e-02,  1.7501e-02,  3.5237e-02, -1.2240e-02, -4.7082e-03,\n",
            "          4.2200e-02, -6.0291e-04,  2.1808e-02,  3.6579e-02,  1.9530e-02,\n",
            "          4.3034e-02],\n",
            "        [ 3.3903e-02, -4.0909e-02,  3.5583e-02, -4.0417e-02,  4.4614e-02,\n",
            "         -3.3709e-02,  4.4544e-03,  1.4489e-02,  3.3484e-02, -4.6776e-02,\n",
            "          3.3744e-02,  1.8253e-03,  1.1853e-02, -2.8479e-02, -4.4139e-02,\n",
            "          3.0736e-02, -2.2360e-02, -1.7586e-02, -1.8373e-02,  1.6713e-02,\n",
            "         -5.3848e-03,  2.2747e-02, -3.1827e-02, -1.2795e-02, -4.3829e-02,\n",
            "         -4.2121e-02,  4.1140e-02,  1.8015e-02,  4.2420e-02, -2.0235e-02,\n",
            "          6.1873e-03, -2.0976e-02,  2.5585e-02,  2.7494e-02,  7.0417e-03,\n",
            "         -4.6530e-02, -2.4677e-02,  3.8077e-02,  3.1109e-02,  4.3254e-02,\n",
            "         -3.0324e-02, -3.6158e-02, -2.3463e-02, -1.9359e-02,  1.8768e-04,\n",
            "         -8.3840e-03,  1.1837e-02, -3.3278e-02, -4.3936e-02,  4.2258e-02,\n",
            "         -1.7651e-02,  3.7197e-02,  1.9461e-02, -2.8322e-02,  8.3180e-03,\n",
            "         -3.4899e-02, -1.9480e-02,  4.4505e-02,  1.5948e-02, -4.5035e-03,\n",
            "          2.7267e-02,  3.9454e-02,  8.1575e-03, -1.8945e-02, -9.4672e-03,\n",
            "          8.1987e-03,  8.0804e-03, -6.2276e-03,  5.9735e-03, -2.9205e-02,\n",
            "          3.3898e-02, -1.8596e-02,  2.0772e-02, -3.2672e-02,  4.6056e-02,\n",
            "         -4.0934e-02, -1.4888e-02, -2.9460e-02,  4.6435e-02,  2.1506e-02,\n",
            "         -1.8472e-02,  2.4357e-02,  3.8926e-02, -4.5656e-02,  4.3448e-03,\n",
            "          3.2138e-02,  9.0320e-03, -2.1608e-02,  4.5566e-02,  4.8293e-03,\n",
            "         -6.3081e-03,  2.6430e-02, -4.2722e-03, -2.1193e-02, -3.0767e-02,\n",
            "          2.1646e-02, -6.3992e-03, -3.5278e-04, -6.9379e-03, -2.6334e-02,\n",
            "          2.5049e-03,  3.9184e-02,  2.1940e-02,  3.7082e-02,  3.6710e-02,\n",
            "         -2.4008e-02, -1.3027e-03, -1.4804e-02,  2.6735e-03, -3.7793e-02,\n",
            "         -3.6401e-02,  3.5524e-02,  3.3206e-02, -4.5411e-02, -1.7232e-02,\n",
            "          5.8472e-03,  1.8963e-02, -2.8486e-02, -3.7874e-02,  3.8743e-02,\n",
            "         -2.7458e-02,  6.6266e-03,  2.3252e-02,  6.1911e-03, -1.0117e-02,\n",
            "          4.6764e-02, -9.2287e-03, -4.6457e-02,  2.0156e-02,  1.1102e-02,\n",
            "         -2.6877e-02,  3.6526e-02, -4.4394e-02,  8.8719e-03,  2.7303e-02,\n",
            "          1.1261e-02,  1.9616e-02, -3.0110e-03, -1.7848e-02,  2.5896e-03,\n",
            "         -2.4783e-03,  2.2321e-03,  3.5471e-02,  1.4686e-02, -1.4461e-02,\n",
            "          3.7167e-02, -1.7868e-02, -2.9340e-02, -4.2197e-02, -4.1637e-02,\n",
            "         -4.2791e-02, -3.1817e-02,  3.9837e-02, -1.2348e-02,  4.4090e-02,\n",
            "         -1.2880e-02, -4.2972e-02, -4.6233e-02, -5.9660e-03,  2.2733e-02,\n",
            "          2.3559e-02,  3.8332e-02,  1.9546e-02, -8.5173e-03,  2.0739e-02,\n",
            "          3.5931e-02, -2.7820e-02, -1.6670e-02, -1.0490e-02, -1.7876e-02,\n",
            "         -4.1342e-02, -4.0787e-02, -1.5214e-03, -3.3221e-02, -3.7995e-02,\n",
            "          3.9523e-03,  3.1709e-02,  2.3838e-02, -2.4367e-03, -3.7975e-02,\n",
            "          3.7990e-02,  2.7413e-02, -4.8559e-03, -8.1437e-03,  4.6428e-02,\n",
            "          4.2606e-03, -4.6295e-02, -2.0714e-02, -8.8502e-03, -1.0201e-02,\n",
            "         -2.4658e-02, -4.6099e-02,  2.0170e-02, -6.5954e-03,  4.5546e-02,\n",
            "          6.2609e-03,  3.5321e-02, -1.9894e-02,  1.0568e-02,  3.2995e-02,\n",
            "          4.6645e-02, -3.3769e-02,  1.1828e-02, -2.0156e-02, -2.3289e-02,\n",
            "          9.4070e-03, -4.2712e-02, -6.7716e-03,  1.7829e-02, -6.5324e-03,\n",
            "         -4.3238e-02, -2.7423e-02,  1.1745e-02,  3.5189e-02,  2.2986e-02,\n",
            "         -7.8685e-03,  2.8780e-02,  2.9487e-03,  1.7104e-02,  4.4427e-02,\n",
            "          3.3935e-02,  2.3515e-03,  1.1240e-02, -2.3876e-02, -4.0842e-02,\n",
            "         -4.3199e-02,  5.5753e-04,  4.6054e-02, -3.3062e-02, -1.7146e-02,\n",
            "          3.6782e-02,  1.4947e-02,  1.0612e-02, -4.0865e-02, -1.5017e-02,\n",
            "          2.7953e-02,  2.0684e-03, -7.9597e-03,  4.0059e-02,  9.1211e-03,\n",
            "         -1.2964e-02, -4.5181e-02,  4.5012e-02, -3.6241e-02,  1.9492e-02,\n",
            "          4.0853e-02,  1.2051e-02,  2.2693e-03,  7.9927e-03, -2.0491e-02,\n",
            "         -1.1800e-02,  1.0175e-02, -3.2121e-03,  2.4384e-02, -2.5849e-02,\n",
            "         -4.5015e-02, -4.5303e-03, -3.9912e-03, -7.6854e-04, -1.1305e-02,\n",
            "          3.1199e-03,  3.3444e-02,  3.1725e-02,  1.0735e-02, -1.0612e-02,\n",
            "         -4.4213e-02, -2.5235e-02, -1.9485e-02, -1.6515e-02,  3.7936e-02,\n",
            "         -3.7551e-02,  7.2866e-03, -2.9224e-03, -7.8635e-03,  3.0736e-02,\n",
            "          1.3471e-02, -4.5437e-02,  2.8748e-02, -3.8336e-02,  3.2545e-02,\n",
            "          1.2290e-02,  4.6726e-02, -3.2918e-02,  2.0964e-03,  3.0796e-02,\n",
            "         -1.0367e-02,  7.1865e-03, -7.1403e-03, -2.5231e-02,  8.8702e-03,\n",
            "          4.6757e-02, -4.1559e-02, -4.2687e-02, -1.1821e-02, -1.3521e-02,\n",
            "         -3.3054e-02,  3.4931e-02, -1.6972e-02, -2.2915e-02, -4.2167e-02,\n",
            "          2.7102e-02, -3.2728e-02,  4.2253e-02, -4.0696e-02,  2.9892e-02,\n",
            "         -2.6837e-02, -1.8809e-02, -1.2849e-02, -1.7182e-02, -9.3730e-03,\n",
            "         -1.8021e-02, -3.7575e-03,  2.3843e-02, -4.3107e-02,  2.7098e-02,\n",
            "         -2.0857e-02,  4.3180e-02,  1.3774e-02,  1.5647e-03,  2.3999e-02,\n",
            "          4.6151e-02,  4.5684e-02, -2.0699e-02, -1.5060e-02,  6.6444e-03,\n",
            "         -3.4815e-02, -3.7341e-02, -2.2742e-02, -3.2053e-02,  3.2069e-02,\n",
            "          2.8526e-02, -3.7511e-02,  3.0550e-02, -4.6118e-02, -1.5662e-02,\n",
            "         -9.2503e-03, -9.6266e-03, -5.9026e-03,  1.9815e-02,  4.2568e-02,\n",
            "         -2.9217e-02, -2.6714e-03,  2.3898e-02, -2.3475e-02, -7.5563e-03,\n",
            "          2.7032e-02, -4.5833e-02,  3.9903e-02, -4.0217e-02,  4.3698e-02,\n",
            "         -1.3829e-02, -3.9448e-02,  1.5365e-03,  7.5610e-03, -1.6525e-03,\n",
            "          2.4376e-02,  2.4064e-02,  1.7624e-02,  6.3116e-03, -1.2006e-02,\n",
            "         -2.7274e-02, -2.9466e-02,  2.6181e-02,  3.4626e-02, -2.4363e-02,\n",
            "          8.9461e-03,  1.3726e-02,  3.6278e-02, -2.7085e-02,  1.2130e-02,\n",
            "         -4.4471e-02,  2.6853e-02,  4.6732e-02, -4.1824e-02, -3.9453e-02,\n",
            "          7.0402e-04,  3.7868e-02, -1.1259e-02,  5.1811e-04,  1.5932e-02,\n",
            "          2.8699e-02, -4.3638e-02,  2.5191e-02, -4.2364e-02,  1.5585e-02,\n",
            "          2.1243e-03,  1.8410e-02,  5.0408e-03,  2.4893e-02, -3.0981e-02,\n",
            "         -4.4623e-02, -8.8658e-03,  3.4858e-02,  1.5573e-04, -2.3775e-02,\n",
            "         -3.2450e-04, -4.3930e-03, -2.9977e-02,  1.3491e-02, -2.4854e-02,\n",
            "          3.5508e-02,  1.6929e-02,  3.4393e-02,  1.1902e-02, -1.3980e-02,\n",
            "         -1.3306e-02,  6.9780e-03, -1.7300e-02,  3.3519e-02,  2.0416e-02,\n",
            "         -2.0136e-02,  3.8429e-02,  4.1002e-03,  9.1963e-03,  1.3279e-02,\n",
            "         -4.1603e-02, -4.2346e-02, -3.7271e-02,  4.0413e-02,  2.7810e-02,\n",
            "         -1.1658e-02, -1.6432e-02,  2.9678e-02, -4.2500e-02,  4.1514e-02,\n",
            "          1.4365e-02, -8.6797e-03,  9.8454e-04, -1.0298e-02, -1.6126e-02,\n",
            "          4.2775e-02,  2.0318e-03, -2.7013e-02, -4.3617e-02,  2.6367e-02,\n",
            "         -4.7691e-03,  1.8237e-02, -4.1745e-02,  1.7467e-02, -4.1923e-02,\n",
            "          3.8899e-02,  3.6595e-02,  4.4098e-02,  1.5017e-02, -4.3788e-02,\n",
            "         -1.3125e-02, -1.7768e-02, -6.7996e-03,  2.3892e-02, -2.1088e-02,\n",
            "          2.2006e-03, -8.4992e-03, -3.2695e-02, -1.4822e-02,  2.7238e-02,\n",
            "          1.9812e-02]], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.bias | Size: torch.Size([512]) | Values : tensor([-0.0433, -0.0129], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.weight | Size: torch.Size([10, 512]) | Values : tensor([[ 0.0211, -0.0234,  0.0028,  ..., -0.0396, -0.0380, -0.0033],\n",
            "        [ 0.0399, -0.0278, -0.0282,  ..., -0.0141,  0.0067, -0.0353]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.bias | Size: torch.Size([10]) | Values : tensor([-0.0216,  0.0143], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(f\"Model structure: {model}\\n\\n\")\n",
        "\n",
        "for name, param in model.named_parameters():\n",
        "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsE7d9kbelO-"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mS_0T_CQelO-"
      },
      "source": [
        "## Further Reading\n",
        "- [torch.nn API](https://pytorch.org/docs/stable/nn.html)\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    },
    "colab": {
      "provenance": []
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}