task-2-nn/IMT2020090.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "py3U6WiLe0Ay"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOvBIspje0A0"
      },
      "source": [
        "\n",
        "[Learn the Basics](intro.html) ||\n",
        "[Quickstart](quickstart_tutorial.html) ||\n",
        "[Tensors](tensorqs_tutorial.html) ||\n",
        "[Datasets & DataLoaders](data_tutorial.html) ||\n",
        "[Transforms](transforms_tutorial.html) ||\n",
        "**Build Model** ||\n",
        "[Autograd](autogradqs_tutorial.html) ||\n",
        "[Optimization](optimization_tutorial.html) ||\n",
        "[Save & Load Model](saveloadrun_tutorial.html)\n",
        "\n",
        "# Build the Neural Network\n",
        "\n",
        "Neural networks comprise of layers/modules that perform operations on data.\n",
        "The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace provides all the building blocks you need to\n",
        "build your own neural network. Every module in PyTorch subclasses the [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html).\n",
        "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
        "building and managing complex architectures easily.\n",
        "\n",
        "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "Md1DWdyEe0A1"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision import datasets, transforms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xh_PYTPTe0A1"
      },
      "source": [
        "## Get Device for Training\n",
        "We want to be able to train our model on a hardware accelerator like the GPU,\n",
        "if it is available. Let's check to see if\n",
        "[torch.cuda](https://pytorch.org/docs/stable/notes/cuda.html) is available, else we\n",
        "continue to use the CPU.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "mZpNCjQge0A1",
        "outputId": "58ca0aea-7c5d-4386-bad6-03e76bee6a62",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n"
          ]
        }
      ],
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(f\"Using {device} device\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DeJ0WEGhe0A1"
      },
      "source": [
        "## Define the Class\n",
        "We define our neural network by subclassing ``nn.Module``, and\n",
        "initialize the neural network layers in ``__init__``. Every ``nn.Module`` subclass implements\n",
        "the operations on input data in the ``forward`` method.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "HlLMmQnde0A2"
      },
      "outputs": [],
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NeuralNetwork, self).__init__()\n",
        "        self.flatten = nn.Flatten()\n",
        "        self.linear_relu_stack = nn.Sequential(\n",
        "            nn.Linear(28*28, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 328),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(328,512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 10),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.flatten(x)\n",
        "        logits = self.linear_relu_stack(x)\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05rSkg7Te0A2"
      },
      "source": [
        "We create an instance of ``NeuralNetwork``, and move it to the ``device``, and print\n",
        "its structure.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "sc0wtQfqe0A2",
        "outputId": "68d9c32b-5d61-4867-84f1-57ff686dee74",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=328, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=328, out_features=512, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "model = NeuralNetwork().to(device)\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0d9bPCXe0A3"
      },
      "source": [
        "To use the model, we pass it the input data. This executes the model's ``forward``,\n",
        "along with some [background operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866).\n",
        "Do not call ``model.forward()`` directly!\n",
        "\n",
        "Calling the model on the input returns a 2-dimensional tensor with dim=0 corresponding to each output of 10 raw predicted values for each class, and dim=1 corresponding to the individual values of each output.  .\n",
        "We get the prediction probabilities by passing it through an instance of the ``nn.Softmax`` module.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "6AuF9QtCe0A3",
        "outputId": "3cea1a42-95b3-4728-ea76-3215e5b0fa65",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predicted class: tensor([1], device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "X = torch.rand(1, 28, 28, device=device)\n",
        "logits = model(X)\n",
        "pred_probab = nn.Softmax(dim=1)(logits)\n",
        "y_pred = pred_probab.argmax(1)\n",
        "print(f\"Predicted class: {y_pred}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L2PQYYhie0A3"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GBZmOEufe0A3"
      },
      "source": [
        "## Model Layers\n",
        "\n",
        "Let's break down the layers in the FashionMNIST model. To illustrate it, we\n",
        "will take a sample minibatch of 3 images of size 28x28 and see what happens to it as\n",
        "we pass it through the network.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "7zDw5Dhwe0A3",
        "outputId": "1fd38827-7fde-4ad0-de21-828adf3862bb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "input_image = torch.rand(3,28,28)\n",
        "print(input_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1RoxTmxke0A3"
      },
      "source": [
        "### nn.Flatten\n",
        "We initialize the [nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html)\n",
        "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (\n",
        "the minibatch dimension (at dim=0) is maintained).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "ugeAjSW1e0A3",
        "outputId": "610dd90d-e005-400d-e0e4-ba7e5892ed79",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 784])\n"
          ]
        }
      ],
      "source": [
        "flatten = nn.Flatten()\n",
        "flat_image = flatten(input_image)\n",
        "print(flat_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zJ9mP5zce0A4"
      },
      "source": [
        "### nn.Linear\n",
        "The [linear layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
        "is a module that applies a linear transformation on the input using its stored weights and biases.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "gPO-E6xme0A4",
        "outputId": "558fd81a-7a79-41f8-9d9e-fbc25832f654",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 20])\n"
          ]
        }
      ],
      "source": [
        "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
        "hidden1 = layer1(flat_image)\n",
        "print(hidden1.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3FApa7ATe0A4"
      },
      "source": [
        "### nn.ReLU\n",
        "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
        "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
        "learn a wide variety of phenomena.\n",
        "\n",
        "In this model, we use [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) between our\n",
        "linear layers, but there's other activations to introduce non-linearity in your model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "4aiJGti3e0A4",
        "outputId": "f2dbd344-0901-4c41-deee-ea6d7a554016",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before ReLU: tensor([[ 0.0144,  0.5025,  0.4261,  0.8007, -0.1248, -0.1734, -0.4778,  0.3211,\n",
            "         -0.1869,  0.2313, -0.3346, -0.2644, -0.1396, -0.2977,  0.2810, -0.2537,\n",
            "          0.2924,  0.2133,  1.2097, -0.0687],\n",
            "        [ 0.3094,  0.4958,  0.3402,  0.6841,  0.2537, -0.0157, -0.5496,  0.2468,\n",
            "          0.1157,  0.2707, -0.1238, -0.2600, -0.3158, -0.4072,  0.0023, -0.2833,\n",
            "          0.2081, -0.0235,  1.0964,  0.1983],\n",
            "        [ 0.2070,  0.1957,  0.3346,  0.8225, -0.0567, -0.0859, -0.6062, -0.1457,\n",
            "         -0.1225,  0.5335, -0.5357, -0.1959, -0.2753, -0.1844, -0.1139, -0.1074,\n",
            "          0.5070,  0.3089,  0.9452, -0.0221]], grad_fn=<AddmmBackward0>)\n",
            "\n",
            "\n",
            "After ReLU: tensor([[0.0144, 0.5025, 0.4261, 0.8007, 0.0000, 0.0000, 0.0000, 0.3211, 0.0000,\n",
            "         0.2313, 0.0000, 0.0000, 0.0000, 0.0000, 0.2810, 0.0000, 0.2924, 0.2133,\n",
            "         1.2097, 0.0000],\n",
            "        [0.3094, 0.4958, 0.3402, 0.6841, 0.2537, 0.0000, 0.0000, 0.2468, 0.1157,\n",
            "         0.2707, 0.0000, 0.0000, 0.0000, 0.0000, 0.0023, 0.0000, 0.2081, 0.0000,\n",
            "         1.0964, 0.1983],\n",
            "        [0.2070, 0.1957, 0.3346, 0.8225, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
            "         0.5335, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5070, 0.3089,\n",
            "         0.9452, 0.0000]], grad_fn=<ReluBackward0>)\n"
          ]
        }
      ],
      "source": [
        "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
        "hidden1 = nn.ReLU()(hidden1)\n",
        "print(f\"After ReLU: {hidden1}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJyZUD_0e0A4"
      },
      "source": [
        "### nn.Sequential\n",
        "[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) is an ordered\n",
        "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
        "sequential containers to put together a quick network like ``seq_modules``.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "RrSQmIO6e0A4"
      },
      "outputs": [],
      "source": [
        "seq_modules = nn.Sequential(\n",
        "    flatten,\n",
        "    layer1,\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(20, 10)\n",
        ")\n",
        "input_image = torch.rand(3,28,28)\n",
        "logits = seq_modules(input_image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fb4oXaT8e0A4"
      },
      "source": [
        "### nn.Softmax\n",
        "The last linear layer of the neural network returns `logits` - raw values in [-\\infty, \\infty] - which are passed to the\n",
        "[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) module. The logits are scaled to values\n",
        "[0, 1] representing the model's predicted probabilities for each class. ``dim`` parameter indicates the dimension along\n",
        "which the values must sum to 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "zH85Ct_5e0A4"
      },
      "outputs": [],
      "source": [
        "softmax = nn.Softmax(dim=1)\n",
        "pred_probab = softmax(logits)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Brjl5Cye0A4"
      },
      "source": [
        "## Model Parameters\n",
        "Many layers inside a neural network are *parameterized*, i.e. have associated weights\n",
        "and biases that are optimized during training. Subclassing ``nn.Module`` automatically\n",
        "tracks all fields defined inside your model object, and makes all parameters\n",
        "accessible using your model's ``parameters()`` or ``named_parameters()`` methods.\n",
        "\n",
        "In this example, we iterate over each parameter, and print its size and a preview of its values.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "Fld_gTfke0A4",
        "outputId": "61f16aec-e516-455c-da0a-cf03f69c59c4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model structure: NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=512, out_features=328, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=328, out_features=512, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n",
            "\n",
            "\n",
            "Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[ 0.0018, -0.0151,  0.0170,  ..., -0.0318, -0.0170,  0.0318],\n",
            "        [ 0.0130, -0.0053,  0.0206,  ...,  0.0128,  0.0014, -0.0049]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([0.0222, 0.0338], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.weight | Size: torch.Size([328, 512]) | Values : tensor([[-0.0088,  0.0366, -0.0160,  ..., -0.0218,  0.0360,  0.0018],\n",
            "        [ 0.0173, -0.0063, -0.0272,  ..., -0.0158, -0.0073, -0.0183]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.bias | Size: torch.Size([328]) | Values : tensor([-0.0435,  0.0331], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.weight | Size: torch.Size([512, 328]) | Values : tensor([[-1.5528e-02,  3.0380e-02,  3.4466e-02, -4.5190e-02,  3.4375e-02,\n",
            "          2.7490e-02, -3.2379e-02,  3.7437e-02, -5.3829e-02, -1.6282e-02,\n",
            "          1.2440e-02,  4.3877e-02, -2.7275e-02, -1.6009e-02, -4.2030e-02,\n",
            "          4.9913e-02,  4.7921e-02, -2.4614e-02,  4.4835e-02,  2.1988e-02,\n",
            "         -9.7273e-03,  2.4417e-02, -2.5661e-02, -1.9406e-02,  4.0596e-02,\n",
            "         -5.0903e-02,  4.2927e-02, -9.4775e-03,  3.1369e-02,  4.9691e-02,\n",
            "          2.2853e-02,  3.0505e-02, -3.1711e-02, -4.9422e-02,  4.4826e-02,\n",
            "          2.3537e-02, -9.8755e-03,  4.4258e-02, -2.4474e-02, -4.1356e-02,\n",
            "          8.6422e-04, -1.8820e-02,  2.4790e-02,  8.2716e-03, -4.7774e-03,\n",
            "          1.5373e-02,  8.6829e-03, -4.5948e-02, -5.2437e-02, -5.2980e-02,\n",
            "          5.3084e-02, -3.0956e-02, -2.4239e-02, -3.7588e-02, -4.1978e-03,\n",
            "          5.1794e-02,  4.2074e-02, -3.8554e-02, -2.3503e-02,  2.4217e-02,\n",
            "          3.4396e-02,  6.3587e-03,  1.6955e-02, -1.3296e-02,  2.3092e-02,\n",
            "          7.7736e-03,  3.9754e-02, -2.7098e-04, -3.4218e-02,  3.0595e-02,\n",
            "          3.7244e-02,  4.6127e-02,  7.3635e-03, -3.9042e-02, -7.8926e-03,\n",
            "         -1.9951e-02,  5.1433e-02, -3.8586e-02, -1.6685e-02, -6.8704e-03,\n",
            "          5.4049e-02,  1.9194e-02,  1.0348e-02,  1.1242e-03,  2.1362e-02,\n",
            "          3.4883e-02,  1.0309e-02,  4.8014e-03,  7.4134e-03, -4.6095e-02,\n",
            "         -3.5690e-02, -2.3892e-02,  1.1999e-02,  2.2032e-02,  4.3687e-02,\n",
            "         -4.0451e-02,  1.3616e-02, -2.3807e-03, -1.3783e-02, -5.5115e-02,\n",
            "          5.0302e-02, -1.7538e-02,  1.0137e-02, -1.9943e-03, -1.0118e-02,\n",
            "          2.2246e-02, -5.3693e-02,  2.5325e-02, -2.8162e-02, -5.8955e-03,\n",
            "          2.4096e-02, -4.3462e-02, -4.6426e-02, -1.0759e-02,  2.5614e-02,\n",
            "          1.7227e-02,  2.8377e-02,  5.4931e-02, -9.7800e-03, -2.9915e-03,\n",
            "          3.3525e-02,  1.5143e-03,  3.9883e-02,  4.1934e-02,  1.1993e-02,\n",
            "          5.5763e-03, -1.4180e-02,  6.3613e-03, -1.5512e-02, -4.9652e-02,\n",
            "         -2.5481e-02, -1.9179e-02,  3.4557e-02, -2.4629e-02,  4.4004e-02,\n",
            "         -2.0748e-02,  4.5622e-02,  5.3504e-02, -1.4524e-02,  3.7180e-02,\n",
            "          2.5364e-03, -4.1921e-02,  1.8813e-02, -4.2243e-02, -1.8482e-02,\n",
            "          8.8963e-03, -3.3389e-02, -2.6929e-02,  4.4067e-02,  1.0342e-02,\n",
            "          2.7430e-02,  6.2077e-03, -4.5045e-02,  2.9552e-02, -4.0780e-02,\n",
            "          4.8606e-02,  1.5299e-02, -3.2560e-02,  2.2678e-02,  5.3187e-02,\n",
            "         -3.8789e-02, -1.1189e-02,  3.2083e-02, -2.1031e-02, -3.4464e-02,\n",
            "          4.0610e-02,  4.6866e-02,  4.6395e-02, -2.4095e-02,  1.4585e-02,\n",
            "         -3.3530e-02,  1.3874e-03, -4.2433e-02,  2.5940e-02,  1.6205e-02,\n",
            "         -3.2208e-02,  3.5971e-02, -2.5797e-03, -5.6989e-03, -1.5639e-02,\n",
            "         -4.6091e-02, -5.1088e-02, -5.1604e-02, -2.9336e-02, -2.9180e-02,\n",
            "          3.8647e-02, -7.3777e-03, -3.7514e-02,  4.1736e-02,  3.6458e-02,\n",
            "         -7.2488e-03,  1.0184e-02,  4.2716e-02, -3.5700e-02, -3.3398e-02,\n",
            "         -1.8736e-02, -3.5317e-02,  8.7934e-03,  4.2445e-03, -3.5709e-02,\n",
            "         -2.1199e-02,  3.1282e-02, -5.7546e-03,  3.1429e-03, -3.2610e-02,\n",
            "         -3.4485e-02, -1.6874e-02,  5.5064e-02,  2.0667e-02,  1.4406e-02,\n",
            "          1.3635e-02, -3.7983e-02,  1.4205e-02,  3.7731e-02, -2.5138e-02,\n",
            "         -5.4421e-02, -3.1024e-02, -3.9254e-02, -4.5479e-02, -1.8397e-02,\n",
            "          4.1894e-03,  4.9697e-02,  2.7912e-02,  3.3202e-02,  4.4268e-02,\n",
            "         -5.3072e-02, -4.3865e-02, -1.0941e-02, -4.1102e-02,  2.5572e-02,\n",
            "          9.6342e-03,  1.9683e-02,  2.1648e-02, -4.8772e-02,  2.7798e-02,\n",
            "         -4.7006e-02, -5.2461e-02,  3.8695e-02,  3.5260e-02, -1.5486e-03,\n",
            "         -2.2732e-02,  2.7353e-02,  1.0148e-02, -4.2073e-02,  4.5484e-02,\n",
            "         -2.2400e-02, -2.5343e-02,  4.7308e-02, -4.4183e-02, -2.7622e-02,\n",
            "          3.6570e-02,  3.5573e-02, -2.9875e-02, -5.0843e-02, -1.3662e-02,\n",
            "         -3.2882e-02, -5.2189e-02,  1.3060e-02, -1.9897e-02, -2.3755e-02,\n",
            "          1.1953e-02,  3.0125e-02, -4.6758e-02,  3.8450e-02, -3.4591e-02,\n",
            "         -8.2008e-05, -4.5251e-02,  2.2123e-02,  3.4871e-02,  1.9386e-02,\n",
            "          4.4610e-02, -3.7173e-02,  5.1238e-02, -4.2678e-02,  5.0570e-02,\n",
            "         -2.0139e-02, -3.7092e-02,  3.6710e-02, -2.3385e-02, -4.3045e-02,\n",
            "         -5.1698e-02, -6.8086e-03,  3.4555e-02,  4.6524e-02, -4.9624e-02,\n",
            "         -3.4952e-02,  5.4709e-02,  4.5659e-02, -1.3075e-02, -2.2014e-02,\n",
            "          2.7983e-02, -3.4055e-02,  1.0926e-02, -4.9069e-02, -2.6066e-02,\n",
            "          2.0546e-02, -4.3937e-02,  4.6133e-02,  4.6368e-02, -5.3588e-02,\n",
            "         -5.6872e-03,  4.2025e-02,  3.8910e-02,  3.1299e-02, -2.3015e-02,\n",
            "          1.7711e-02,  2.7847e-02, -3.8897e-02,  3.5340e-02, -3.2031e-02,\n",
            "          3.1873e-04, -7.9250e-03,  1.2047e-04,  5.4840e-02, -1.9571e-02,\n",
            "          4.8260e-02,  5.0555e-02, -2.7979e-02, -1.4201e-02,  6.9027e-03,\n",
            "          4.3228e-02, -4.5624e-02, -3.9411e-03,  3.1912e-03, -4.6694e-03,\n",
            "         -2.6699e-02, -5.3165e-02, -2.6702e-02],\n",
            "        [-4.6104e-02,  1.5962e-02,  4.0668e-02,  4.1793e-02, -5.1232e-02,\n",
            "          8.3213e-03, -9.4805e-03, -5.4136e-02, -2.8021e-02,  3.1557e-02,\n",
            "         -4.6069e-02,  2.9824e-03,  7.1309e-03,  5.2133e-02,  4.7050e-02,\n",
            "         -4.8981e-02, -2.1045e-02, -2.2643e-02,  4.1176e-02,  4.8556e-02,\n",
            "          5.3361e-03, -4.5910e-02,  9.5148e-03, -6.3092e-03,  6.7221e-03,\n",
            "         -2.1300e-02,  2.9074e-02, -4.0272e-02,  4.7055e-02,  3.8080e-02,\n",
            "         -5.0074e-02, -2.9177e-02,  1.8356e-02, -1.7439e-02, -4.8586e-02,\n",
            "          1.3695e-02, -3.6849e-02,  2.2827e-02,  2.6627e-02,  3.7073e-02,\n",
            "          1.9802e-02,  4.2455e-02, -2.4997e-02,  5.0144e-02,  4.0409e-02,\n",
            "          1.4338e-02, -3.9349e-02, -1.0307e-02,  2.9863e-02,  4.5674e-02,\n",
            "         -2.1521e-02,  2.1020e-02, -1.8492e-03, -2.2010e-02, -1.4552e-02,\n",
            "         -5.4008e-02, -3.4212e-02, -2.4991e-02, -3.3890e-02, -2.5532e-02,\n",
            "          3.2167e-02, -2.9697e-02,  9.4234e-03,  2.0449e-03,  1.8144e-02,\n",
            "         -4.9713e-02,  3.8036e-03, -3.2887e-02, -3.8270e-02, -2.7078e-02,\n",
            "         -2.8058e-02, -4.8150e-02, -1.6017e-02,  2.6499e-02, -4.3192e-02,\n",
            "         -4.3852e-02, -5.3928e-02, -7.2176e-03, -4.2652e-02,  1.0063e-02,\n",
            "         -7.4009e-04,  4.9116e-02,  2.2914e-02,  7.7171e-03,  4.4622e-03,\n",
            "         -6.4518e-03, -3.5438e-02, -2.8966e-02, -3.7355e-02,  1.2227e-02,\n",
            "          5.9363e-03,  3.7511e-02, -2.2562e-03,  1.6104e-02,  1.5158e-02,\n",
            "          2.2914e-02, -3.3680e-02,  5.9854e-04, -4.8240e-02, -2.2647e-02,\n",
            "         -2.9497e-02,  4.2015e-03,  2.0005e-02, -4.3499e-02, -4.6525e-02,\n",
            "          9.2852e-03,  3.8453e-02, -2.2026e-02,  1.8177e-02,  1.5954e-03,\n",
            "          3.1901e-02, -3.6279e-02, -5.5164e-02, -2.9943e-02, -5.0340e-02,\n",
            "         -3.8772e-02,  4.5400e-02,  2.4703e-02, -9.4003e-03, -4.9762e-02,\n",
            "         -4.9367e-02,  1.2424e-02, -3.1250e-02, -3.1573e-02,  2.0158e-02,\n",
            "          1.7757e-02,  2.6357e-02, -4.9972e-02,  1.0596e-02,  1.3259e-02,\n",
            "          3.4534e-02,  2.8884e-02, -9.6472e-03, -2.7469e-02,  1.7781e-02,\n",
            "         -1.3504e-02,  2.3235e-02,  1.5248e-02,  3.5716e-02,  3.6291e-02,\n",
            "          7.4492e-03,  4.6729e-02, -2.4389e-03, -2.2337e-02, -4.3802e-02,\n",
            "          2.1021e-02, -4.8130e-02,  2.4677e-02,  6.6684e-03, -3.3150e-02,\n",
            "         -2.8454e-02, -3.7048e-02, -4.3205e-02,  1.0985e-02, -8.4028e-03,\n",
            "         -4.8356e-02,  2.6802e-02,  5.5200e-02, -3.1487e-02, -9.3858e-03,\n",
            "         -1.1266e-02, -3.2312e-02,  5.0830e-02, -5.3248e-02,  5.4172e-02,\n",
            "         -1.1360e-02,  1.5379e-02,  4.8405e-04,  3.1761e-02, -6.4832e-04,\n",
            "         -5.7080e-03,  2.1903e-02, -2.3799e-02, -2.5012e-02, -5.0735e-02,\n",
            "          1.5755e-02, -1.6141e-02, -5.2820e-02,  2.5329e-02, -9.9811e-03,\n",
            "         -5.1919e-02, -2.7700e-02,  1.8922e-02, -2.8528e-02,  1.4619e-03,\n",
            "         -6.9283e-03,  2.1424e-03,  6.9466e-04, -1.9924e-02, -3.8953e-02,\n",
            "         -2.4613e-02,  3.1477e-03,  4.4681e-02, -2.6349e-02,  5.0482e-02,\n",
            "          4.4631e-02, -2.4964e-02, -3.1163e-02, -1.4913e-02,  2.3651e-02,\n",
            "          2.1696e-02, -9.9053e-03,  2.9669e-02, -5.4855e-02,  1.9037e-02,\n",
            "          3.2105e-02,  4.7193e-02, -2.8311e-03, -3.4736e-02, -3.7421e-02,\n",
            "          4.1179e-02, -1.9121e-03, -2.6213e-02,  2.2060e-02,  4.7202e-02,\n",
            "         -7.2074e-03, -1.0322e-02,  2.2486e-02,  1.0652e-03,  5.0959e-02,\n",
            "          1.3034e-04, -3.0868e-02,  8.7086e-03, -4.2950e-02,  2.8768e-03,\n",
            "         -2.0218e-02, -5.3100e-02,  2.5041e-02,  3.3138e-02,  4.3425e-02,\n",
            "          1.5626e-02, -1.6479e-04, -2.1552e-02, -5.4044e-02, -8.9198e-03,\n",
            "         -3.8307e-02, -2.7416e-02, -1.7910e-02,  2.4444e-02,  2.7103e-02,\n",
            "          6.9414e-04,  3.0311e-02,  1.0918e-02, -4.1378e-02,  6.2865e-03,\n",
            "         -5.1390e-02, -4.8344e-02,  8.5050e-03,  1.2165e-02,  3.0260e-02,\n",
            "          2.5337e-02,  5.7589e-04, -2.3570e-02, -1.9374e-02,  8.3965e-03,\n",
            "          5.2133e-02,  4.1604e-02, -1.6214e-02,  3.5804e-02, -2.5324e-02,\n",
            "          1.1327e-02,  1.4210e-02, -5.3164e-02,  2.1071e-03,  3.0312e-02,\n",
            "          4.7746e-02,  1.8826e-02,  3.8312e-02,  1.1532e-02, -7.2146e-03,\n",
            "          4.2910e-02,  4.6961e-02,  4.0926e-02, -6.1875e-03,  3.9744e-02,\n",
            "         -5.0771e-02,  1.5040e-02,  1.4322e-03,  3.1684e-02,  4.7694e-02,\n",
            "         -1.2734e-02, -2.5526e-02, -3.1485e-02, -7.9233e-03, -4.8467e-02,\n",
            "         -2.1004e-02,  3.9418e-02,  3.9045e-02, -8.7103e-03,  9.3730e-03,\n",
            "         -5.1709e-02, -4.4018e-02, -1.0074e-02,  2.3512e-04,  2.7250e-02,\n",
            "         -3.6664e-02, -1.7859e-02,  3.3656e-02, -2.0607e-02,  5.4245e-02,\n",
            "          3.0554e-02,  2.9313e-03, -3.3242e-02, -4.4234e-02, -4.6126e-02,\n",
            "         -3.8763e-02,  1.7408e-02, -3.5039e-02,  1.9103e-02,  4.3617e-02,\n",
            "         -4.6651e-02,  3.2195e-03, -6.9419e-03, -3.3197e-02, -4.2461e-02,\n",
            "          5.1398e-02,  4.1181e-02,  5.0172e-02, -2.3278e-02,  2.6065e-02,\n",
            "          2.6749e-02, -4.2466e-03,  4.8077e-02, -6.7241e-03,  8.0650e-03,\n",
            "          1.2397e-02,  2.8180e-02,  1.8803e-02]], device='cuda:0',\n",
            "       grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.bias | Size: torch.Size([512]) | Values : tensor([0.0469, 0.0400], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.weight | Size: torch.Size([10, 512]) | Values : tensor([[-0.0049, -0.0256,  0.0138,  ...,  0.0418, -0.0080, -0.0132],\n",
            "        [-0.0127, -0.0082,  0.0202,  ..., -0.0004,  0.0168, -0.0088]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.bias | Size: torch.Size([10]) | Values : tensor([0.0115, 0.0437], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(f\"Model structure: {model}\\n\\n\")\n",
        "\n",
        "for name, param in model.named_parameters():\n",
        "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3dJJO1KVe0A4"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f85Qoovbe0A4"
      },
      "source": [
        "## Further Reading\n",
        "- [torch.nn API](https://pytorch.org/docs/stable/nn.html)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ecuAkTCRjoz4"
      },
      "execution_count": 36,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    },
    "colab": {
      "provenance": []
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}