task-2-nn/MT2022156.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "mHcwaYxAfEMg"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMy6mZMVfEMi"
      },
      "source": [
        "\n",
        "[Learn the Basics](intro.html) ||\n",
        "[Quickstart](quickstart_tutorial.html) ||\n",
        "[Tensors](tensorqs_tutorial.html) ||\n",
        "[Datasets & DataLoaders](data_tutorial.html) ||\n",
        "[Transforms](transforms_tutorial.html) ||\n",
        "**Build Model** ||\n",
        "[Autograd](autogradqs_tutorial.html) ||\n",
        "[Optimization](optimization_tutorial.html) ||\n",
        "[Save & Load Model](saveloadrun_tutorial.html)\n",
        "\n",
        "# Build the Neural Network\n",
        "\n",
        "Neural networks comprise of layers/modules that perform operations on data.\n",
        "The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace provides all the building blocks you need to\n",
        "build your own neural network. Every module in PyTorch subclasses the [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html).\n",
        "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
        "building and managing complex architectures easily.\n",
        "\n",
        "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "rqvI3xO2fEMk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision import datasets, transforms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tCXD7v4AfEMl"
      },
      "source": [
        "## Get Device for Training\n",
        "We want to be able to train our model on a hardware accelerator like the GPU,\n",
        "if it is available. Let's check to see if\n",
        "[torch.cuda](https://pytorch.org/docs/stable/notes/cuda.html) is available, else we\n",
        "continue to use the CPU.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "OoOq6vLGfEMl",
        "outputId": "e737de50-0159-46a5-d923-0c6c127e2afe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n"
          ]
        }
      ],
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(f\"Using {device} device\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AvU1Oa4gfEMm"
      },
      "source": [
        "## Define the Class\n",
        "We define our neural network by subclassing ``nn.Module``, and\n",
        "initialize the neural network layers in ``__init__``. Every ``nn.Module`` subclass implements\n",
        "the operations on input data in the ``forward`` method.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "wazz-PjJfEMm"
      },
      "outputs": [],
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NeuralNetwork, self).__init__()\n",
        "        self.flatten = nn.Flatten()\n",
        "        self.linear_relu_stack = nn.Sequential(\n",
        "            nn.Linear(28*28, 1024),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 10),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.flatten(x)\n",
        "        logits = self.linear_relu_stack(x)\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FtS7dpy9fEMn"
      },
      "source": [
        "We create an instance of ``NeuralNetwork``, and move it to the ``device``, and print\n",
        "its structure.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "XsVqLj89fEMn",
        "outputId": "61fd0086-4b20-45d7-8987-35478fabfcaf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=1024, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=1024, out_features=512, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=512, out_features=512, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "model = NeuralNetwork().to(device)\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fpceIpn_fEMo"
      },
      "source": [
        "To use the model, we pass it the input data. This executes the model's ``forward``,\n",
        "along with some [background operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866).\n",
        "Do not call ``model.forward()`` directly!\n",
        "\n",
        "Calling the model on the input returns a 2-dimensional tensor with dim=0 corresponding to each output of 10 raw predicted values for each class, and dim=1 corresponding to the individual values of each output.  .\n",
        "We get the prediction probabilities by passing it through an instance of the ``nn.Softmax`` module.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "Iml_QvtLfEMo",
        "outputId": "09be0650-bd77-41e9-b37a-9a91d6a774ab",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predicted class: tensor([4], device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "X = torch.rand(1, 28, 28, device=device)\n",
        "logits = model(X)\n",
        "pred_probab = nn.Softmax(dim=1)(logits)\n",
        "y_pred = pred_probab.argmax(1)\n",
        "print(f\"Predicted class: {y_pred}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "alw9QxWtfEMp"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9MGhEQJTfEMp"
      },
      "source": [
        "## Model Layers\n",
        "\n",
        "Let's break down the layers in the FashionMNIST model. To illustrate it, we\n",
        "will take a sample minibatch of 3 images of size 28x28 and see what happens to it as\n",
        "we pass it through the network.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "kNuzDmX2fEMp",
        "outputId": "8b12fbb7-75cd-4532-b23f-75d3bbadd75d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "input_image = torch.rand(3,28,28)\n",
        "print(input_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ak_nSRgNfEMp"
      },
      "source": [
        "### nn.Flatten\n",
        "We initialize the [nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html)\n",
        "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (\n",
        "the minibatch dimension (at dim=0) is maintained).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "0iGes1cwfEMq",
        "outputId": "ebd07491-5582-4542-d4ff-b734b54d25be",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 784])\n"
          ]
        }
      ],
      "source": [
        "flatten = nn.Flatten()\n",
        "flat_image = flatten(input_image)\n",
        "print(flat_image.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQLw5bsvfEMq"
      },
      "source": [
        "### nn.Linear\n",
        "The [linear layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
        "is a module that applies a linear transformation on the input using its stored weights and biases.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "U_y4R0tzfEMq",
        "outputId": "64237e25-5569-4c97-c7c5-8d4ee70d4139",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 20])\n"
          ]
        }
      ],
      "source": [
        "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
        "hidden1 = layer1(flat_image)\n",
        "print(hidden1.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ckfN6PffEMq"
      },
      "source": [
        "### nn.ReLU\n",
        "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
        "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
        "learn a wide variety of phenomena.\n",
        "\n",
        "In this model, we use [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) between our\n",
        "linear layers, but there's other activations to introduce non-linearity in your model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "8PtDJt-VfEMq",
        "outputId": "272e25ff-2b25-41d7-c112-d81ab4f930a6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before ReLU: tensor([[-2.9667e-01,  3.9157e-01, -1.4037e-02,  2.0447e-02,  5.6311e-01,\n",
            "         -4.6899e-02,  1.0410e-01, -4.4897e-01,  5.6446e-01,  1.4229e-01,\n",
            "         -3.3783e-01, -5.3563e-01, -6.4743e-01, -2.3990e-01,  2.3750e-01,\n",
            "         -2.8378e-02, -4.8492e-01,  4.3875e-01, -1.3211e-01, -3.1086e-02],\n",
            "        [-2.1792e-01, -1.5149e-01, -2.9608e-01,  1.7634e-02,  5.0024e-01,\n",
            "         -3.7060e-02, -6.2664e-02, -2.0283e-01,  3.2303e-01, -3.0754e-01,\n",
            "         -1.4656e-01, -6.7547e-01, -1.2283e-01, -4.4760e-01, -1.9783e-01,\n",
            "         -1.1094e-01, -4.9956e-01,  4.7540e-01, -1.3761e-01, -7.9578e-02],\n",
            "        [ 5.5399e-04,  2.7743e-01,  3.7836e-02,  1.3220e-01,  2.4749e-01,\n",
            "         -2.4166e-01,  6.2884e-02, -2.9266e-01,  4.7450e-01,  1.0545e-01,\n",
            "          1.7196e-01, -2.7761e-01, -5.4569e-01, -4.8985e-01,  8.7435e-02,\n",
            "         -3.3536e-01, -4.8525e-01,  6.5346e-01,  1.8820e-01, -2.3579e-01]],\n",
            "       grad_fn=<AddmmBackward0>)\n",
            "\n",
            "\n",
            "After ReLU: tensor([[0.0000e+00, 3.9157e-01, 0.0000e+00, 2.0447e-02, 5.6311e-01, 0.0000e+00,\n",
            "         1.0410e-01, 0.0000e+00, 5.6446e-01, 1.4229e-01, 0.0000e+00, 0.0000e+00,\n",
            "         0.0000e+00, 0.0000e+00, 2.3750e-01, 0.0000e+00, 0.0000e+00, 4.3875e-01,\n",
            "         0.0000e+00, 0.0000e+00],\n",
            "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.7634e-02, 5.0024e-01, 0.0000e+00,\n",
            "         0.0000e+00, 0.0000e+00, 3.2303e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
            "         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 4.7540e-01,\n",
            "         0.0000e+00, 0.0000e+00],\n",
            "        [5.5399e-04, 2.7743e-01, 3.7836e-02, 1.3220e-01, 2.4749e-01, 0.0000e+00,\n",
            "         6.2884e-02, 0.0000e+00, 4.7450e-01, 1.0545e-01, 1.7196e-01, 0.0000e+00,\n",
            "         0.0000e+00, 0.0000e+00, 8.7435e-02, 0.0000e+00, 0.0000e+00, 6.5346e-01,\n",
            "         1.8820e-01, 0.0000e+00]], grad_fn=<ReluBackward0>)\n"
          ]
        }
      ],
      "source": [
        "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
        "hidden1 = nn.ReLU()(hidden1)\n",
        "print(f\"After ReLU: {hidden1}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHiNZhd5fEMr"
      },
      "source": [
        "### nn.Sequential\n",
        "[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) is an ordered\n",
        "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
        "sequential containers to put together a quick network like ``seq_modules``.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "9raqkdFBfEMr"
      },
      "outputs": [],
      "source": [
        "seq_modules = nn.Sequential(\n",
        "    flatten,\n",
        "    layer1,\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(20, 10)\n",
        ")\n",
        "input_image = torch.rand(3,28,28)\n",
        "logits = seq_modules(input_image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-oUNuEKfEMr"
      },
      "source": [
        "### nn.Softmax\n",
        "The last linear layer of the neural network returns `logits` - raw values in [-\\infty, \\infty] - which are passed to the\n",
        "[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) module. The logits are scaled to values\n",
        "[0, 1] representing the model's predicted probabilities for each class. ``dim`` parameter indicates the dimension along\n",
        "which the values must sum to 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "SkEDq1ibfEMr"
      },
      "outputs": [],
      "source": [
        "softmax = nn.Softmax(dim=1)\n",
        "pred_probab = softmax(logits)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MVEafkOWfEMr"
      },
      "source": [
        "## Model Parameters\n",
        "Many layers inside a neural network are *parameterized*, i.e. have associated weights\n",
        "and biases that are optimized during training. Subclassing ``nn.Module`` automatically\n",
        "tracks all fields defined inside your model object, and makes all parameters\n",
        "accessible using your model's ``parameters()`` or ``named_parameters()`` methods.\n",
        "\n",
        "In this example, we iterate over each parameter, and print its size and a preview of its values.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "8JxYBhjpfEMr",
        "outputId": "f9e78e45-a957-49dc-e27e-e8ff0b6f1068",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model structure: NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=784, out_features=1024, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=1024, out_features=512, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=512, out_features=512, bias=True)\n",
            "    (5): ReLU()\n",
            "    (6): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n",
            "\n",
            "\n",
            "Layer: linear_relu_stack.0.weight | Size: torch.Size([1024, 784]) | Values : tensor([[ 0.0227,  0.0179,  0.0030,  ..., -0.0332, -0.0143, -0.0186],\n",
            "        [-0.0298, -0.0342, -0.0350,  ...,  0.0062, -0.0315,  0.0093]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.0.bias | Size: torch.Size([1024]) | Values : tensor([0.0120, 0.0342], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 1024]) | Values : tensor([[ 0.0301, -0.0281, -0.0106,  ..., -0.0021, -0.0198, -0.0040],\n",
            "        [-0.0061,  0.0052, -0.0017,  ..., -0.0112,  0.0009, -0.0286]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([0.0073, 0.0209], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.weight | Size: torch.Size([512, 512]) | Values : tensor([[ 0.0206,  0.0440,  0.0218,  ..., -0.0369,  0.0351, -0.0269],\n",
            "        [ 0.0015, -0.0108, -0.0062,  ..., -0.0418,  0.0185, -0.0080]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.4.bias | Size: torch.Size([512]) | Values : tensor([0.0216, 0.0408], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.weight | Size: torch.Size([10, 512]) | Values : tensor([[-0.0433, -0.0270, -0.0413,  ...,  0.0433, -0.0056,  0.0404],\n",
            "        [ 0.0132, -0.0382,  0.0273,  ...,  0.0224, -0.0194,  0.0152]],\n",
            "       device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n",
            "Layer: linear_relu_stack.6.bias | Size: torch.Size([10]) | Values : tensor([ 0.0245, -0.0073], device='cuda:0', grad_fn=<SliceBackward0>) \n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(f\"Model structure: {model}\\n\\n\")\n",
        "\n",
        "for name, param in model.named_parameters():\n",
        "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EbaMkQ_OfEMs"
      },
      "source": [
        "--------------\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h3NGIhKMfEMs"
      },
      "source": [
        "## Further Reading\n",
        "- [torch.nn API](https://pytorch.org/docs/stable/nn.html)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "sZMujC2Jf09o"
      },
      "execution_count": 27,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    },
    "colab": {
      "provenance": []
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}