Cartpole_StepsEpisodes.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Week 2 - AI Lab\n",
        "\n",
        "Author: Khushee Kapoor\n",
        "\n",
        "Registration Number: 200968052"
      ],
      "metadata": {
        "id": "TvtljaFb5ZuT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To start, we set up the tensorflow gym suite in the jupyter environment. We also import the required libraries for creating an environment. In TF-Agents, environments can be implemented either in Python or TensorFlow. Python environments are usually easier to implement, understand, and debug, but TensorFlow environments are more efficient and allow natural parallelization. The most common workflow is to implement an environment in Python and use one of the wrappers to automatically convert it into TensorFlow."
      ],
      "metadata": {
        "id": "_Wnv5enN5fEY"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZH-EDFSTGzo4"
      },
      "outputs": [],
      "source": [
        "# setting up the tensorflow gym suite\n",
        "!pip install tf-agents[reverb]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# importing the required libraries\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "import abc\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "from tf_agents.environments import py_environment\n",
        "from tf_agents.environments import tf_environment\n",
        "from tf_agents.environments import tf_py_environment\n",
        "from tf_agents.environments import utils\n",
        "from tf_agents.specs import array_spec\n",
        "from tf_agents.environments import wrappers\n",
        "from tf_agents.environments import suite_gym\n",
        "from tf_agents.trajectories import time_step as ts"
      ],
      "metadata": {
        "id": "4gnvi7mbIR7b"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Cartpole Game\n",
        "\n",
        "CartPole problem is the problem of balancing the CartPole. CartPole is the structure where a pole is attached to the cart and the cart is free to slide across the frictionless surface. By sliding the cart left or right, the CartPole is balanced.\n",
        "\n",
        "### Objective\n",
        "\n",
        "The objective of the CartPole is to keep it from falling or moving out of the range. Therefore, the failure conditions are:\n",
        "- Magnitude of the angle of the pole with respect to the vertical exceeding some threshold.\n",
        "- Distance of the CartPole from the center exceeding some threshold.\n",
        "\n",
        "### Steps vs Episodes\n",
        "\n",
        "- An episode is an instance of a game (or life of a game). If the game ends or life decreases, the episode ends.\n",
        "-  A step is the time or some discrete value which increases monotonically in an episode. With each change in the state of the game, the value of step increases until the game ends.\n",
        "\n",
        "For each instance of CartPole not toppling down or going out of range, we have a reward of +1.0."
      ],
      "metadata": {
        "id": "ZgLVML1w6Sfz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Implementing the Cartpole environment for a certain number of steps.\n",
        "\n",
        "To implement cartpole environment for a certain number of steps, we iterate over the steps, and for every step, we identify and perform an action. In correspondence to our action, we get an appropriate reward."
      ],
      "metadata": {
        "id": "TnN6To6V8sVZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# loading the cartpole environment\n",
        "env = suite_gym.load('CartPole-v0')\n",
        "\n",
        "# wrapping the python environment into a tensorflow environment\n",
        "tf_env = tf_py_environment.TFPyEnvironment(env)\n",
        "\n",
        "time_step = tf_env.reset()\n",
        "num_steps = 26\n",
        "transitions = []\n",
        "reward = []\n",
        "\n",
        "# iterating over the steps\n",
        "for i in range(num_steps):\n",
        "  action = tf.constant([i % 2])\n",
        "  # applies the action and returns the new TimeStep\n",
        "  next_time_step = tf_env.step(action)\n",
        "  transitions.append([time_step, action, next_time_step])\n",
        "  reward.append(next_time_step.reward)\n",
        "  time_step = next_time_step\n",
        "\n",
        "np_transitions = tf.nest.map_structure(lambda x: x.numpy(), transitions)\n",
        "print('\\n'.join(map(str, np_transitions)))\n",
        "print('Total reward:', np.sum(reward))"
      ],
      "metadata": {
        "id": "fqtOdsmyI5Pr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5a00677e-437a-4632-f13e-42d8b81c0ec5"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01512762,  0.01521305,  0.01495875, -0.00404121]],\n",
            "      dtype=float32),\n",
            " 'reward': array([0.], dtype=float32),\n",
            " 'step_type': array([0], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01482336, -0.1801202 ,  0.01487792,  0.2933236 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01482336, -0.1801202 ,  0.01487792,  0.2933236 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01842576,  0.0147865 ,  0.0207444 ,  0.00536984]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01842576,  0.0147865 ,  0.0207444 ,  0.00536984]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01813003, -0.18062672,  0.02085179,  0.30452502]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.01813003, -0.18062672,  0.02085179,  0.30452502]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02174257,  0.01419196,  0.02694229,  0.01849051]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02174257,  0.01419196,  0.02694229,  0.01849051]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02145872, -0.1813058 ,  0.0273121 ,  0.3195508 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02145872, -0.1813058 ,  0.0273121 ,  0.3195508 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02508484,  0.01341674,  0.03370312,  0.03560468]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02508484,  0.01341674,  0.03370312,  0.03560468]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02481651, -0.1821719 ,  0.03441522,  0.3387279 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02481651, -0.1821719 ,  0.03441522,  0.3387279 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02845994,  0.01244387,  0.04118977,  0.05709316]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02845994,  0.01244387,  0.04118977,  0.05709316]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02821107, -0.18324374,  0.04233164,  0.36248195]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.02821107, -0.18324374,  0.04233164,  0.36248195]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03187594,  0.01125179,  0.04958127,  0.08344182]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03187594,  0.01125179,  0.04958127,  0.08344182]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0316509 , -0.18454453,  0.05125011,  0.3913463 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0316509 , -0.18454453,  0.05125011,  0.3913463 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0353418 ,  0.00981403,  0.05907704,  0.11525219]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0353418 ,  0.00981403,  0.05907704,  0.11525219]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03514552, -0.18610246,  0.06138208,  0.4259728 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03514552, -0.18610246,  0.06138208,  0.4259728 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03886757,  0.00809877,  0.06990153,  0.15325455]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03886757,  0.00809877,  0.06990153,  0.15325455]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03870559, -0.18795082,  0.07296663,  0.46714512]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.03870559, -0.18795082,  0.07296663,  0.46714512]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04246461,  0.00606846,  0.08230953,  0.19832373]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04246461,  0.00606846,  0.08230953,  0.19832373]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04234324, -0.19012842,  0.086276  ,  0.5157944 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04234324, -0.19012842,  0.086276  ,  0.5157944 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0461458 ,  0.00367942,  0.09659189,  0.25149763]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.0461458 ,  0.00367942,  0.09659189,  0.25149763]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04607222, -0.19267961,  0.10162184,  0.5730171 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04607222, -0.19267961,  0.10162184,  0.5730171 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04992581,  0.00088168,  0.11308219,  0.31399846]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04992581,  0.00088168,  0.11308219,  0.31399846]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04990818, -0.19565429,  0.11936215,  0.6400949 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.04990818, -0.19565429,  0.11936215,  0.6400949 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05382126, -0.00238088,  0.13216405,  0.3872567 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05382126, -0.00238088,  0.13216405,  0.3872567 ]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05386888, -0.199107  ,  0.1399092 ,  0.71851563]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05386888, -0.199107  ,  0.1399092 ,  0.71851563]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05785102, -0.00616978,  0.1542795 ,  0.47293693]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05785102, -0.00616978,  0.1542795 ,  0.47293693]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([0], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05797442, -0.20309559,  0.16373824,  0.80999565]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "[TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.05797442, -0.20309559,  0.16373824,  0.80999565]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)}), array([1], dtype=int32), TimeStep(\n",
            "{'discount': array([1.], dtype=float32),\n",
            " 'observation': array([[-0.06203633, -0.01055015,  0.17993815,  0.57296467]],\n",
            "      dtype=float32),\n",
            " 'reward': array([1.], dtype=float32),\n",
            " 'step_type': array([1], dtype=int32)})]\n",
            "Total reward: 26.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see, the total reward achieved at the end of 26 steps is 26.\n",
        "\n",
        "For every step, we also see the discount, observation - the four variables - linear position, angular position, linear velocity, angular velocity representing the state of the environment, reward and step_type/ action. Action is the activity agent performs. Here, the agent can either make the cart go right or left. So, the actions are represented by 0 and 1. 0 means left and 1 means right. "
      ],
      "metadata": {
        "id": "SN7hJ0TB8m_R"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Implementing the Cartpole environment for a certain number of episodes.\n",
        "\n",
        "To implement cartpole environment for a certain number of episodes, we iterate over the episodes, and for every step that is till the last one in the episodes, we identify and perform an action. In correspondence to our action, we get an appropriate reward."
      ],
      "metadata": {
        "id": "aygqq6sl8yLa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# loading the cartpole environment\n",
        "env = suite_gym.load('CartPole-v0')\n",
        "\n",
        "# wrapping the python environment into a tensorflow environment\n",
        "tf_env = tf_py_environment.TFPyEnvironment(env)\n",
        "\n",
        "time_step = tf_env.reset()\n",
        "rewards = []\n",
        "steps = []\n",
        "num_episodes = 10\n",
        "\n",
        "# iterating over the episodes\n",
        "for _ in range(num_episodes):\n",
        "  episode_reward = 0\n",
        "  episode_steps = 0\n",
        "  while not time_step.is_last():\n",
        "    action = tf.random.uniform([1], 0, 2, dtype=tf.int32)\n",
        "    time_step = tf_env.step(action)\n",
        "    episode_steps += 1\n",
        "    episode_reward += time_step.reward.numpy()\n",
        "  rewards.append(episode_reward)\n",
        "  steps.append(episode_steps)\n",
        "  time_step = tf_env.reset()\n",
        "\n",
        "# calculating the statistics\n",
        "num_steps = np.sum(steps)\n",
        "avg_length = np.mean(steps)\n",
        "avg_reward = np.mean(rewards)\n",
        "tot_reward = np.sum(rewards)\n",
        "print('Total Episodes: ', num_episodes)\n",
        "print('Total Steps: ', num_steps)\n",
        "print('Total Rewards: ', tot_reward)\n",
        "print('Average Length: ', avg_length)\n",
        "print('Average Reward: ', avg_reward)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kuJajQxHgypu",
        "outputId": "9f1b9fb7-c61c-47fb-f899-02a3189ac9ca"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total Episodes:  10\n",
            "Total Steps:  261\n",
            "Total Rewards:  261.0\n",
            "Average Length:  26.1\n",
            "Average Reward:  26.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see, for 10 episodes having 261 steps in total, we have earned 261 points in total. "
      ],
      "metadata": {
        "id": "Od1lVKKf-uYa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Comparing rewards earned for both approaches."
      ],
      "metadata": {
        "id": "mGimM7XgdvQj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# step-wise rewards\n",
        "reward"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "59gD9Fvrdm_-",
        "outputId": "479d9caf-57e6-4e0f-bca6-b461e2f3354b"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>]"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# episode-wise reward\n",
        "rewards"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L2nE2z35dQkE",
        "outputId": "0b7a1ef0-1679-423a-f8e2-3f9785a7926e"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([23.], dtype=float32),\n",
              " array([35.], dtype=float32),\n",
              " array([19.], dtype=float32),\n",
              " array([43.], dtype=float32),\n",
              " array([12.], dtype=float32),\n",
              " array([25.], dtype=float32),\n",
              " array([10.], dtype=float32),\n",
              " array([46.], dtype=float32),\n",
              " array([27.], dtype=float32),\n",
              " array([21.], dtype=float32)]"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Comparing the rewards earned for both approaches, we can see that the total rewards for each step are generally smaller than the total rewards for each episode. This is because each step only allows for a small number of actions, while each episode allows for many actions. Additionally, the total rewards can vary significantly between episodes, while the rewards for each step are generally more consistent."
      ],
      "metadata": {
        "id": "heVaJv46d874"
      }
    }
  ]
}