Practicals01.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q2eZvMS3DQF7"
      },
      "source": [
        "# Graph Neural Networks - Practicals 1\n",
        "\n",
        "Welcome to our hands-on practical guide to Graph Neural Networks (GNNs)! This tutorial is designed to provide you with a complete end-to-end experience in training GNNs and to expose you to the intricacies involved in the process.\n",
        "\n",
        "Over the course of this practical session, you will:\n",
        "\n",
        "- Experience running a complete end-to-end training of Graph Neural Networks.\n",
        "- Implement two unique GNN layers, providing a deeper understanding of the internal operations of these networks.\n",
        "- Be introduced to the innovative technique of Ordered Subgraph Aggregation, which we will explore in detail.\n",
        "\n",
        "By the end of this session, you will have a solid understanding of GNNs and be equipped with the knowledge to experiment and build upon these foundational concepts.\n",
        "\n",
        "Now, let's dive into it!\n",
        "\n",
        "### [RUN] PyTorch Installation\n",
        "\n",
        "Before we start, let's ensure that PyTorch, one of the key libraries we'll be using, is correctly installed. PyTorch is a popular open-source machine learning library and it's essential for training our GNNs.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "Iit8X-n79clb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ee6cab3f-6897-42e6-bd16-f9e3cba60efa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m504.1/504.1 kB\u001b[0m \u001b[31m30.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m732.3/732.3 kB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m205.7/205.7 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m661.6/661.6 kB\u001b[0m \u001b[31m23.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for torch-geometric (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m728.8/728.8 kB\u001b[0m \u001b[31m29.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m59.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "torch.manual_seed(0)\n",
        "TORCH = torch.__version__.split('+')[0]\n",
        "DEVICE = 'cu' + torch.version.cuda.replace('.', '') if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "!pip install -q torch-sparse torch-scatter torch-cluster torch-spline-conv -f https://pytorch-geometric.com/whl/torch-{TORCH}+{DEVICE}.html\n",
        "!pip install -q torch-geometric\n",
        "!pip install -q networkx==3.1 matplotlib tqdm torchmetrics ipywidgets\n",
        "\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torchmetrics\n",
        "import torch_geometric.nn as gnn\n",
        "import torch_geometric as tg\n",
        "from torch_geometric.data import Dataset\n",
        "from torch_geometric.loader import DataLoader\n",
        "from torch_geometric.utils import to_networkx\n",
        "import networkx as nx\n",
        "import matplotlib.pyplot as plt\n",
        "from typing import Tuple\n",
        "from tqdm import tqdm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r4FG_9O4Ickb"
      },
      "source": [
        "### [INFO 1] How are we training neural networks?\n",
        "\n",
        "\n",
        "Recently, deep learning on graphs has emerged to one of the hottest research fields in the deep learning community.\n",
        "Here, **Graph Neural Networks (GNNs)** aim to generalize classical deep learning concepts to irregular structured data (in contrast to images or texts) and to enable neural networks to reason about objects and their relations.\n",
        "\n",
        "This is done by following a simple **neural message passing scheme**, where node features $\\mathbf{x}_v^{(\\ell)}$ of all nodes $v \\in \\mathcal{V}$ in a graph $\\mathcal{G} = (\\mathcal{V}, \\mathcal{E})$ are iteratively updated by aggregating localized information from their neighbors $\\mathcal{N}(v)$:\n",
        "\n",
        "$$\n",
        "\\mathbf{x}_v^{(\\ell + 1)} = f^{(\\ell + 1)}_{\\theta} \\left( \\mathbf{x}_v^{(\\ell)}, \\left\\{ \\mathbf{x}_w^{(\\ell)} : w \\in \\mathcal{N}(v) \\right\\} \\right)\n",
        "$$\n",
        "\n",
        "This tutorial will introduce you to some fundamental concepts regarding deep learning on graphs via Graph Neural Networks based on the **[PyTorch Geometric (PyG) library](https://github.com/rusty1s/pytorch_geometric)**.\n",
        "PyTorch Geometric is an extension library to the popular deep learning framework [PyTorch](https://pytorch.org/), and consists of various methods and utilities to ease the implementation of Graph Neural Networks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9g4f01pgDW3v"
      },
      "source": [
        "### Download Training Data\n",
        "\n",
        "**Datasets**\n",
        "\n",
        "When comparing Graph Neural Networks (GNNs) to other neural network architectures such as Fully-Connected Neural Networks (FCNNs) or Convolutional Neural Networks (CNNs), there are unique requirements for the data used in GNNs.\n",
        "\n",
        "GNNs typically require two fundamental data points: a graph structure, which describes how nodes are connected, and a feature matrix, providing information associated with each node.\n",
        "\n",
        "For the purpose of this tutorial, we aim to keep the focus on benchmarking and comparing various architectures. Therefore, we'll reduce the number of input features with `reduce_features` initially. This step will help us evaluate the efficiency and performance of our models in a more streamlined context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "5dfqiBpBDS2g",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1e849aa7-186a-4111-9bde-539fa14bd207"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Downloading https://www.chrsmrrs.com/graphkerneldatasets/MUTAG.zip\n",
            "Extracting data/TUDataset/MUTAG/MUTAG.zip\n",
            "Processing...\n",
            "Done!\n"
          ]
        }
      ],
      "source": [
        "from torch_geometric.datasets import TUDataset\n",
        "\n",
        "def reduce_features(data):\n",
        "    data.x = data.x.sum(dim=1, keepdim=True)\n",
        "    return data\n",
        "\n",
        "dataset = TUDataset(root='data/TUDataset', name='MUTAG', pre_transform=reduce_features)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R7o6k92FIckc"
      },
      "source": [
        "### [INFO 2] Explore dataset\n",
        "\n",
        "The most common task for graph classification is **molecular property prediction**, in which molecules are represented as graphs, and the task may be to infer whether a molecule inhibits HIV virus replication or not.\n",
        "\n",
        "The TU Dortmund University has collected a wide range of different graph classification datasets, known as the [**TUDatasets**](https://chrsmrrs.github.io/datasets/), which are also accessible via [`torch_geometric.datasets.TUDataset`](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html#torch_geometric.datasets.TUDataset) in PyTorch Geometric.\n",
        "Let's load and inspect one of the smaller ones, the **MUTAG dataset**:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "AM4qd9XBIckc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 460,
          "referenced_widgets": [
            "52b146f26a83420db8c1f805d9126a42",
            "59877be825be4d0fb30cbe395d95d886",
            "0f26010ed4b24a0b9370c50315b7e6df",
            "5713ef75aaa049d6a2d52672895966e5",
            "e0ed7d65b4c2420a87e66fbcd8975abc",
            "bdb2ae1e627e4a188978c0e481be4fd8",
            "b30d88594e2540ecb30e5e318468bbaf"
          ]
        },
        "outputId": "78f508dd-4271-480c-f3e4-4b174256a847"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "interactive(children=(IntSlider(value=0, description='index', max=187), Output()), _dom_classes=('widget-inter…"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "52b146f26a83420db8c1f805d9126a42"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from ipywidgets import interact, IntSlider\n",
        "\n",
        "@interact(index=IntSlider(max=len(dataset)-1))\n",
        "def display_dataset_element(index):\n",
        "    g = to_networkx(dataset[index], to_undirected=True)\n",
        "    plt.title(f\"Class: {dataset[index].y.item()}\")\n",
        "    nx.draw(g, with_labels=True)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1ViiBx3Ickd"
      },
      "source": [
        "## [INFO 3] Mini-batching of graphs\n",
        "\n",
        "Since graphs in graph classification datasets are usually small, a good idea is to **batch the graphs** before inputting them into a Graph Neural Network to guarantee full GPU utilization.\n",
        "In the image or language domain, this procedure is typically achieved by **rescaling** or **padding** each example into a set of equally-sized shapes, and examples are then grouped in an additional dimension.\n",
        "The length of this dimension is then equal to the number of examples grouped in a mini-batch and is typically referred to as the `batch_size`.\n",
        "\n",
        "However, for GNNs the two approaches described above are either not feasible or may result in a lot of unnecessary memory consumption.\n",
        "Therefore, PyTorch Geometric opts for another approach to achieve parallelization across a number of examples. Here, adjacency matrices are stacked in a diagonal fashion (creating a giant graph that holds multiple isolated subgraphs), and node and target features are simply concatenated in the node dimension:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kVS-K0lIckd"
      },
      "source": [
        "This procedure has some crucial advantages over other batching procedures:\n",
        "\n",
        "1. GNN operators that rely on a message passing scheme do not need to be modified since messages are not exchanged between two nodes that belong to different graphs.\n",
        "\n",
        "2. There is no computational or memory overhead since adjacency matrices are saved in a sparse fashion holding only non-zero entries, *i.e.*, the edges.\n",
        "\n",
        "PyTorch Geometric automatically takes care of **batching multiple graphs into a single giant graph** with the help of the [`torch_geometric.data.DataLoader`](https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.DataLoader) class."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5Aww184Ickd"
      },
      "source": [
        "## [INFO 4] Exploring Graph Classification with Graph Neural Networks\n",
        "\n",
        "In this tutorial session, we will delve into the application of **Graph Neural Networks (GNNs) for graph classification**. Unlike node classification, graph classification is tasked with classifying entire graphs, given a **dataset of graphs**, based on their structural properties. Our aim is to generate embeddings for entire graphs in such a way that they are linearly separable for a given task.\n",
        "\n",
        "A key part of this process is the training and evaluation of our model. We will use the `train_and_eval` function for this purpose:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "SWnFhEXEIckf"
      },
      "outputs": [],
      "source": [
        "def train_and_eval(model : nn.Module, dataset : Dataset, hparams: dict = {}) -> Tuple[nn.Module, float]:\n",
        "    \"\"\"\n",
        "    Train and evaluate the model on the dataset.\n",
        "    A boilerplate for training and evaluating a neural network.\n",
        "\n",
        "    Args:\n",
        "        model (nn.Module): The model to train and evaluate.\n",
        "        dataset (Dataset): The dataset to train and evaluate on.\n",
        "        hparams (dict): A dictionary of hyperparameters.\n",
        "\n",
        "    Returns:\n",
        "        nn.Module: The trained model with the best validation accuracy.\n",
        "        float: The best validation accuracy.\n",
        "    \"\"\"\n",
        "\n",
        "    hparams[\"num_epochs\"] = hparams.get(\"num_epochs\", 100)\n",
        "    hparams[\"lr\"] = hparams.get(\"lr\", 0.01)\n",
        "\n",
        "    # shuffle dataset\n",
        "    dataset = dataset.shuffle()\n",
        "    train_sizes = {\"ENZYMES\": 540, \"MUTAG\": 150}\n",
        "    sz = train_sizes[dataset.name]\n",
        "\n",
        "    # init training and validation dataloaders\n",
        "    dataloader_train = DataLoader(dataset[:sz], batch_size=16, shuffle=True)\n",
        "    dataloader_val = DataLoader(dataset[sz:], batch_size=16)\n",
        "\n",
        "    # init optimizer and loss\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=hparams[\"lr\"])\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "    # init metrics\n",
        "    metric_loss_train = torchmetrics.MeanMetric()\n",
        "    metric_accuracy_train = torchmetrics.Accuracy(task=\"multiclass\", num_classes=dataset.num_classes)\n",
        "    metric_accuracy_val = torchmetrics.Accuracy(task=\"multiclass\", num_classes=dataset.num_classes)\n",
        "\n",
        "    best_model_state = None\n",
        "    best_val_acc = 0.0\n",
        "\n",
        "    # training loop\n",
        "    num_epochs = hparams[\"num_epochs\"]\n",
        "    epoch_bar = tqdm(list(range(num_epochs)), desc=\"Epochs\" + \" \"*100)\n",
        "    for epoch in epoch_bar:\n",
        "\n",
        "        # training loop\n",
        "        model.train()\n",
        "        for data in dataloader_train:\n",
        "            optimizer.zero_grad()\n",
        "            logits = model(data.x, data.edge_index, data.batch)\n",
        "            loss = criterion(logits, data.y)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "            metric_loss_train(loss)\n",
        "            metric_accuracy_train(logits.argmax(dim=1), data.y)\n",
        "\n",
        "\n",
        "        # evaluation loop\n",
        "        model.eval()\n",
        "        for data in dataloader_val:\n",
        "            with torch.no_grad():\n",
        "                logits = model(data.x, data.edge_index, data.batch)\n",
        "                pred = logits.argmax(dim=1)\n",
        "                metric_accuracy_val(pred, data.y)\n",
        "\n",
        "        train_loss = metric_loss_train.compute().item()\n",
        "        train_acc = metric_accuracy_train.compute().item()\n",
        "        val_acc = metric_accuracy_val.compute().item()\n",
        "\n",
        "        # progress bar\n",
        "        epoch_bar.set_description(\"\".join([\n",
        "            f'Epoch: {epoch+1:03d} | ',\n",
        "            f'Val Acc: {val_acc:.3f} | ',\n",
        "            f'Train Acc: {train_acc:.3f} | ',\n",
        "            f'Train Loss: {train_loss:.3f} | ',\n",
        "            ]))\n",
        "\n",
        "        # update best model\n",
        "        if val_acc > best_val_acc:\n",
        "            best_model_state = model.state_dict()\n",
        "            best_val_acc = val_acc\n",
        "\n",
        "    best_model = model\n",
        "    best_model.load_state_dict(best_model_state)\n",
        "\n",
        "    return best_model, best_val_acc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ap0EHntz5kLI"
      },
      "source": [
        "### [INFO 5] Define our model\n",
        "\n",
        "We will use the `nn.Module` class from PyTorch to define our model.\n",
        "Since having multiple datapoints we need to define a model that has multiple inputs in the `forward` function.\n",
        "\n",
        "Training a GNN for graph classification usually follows a simple recipe:\n",
        "\n",
        "1. Embed each node by performing multiple rounds of message passing\n",
        "2. Aggregate node embeddings into a unified graph embedding (**readout layer**)\n",
        "3. Train a final classifier on the graph embedding\n",
        "\n",
        "There exists multiple **readout layers** in literature, but the most common one is to simply take the average of node embeddings:\n",
        "\n",
        "$$\n",
        "\\mathbf{x}_{\\mathcal{G}} = \\frac{1}{|\\mathcal{V}|} \\sum_{v \\in \\mathcal{V}} \\mathcal{x}^{(L)}_v\n",
        "$$\n",
        "\n",
        "PyTorch Geometric provides this functionality via [`gnn.global_mean_pool`](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.glob.global_mean_pool), which takes in the node embeddings of all nodes in the mini-batch and the assignment vector `batch` to compute a graph embedding of size `[batch_size, hidden_channels]` for each graph in the batch.\n",
        "\n",
        "The final architecture for applying GNNs to the task of graph classification then looks as follows and allows for complete end-to-end training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "ZLlNwk-3DZu7"
      },
      "outputs": [],
      "source": [
        "class SetConv(nn.Linear):\n",
        "    \"\"\"\n",
        "    A simple linear layer that performs action on every node feature\n",
        "    \"\"\"\n",
        "\n",
        "    def forward(self, x, edge_index):\n",
        "        return super().forward(x)\n",
        "\n",
        "\n",
        "class GraphNetModel(nn.Module):\n",
        "\n",
        "    def __init__(self, num_features, num_classes, conv_layer_cls=None):\n",
        "        super().__init__()\n",
        "\n",
        "        # in case no gnnLayer is provided, use a simple linear layer\n",
        "        if conv_layer_cls is None:\n",
        "            conv_layer_cls = SetConv\n",
        "\n",
        "        # first graph covolutional layer\n",
        "        self.conv1 = conv_layer_cls(num_features, 32)\n",
        "        self.conv2 = conv_layer_cls(32, 32)\n",
        "        self.conv3 = conv_layer_cls(32, 32)\n",
        "        # TODO🚀(optional):\n",
        "        # - add more conv layers\n",
        "\n",
        "        # last graph convolutional layer\n",
        "        self.convL = conv_layer_cls(32, 32)\n",
        "\n",
        "        # TODO🚀(optional):\n",
        "        # - in many tutorials, the stantard dropout layer\n",
        "        #    on **features** is used which can help\n",
        "        #    with overfitting. Simlarly, the standard\n",
        "        #    batch normalization can be used as well.\n",
        "        # - `self.dropout = nn.Dropout(0.5)`\n",
        "\n",
        "        # TODO🚀(optional):\n",
        "        # - you might try to add batchnorm on features\n",
        "        #   between the conv layers\n",
        "        # - `self.batchnorm = nn.BatchNorm1d(32)`\n",
        "\n",
        "        self.linear = nn.Linear(32, num_classes)\n",
        "\n",
        "    def forward(self, x, edge_index, batch):\n",
        "\n",
        "        # apply first layer and follow with ReLU\n",
        "        x = self.conv1(x, edge_index).relu()\n",
        "        x = self.conv2(x, edge_index).relu()\n",
        "        x = self.conv3(x, edge_index).relu()\n",
        "\n",
        "        # TODO🚀(optional):\n",
        "        # - apply more conv layers\n",
        "\n",
        "        x = self.convL(x, edge_index).relu()\n",
        "\n",
        "        # this (set) pooling makes the function\n",
        "        # invariant to the order of the nodes\n",
        "        x = gnn.global_mean_pool(x, batch)\n",
        "\n",
        "        # TODO🚀(optional):\n",
        "        # add dropout or batchnorm of features\n",
        "        # - `x = self.dropout(x)`\n",
        "\n",
        "        # apply linear layer\n",
        "        x = self.linear(x)\n",
        "\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6M2kk0J45kLO"
      },
      "source": [
        "### [RUN] Initial Training and Evaluation without Incorporating Graph Convolutional Neural Network Layer\n",
        "\n",
        "Let's begin our experiment with a simple run. This initial training and evaluation process does not involve any Graph Convolutional Neural Network layers. It allows us to gauge baseline performance and understand the effect of these layers when we introduce them later in the process.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vsxGpG_tDuoV",
        "outputId": "bf5f3a87-7fdd-43d6-d007-7a2809d60461"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Epoch: 100 | Val Acc: 0.632 | Train Acc: 0.673 | Train Loss: 0.632 | : 100%|██████████| 100/100 [00:13<00:00,  7.36it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model size: 3298\n",
            "Best Accuracy: 0.6315789222717285\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "model = GraphNetModel(dataset.num_features, dataset.num_classes)\n",
        "model, val_accuracy = train_and_eval(model, dataset)\n",
        "\n",
        "print(\"Model size:\", sum(p.numel() for p in model.parameters()))\n",
        "print(\"Best Accuracy:\", val_accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LknFIgBmIckf"
      },
      "source": [
        "### [Task 1] Implementation of the GraphConv Layer\n",
        "\n",
        "In this task, we will implement a Graph Convolutional (GraphConv) Layer. This layer is an integral part of Graph Neural Networks (GNNs) and allows for the propagation of information through the nodes of a graph.\n",
        "\n",
        "The equation that governs the operation of the GraphConv layer is:\n",
        "\n",
        "$$\n",
        "\\begin{align}\n",
        "F^{(t+1)} &= \\sigma(F^{(t)} W_1^{(t)} + A F^{(t)} W_2^{(t)} + b^{(t)})\n",
        "\\end{align}\n",
        "$$\n",
        "\n",
        "where,\n",
        "\n",
        "- $F^{(t+1)}$ represents the features of the nodes in the graph at the next step (t+1).\n",
        "- $F^{(t)}$ represents the features of the nodes in the graph at the current step (t).\n",
        "- $W_1^{(t)}$ and $W_2^{(t)}$ are learnable weights at the current step (t).\n",
        "- $A$ is the adjacency matrix representing the connections between nodes in the graph.\n",
        "- $\\sigma$ is the activation function (e.g., ReLU, sigmoid, tanh, etc.).\n",
        "- $b^{(t)}$ is the learnable bias at the current step (t).\n",
        "\n",
        "In this equation, we are updating the features of each node based on its own features and the features of its neighbors. The learnable weights and bias allow the model to adjust and optimize the impact of each node's features and its neighbors' features in the propagation process.\n",
        "\n",
        "Your task is to implement this functionality in a GraphConv layer.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "nV9rtP9MIckg"
      },
      "outputs": [],
      "source": [
        "class OurGraphConvLayer(nn.Module):\n",
        "\n",
        "    def __init__(self, in_channels, out_channels):\n",
        "        super().__init__()\n",
        "\n",
        "        self.linear1 = nn.Linear(in_channels, out_channels)\n",
        "        self.linear2 = nn.Linear(in_channels, out_channels, bias=False)\n",
        "\n",
        "    def forward(self, x, edge_index):\n",
        "        nx = x.shape[0]\n",
        "\n",
        "        # create abstraction of sparse adjacency matrix from edge_index\n",
        "        A = torch.sparse_coo_tensor(edge_index, torch.ones_like(edge_index[0]).float(), size=(nx, nx))\n",
        "\n",
        "\n",
        "        W1 = self.linear1.weight.T\n",
        "        W2 = self.linear2.weight.T\n",
        "        b = self.linear1.bias\n",
        "\n",
        "        out = x @ W1 + A @ x @ W2 + b\n",
        "\n",
        "        return out"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENmzuvhd5kLP"
      },
      "source": [
        "**test your model with GraphConv Layer:**\n",
        "- This layer is also implemented in PyG as [torch_geometric.nn.GraphConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCNConv.html#torch_geometric.nn.conv.GraphConv).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 398
        },
        "id": "BJHNkF2z5kLP",
        "outputId": "31e0ad65-a9a7-4d3a-b904-e4c4dbfd4379"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Epoch: 067 | Val Acc: 0.822 | Train Acc: 0.829 | Train Loss: 0.359 | :  67%|██████▋   | 67/100 [00:13<00:06,  4.89it/s]\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-1b7efe694486>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmodel_our1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGraphNetModel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_features\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_classes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconv_layer_cls\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mOurGraphConvLayer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtrained_model_our1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mval_accuracy_our1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_and_eval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_our1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Model size:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel_our1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Best Accuracy:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mval_accuracy_our1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-4-6d2aafd7eb15>\u001b[0m in \u001b[0;36mtrain_and_eval\u001b[0;34m(model, dataset, hparams)\u001b[0m\n\u001b[1;32m     45\u001b[0m         \u001b[0;31m# training loop\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m         \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 47\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdataloader_train\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     48\u001b[0m             \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     49\u001b[0m             \u001b[0mlogits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0medge_index\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    631\u001b[0m                 \u001b[0;31m# TODO(https://github.com/pytorch/pytorch/issues/76750)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    632\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[call-arg]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 633\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    634\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_yielded\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    635\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dataset_kind\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_DatasetKind\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mIterable\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m_next_data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    675\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    676\u001b[0m         \u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 677\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dataset_fetcher\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfetch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    678\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_pin_memory\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    679\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_pin_memory_device\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py\u001b[0m in \u001b[0;36mfetch\u001b[0;34m(self, possibly_batched_index)\u001b[0m\n\u001b[1;32m     52\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollate_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch_geometric/loader/dataloader.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m     18\u001b[0m         \u001b[0melem\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0melem\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mBaseData\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m             return Batch.from_data_list(batch, self.follow_batch,\n\u001b[0m\u001b[1;32m     21\u001b[0m                                         self.exclude_keys)\n\u001b[1;32m     22\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0melem\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch_geometric/data/batch.py\u001b[0m in \u001b[0;36mfrom_data_list\u001b[0;34m(cls, data_list, follow_batch, exclude_keys)\u001b[0m\n\u001b[1;32m     74\u001b[0m         Will exclude any keys given in :obj:`exclude_keys`.\"\"\"\n\u001b[1;32m     75\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 76\u001b[0;31m         batch, slice_dict, inc_dict = collate(\n\u001b[0m\u001b[1;32m     77\u001b[0m             \u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m             \u001b[0mdata_list\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch_geometric/data/collate.py\u001b[0m in \u001b[0;36mcollate\u001b[0;34m(cls, data_list, increment, add_batch, follow_batch, exclude_keys)\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m             \u001b[0;31m# Collate attributes into a unified representation:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m             value, slices, incs = _collate(attr, values, data_list, stores,\n\u001b[0m\u001b[1;32m     86\u001b[0m                                            increment)\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch_geometric/data/collate.py\u001b[0m in \u001b[0;36m_collate\u001b[0;34m(key, values, data_list, stores, increment)\u001b[0m\n\u001b[1;32m    132\u001b[0m         \u001b[0mslices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcumsum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcat_dim\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mincrement\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m             \u001b[0mincs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_incs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstores\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    135\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mincs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdim\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mincs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m                 values = [\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch_geometric/data/collate.py\u001b[0m in \u001b[0;36mget_incs\u001b[0;34m(key, values, data_list, stores)\u001b[0m\n\u001b[1;32m    263\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 265\u001b[0;31m def get_incs(key, values: List[Any], data_list: List[BaseData],\n\u001b[0m\u001b[1;32m    266\u001b[0m              stores: List[BaseStorage]) -> Tensor:\n\u001b[1;32m    267\u001b[0m     repeats = [\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "model_our1 = GraphNetModel(dataset.num_features, dataset.num_classes, conv_layer_cls=OurGraphConvLayer)\n",
        "trained_model_our1, val_accuracy_our1 = train_and_eval(model_our1, dataset)\n",
        "\n",
        "print(\"Model size:\", sum(p.numel() for p in model_our1.parameters()))\n",
        "print(\"Best Accuracy:\", val_accuracy_our1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJkXpjIYIckg"
      },
      "source": [
        "### [Task 2] Implementation of the Simplified Graph Convolutional Layer (GCN)\n",
        "\n",
        "In this task, we delve deeper into the mechanics of Graph Neural Networks (GNNs) by implementing a simplified version of the Graph Convolutional Layer (GCN).\n",
        "\n",
        "The operation of this simplified GCN is determined by the following equations:\n",
        "\n",
        "$$\n",
        "\\begin{align}\n",
        "\\hat{D} &= D + I \\\\\n",
        "\\hat{A} &= A + I \\\\\n",
        "\\tilde{A} &= \\hat{D}^{-1/2} \\hat{A} \\hat{D}^{-1/2} \\\\\n",
        "F^{(t+1)} &= \\sigma(\\tilde{A} F^{(t)} W^{(t)} + b^{(t)})\n",
        "\\end{align}\n",
        "$$\n",
        "\n",
        "where,\n",
        "\n",
        "- $\\hat{D}$ and $\\hat{A}$ are the degree matrix and adjacency matrix, respectively, with added self-connections (denoted by the identity matrix $I$). This modification ensures that each node receives its own features in addition to its neighbors'.\n",
        "- $\\tilde{A}$ is the normalized adjacency matrix. This normalization step makes the propagation of features across the graph more stable.\n",
        "- $F^{(t+1)}$ represents the features of the nodes in the graph at the next step (t+1).\n",
        "- $F^{(t)}$ represents the features of the nodes in the graph at the current step (t).\n",
        "- $W^{(t)}$ is the learnable weight matrix at the current step (t).\n",
        "- $\\sigma$ is the activation function (e.g., ReLU, sigmoid, tanh, etc.).\n",
        "- $b^{(t)}$ is the learnable bias at the current step (t).\n",
        "\n",
        "The goal of this task is to implement this simplified GCN layer. This layer will help propagate and transform features across the graph, making it possible for our model to learn from the structure and features of the graph.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CWD2p3UkIckg"
      },
      "outputs": [],
      "source": [
        "class OurGCNConvLayer(nn.Module):\n",
        "\n",
        "    def __init__(self, in_channels, out_channels):\n",
        "        super().__init__()\n",
        "\n",
        "        # initialize linear layer with weight `W` and bias `b`\n",
        "        self.linear = nn.Linear(in_channels, out_channels)\n",
        "\n",
        "    def forward(self, x, edge_index):\n",
        "        nx = x.shape[0]\n",
        "\n",
        "        # create abstraction of sparse adjacency matrix from edge_index\n",
        "        A = tg.utils.to_torch_coo_tensor(edge_index)\n",
        "\n",
        "        # also sparse identity matrix\n",
        "        I = torch.sparse_coo_tensor(torch.stack([torch.arange(nx), torch.arange(nx)]), torch.ones(nx).float(), size=(nx, nx))\n",
        "\n",
        "        # add self-loops\n",
        "        A_hat = A + I\n",
        "\n",
        "        # degree vector\n",
        "        deg_inv_sqrt = A_hat.sum(dim=1).to_dense().pow(-0.5)\n",
        "        deg_inv_sqrt[deg_inv_sqrt == float('inf')] = 0.0\n",
        "\n",
        "        # normalized inverse degree matrix\n",
        "        Di = I * deg_inv_sqrt\n",
        "        assert Di.is_sparse\n",
        "\n",
        "        # using bias and weight of linear layer\n",
        "        W = self.linear.weight.T\n",
        "        b = self.linear.bias\n",
        "\n",
        "        # normalize adjacency matrix\n",
        "        A_norm = Di @ A_hat @ Di\n",
        "        assert A_norm.is_sparse\n",
        "\n",
        "        x = A_norm @ x @ W + b\n",
        "        return x\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**test your model with GCNConv Layer:**\n",
        " - note that this layer has less parameters that GraphConv and also does not differentiate between values of node's neightbors and value of node itself (!). This is very important to keep in mind for practical usage -- GCNConv does not work nicely with complente graphs or unlabeled graphs (our case).\n",
        " - Even though, you might find this layer useful in the following Task 3.\n",
        " - This layer is also imple\n",
        " mented in PyG as [torch_geometric.nn.GCNConv](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCNConv.html#torch_geometric.nn.conv.GCNConv)."
      ],
      "metadata": {
        "id": "C0mbkcbVLVis"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "60iXIX3SIckg"
      },
      "outputs": [],
      "source": [
        "model_our2 = GraphNetModel(dataset.num_features, dataset.num_classes, conv_layer_cls=OurGCNConvLayer)\n",
        "trained_model_our2, val_accuracy_our2 = train_and_eval(model_our2, dataset)\n",
        "\n",
        "print(\"Model size:\", sum(p.numel() for p in model_our2.parameters()))\n",
        "print(\"Best Accuracy:\", val_accuracy_our2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcbxEfye5kLP"
      },
      "source": [
        "### But what structures are our models really capturing?\n",
        "\n",
        "Consider the following graphs:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "miuc51C75kLP"
      },
      "outputs": [],
      "source": [
        "graph_G = nx.from_edgelist([(0,1), (1,2), (2,0), (3,4), (4,5), (5,3), (1,4)])\n",
        "graph_H = nx.from_edgelist([(0,1), (1,2), (2,3), (3,4), (4,5), (5,0), (1,4)])\n",
        "\n",
        "plt.subplots(1,2, figsize=(10,5))\n",
        "plt.subplot(121)\n",
        "plt.title(\"G\")\n",
        "nx.draw_networkx(graph_G)\n",
        "plt.subplot(122)\n",
        "plt.title(\"H\")\n",
        "nx.draw_networkx(graph_H)\n",
        "plt.axis('off')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bGXaizmP5kLP"
      },
      "source": [
        "### So why are outputs identical if the graphs are clearly not isomorphic?\n",
        "You might be wondering, why do we encounter identical outputs when the graphs are clearly non-isomorphic? The answer lies in the 1-Weisfeiler-Lehman test, commonly referred to as 1-WL or [Color Refinement](https://en.wikipedia.org/wiki/Colour_refinement_algorithm).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sPTfBnox5kLP"
      },
      "outputs": [],
      "source": [
        "model_our1.eval()\n",
        "\n",
        "def from_networkx(G: nx.Graph) -> tg.data.Data:\n",
        "    \"\"\"Utility function\"\"\"\n",
        "    data = tg.utils.from_networkx(G)\n",
        "    data.x = torch.ones((G.number_of_nodes(), 1), dtype=torch.float)\n",
        "    return data\n",
        "\n",
        "# simple features to test the model\n",
        "data_G = from_networkx(graph_G)\n",
        "data_H = from_networkx(graph_H)\n",
        "# only one batch\n",
        "batch = torch.zeros(6).long()\n",
        "\n",
        "logits_G = model_our1(data_G.x, data_G.edge_index, batch)\n",
        "logits_H = model_our1(data_H.x, data_H.edge_index, batch)\n",
        "\n",
        "print(\"Prediction for G: \", logits_G.detach().numpy())\n",
        "print(\"Prediction for H:\", logits_H.detach().numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ItXFtDxM5kLP"
      },
      "source": [
        "Breaking the Symmetry\n",
        "However, even in such a situation, we are not without options. We can attempt to break this symmetry through permutation-invariant operations such as node marking or node deletion. It is clear that certain sets of graphs do not even share the same degree sequences, highlighting their distinctness. This approach could potentially allow us to distinguish between non-isomorphic graphs that were previously yielding identical outputs. We will cover this in the following section"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zcYJwTzvIckh"
      },
      "source": [
        "# Ordered Subgraph Aggregation\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "The `graph_into_subgraphs` function is a key component in our preprocessing stage. It is responsible for transforming the original graph into subgraphs. This is achieved by duplicating each node representation for every other node and concatenating an identity matrix to these duplicates, which essentially allows us to embed the node's local context. Additionally, it manipulates the edge indices in such a way that edges are created between corresponding nodes across the subgraphs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vt33euERIckh"
      },
      "outputs": [],
      "source": [
        "def graph_into_subgraphs(data):\n",
        "    # 'n' captures the number of nodes in the graph\n",
        "    # 'm' captures the number of edges in the graph\n",
        "    n, m = len(data.x), len(data.edge_index[0])\n",
        "\n",
        "    # Duplicate the node features for each node in the graph, and adds unique\n",
        "    # identifier on each node, allowing us to capture the local context\n",
        "    data.x = torch.cat([data.x.repeat(n, 1), torch.eye(n).view(-1, 1)], dim=1)\n",
        "\n",
        "    # Adjust the edge indices to create new edges in the subgraphs\n",
        "    # This is done by repeating the edge_index tensor 'n' times,\n",
        "    # and adding an offset for each node's corresponding edges\n",
        "    data.edge_index = data.edge_index.repeat(1, n) + n*torch.arange(n).repeat_interleave(m)\n",
        "\n",
        "    return data\n",
        "\n",
        "def draw_graph_with_aligned_components(graph, name):\n",
        "    pos = nx.spring_layout(graph, seed=42)\n",
        "    n = graph.number_of_nodes()\n",
        "    data = from_networkx(graph)\n",
        "    data.x = torch.ones(len(graph), 1)\n",
        "    bagged_data = graph_into_subgraphs(data)\n",
        "    bagged_graph = to_networkx(bagged_data, to_undirected=True)\n",
        "    plt.subplots(1,6, figsize=(3*n,3))\n",
        "    plt.suptitle(name, fontsize=16)\n",
        "    for i, conn_comp in enumerate(nx.connected_components(bagged_graph)):\n",
        "        plt.subplot(1,6,i+1)\n",
        "        conn_g = bagged_graph.subgraph(conn_comp)\n",
        "        nodes = sorted(conn_comp)\n",
        "        pos_updated = {k: pos[j] for j, k in enumerate(nodes)}\n",
        "        marks = bagged_data.x[torch.LongTensor(nodes), -1]\n",
        "        nx.draw_networkx(conn_g, with_labels=True, node_color=marks, font_color='gray', pos=pos_updated)\n",
        "        plt.axis('off')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "\n",
        "draw_graph_with_aligned_components(graph_G, \"graph_into_subgraphs(G)\")\n",
        "draw_graph_with_aligned_components(graph_H, \"graph_into_subgraphs(H)\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Updating dataset\n",
        "\n",
        "We are applying a transformation step on the TUDataset by using the function `graph_into_subgraphs`. The choice of this transformation function is inspired by the approach presented in the research paper titled [Ordered Subgraph Aggregation Networks](https://arxiv.org/pdf/2206.11168.pdf).\n",
        "\n",
        "This function allows us to decompose each graph into subgraphs, capturing the local structural information around each node. When training Graph Neural Networks (GNNs), this local information can be aggregated to generate more meaningful node representations that account for both local and global structural information.\n",
        "\n",
        "By applying this transformation at the dataset level, as `transform=graph_into_subgraphs`, we can efficiently preprocess our data. This is an advantageous approach, as opposed to implementing the same operation as part of the GNN model, which might introduce computational overhead during the training process.\n",
        "\n"
      ],
      "metadata": {
        "id": "RlfgytuKOs_5"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b2Y4h2ixIckh"
      },
      "outputs": [],
      "source": [
        "# Open the dataset with `transform=graph_into_subgraphs`\n",
        "bagged_dataset = TUDataset(root='data/TUDataset', name='MUTAG', pre_transform=reduce_features, transform=graph_into_subgraphs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pfNHD5R85kLQ"
      },
      "outputs": [],
      "source": [
        "model_our_bagged = GraphNetModel(bagged_dataset.num_features, bagged_dataset.num_classes, OurGCNConvLayer)\n",
        "trained_model_our_bagged, val_accuracy_our_bagged = train_and_eval(model_our_bagged, bagged_dataset)\n",
        "\n",
        "print(\"Model size:\", sum(p.numel() for p in model_our_bagged.parameters()))\n",
        "print(\"Best Accuracy:\", val_accuracy_our_bagged)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lap0L-ay5kLQ"
      },
      "source": [
        "**With our modifications, our model has demonstrated improvements not only in terms of accuracy but also in its expressive power:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-n5XbpE45kLQ"
      },
      "outputs": [],
      "source": [
        "model_our_bagged.eval()\n",
        "\n",
        "# simple features to test the model\n",
        "data_G = graph_into_subgraphs(from_networkx(graph_G))\n",
        "data_H = graph_into_subgraphs(from_networkx(graph_H))\n",
        "batch = torch.zeros(len(data_G.x)).long()\n",
        "\n",
        "logits_G = model_our_bagged(data_G.x, data_G.edge_index, batch)\n",
        "logits_H = model_our_bagged(data_H.x, data_H.edge_index, batch)\n",
        "\n",
        "print(\"Prediction for G: \", logits_G.detach().numpy())\n",
        "print(\"Prediction for H:\", logits_H.detach().numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Enhanced Expressivity!\n",
        "This term refers to our model's improved ability to capture and articulate intricate patterns within our data. Through the use of Ordered Subgraph Aggregation, our model can now distinguish between more complex and nuanced graph structures. This elevates the sophistication of our predictions and enhances our model's overall performance."
      ],
      "metadata": {
        "id": "M-VROkugSS9O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [TASK 3] (Optional) Time to Optimize your Graph Neural Network!\n",
        "Now, it's time to head back to the `GraphNetModel` and `train_and_eval` functions. Seize this opportunity to devise a more efficient model on the dataset, or more likely, the bagged_dataset.\n",
        "\n",
        "This step is optional but highly encouraged. By refining and optimizing your model, you not only enhance its performance but also deepen your understanding of the inner workings of Graph Neural Networks. So go ahead, push your boundaries and let's see how much your model can achieve!"
      ],
      "metadata": {
        "id": "OEydIjczTlTA"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4q6lmapnIckh"
      },
      "source": [
        "[*] Text sections INFO1-INFO5 were mostly adapted from Pytorch Documentation, please visit [PyG Colab Tutorial](https://pytorch-geometric.readthedocs.io/en/latest/get_started/colabs.html) for more information.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "52b146f26a83420db8c1f805d9126a42": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [
              "widget-interact"
            ],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_59877be825be4d0fb30cbe395d95d886",
              "IPY_MODEL_0f26010ed4b24a0b9370c50315b7e6df"
            ],
            "layout": "IPY_MODEL_5713ef75aaa049d6a2d52672895966e5"
          }
        },
        "59877be825be4d0fb30cbe395d95d886": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "IntSliderModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "IntSliderModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "IntSliderView",
            "continuous_update": true,
            "description": "index",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_e0ed7d65b4c2420a87e66fbcd8975abc",
            "max": 187,
            "min": 0,
            "orientation": "horizontal",
            "readout": true,
            "readout_format": "d",
            "step": 1,
            "style": "IPY_MODEL_bdb2ae1e627e4a188978c0e481be4fd8",
            "value": 187
          }
        },
        "0f26010ed4b24a0b9370c50315b7e6df": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_b30d88594e2540ecb30e5e318468bbaf",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "<Figure size 640x480 with 1 Axes>",
                  "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGbCAYAAABZBpPkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABkn0lEQVR4nO3deXhM9/4H8Pcs2fdFYk1TIkEEVSnSEiHUrvxQtbZSV1vdtNqii63VjXJb1VZx9Vpq56oghNiJUK1IVRJKQkhkIttMMsnMnN8fmlTINslMzizv1/Pc5zLnzDmfKOe8z/luEkEQBBAREZHVkopdABEREYmLYYCIiMjKMQwQERFZOYYBIiIiK8cwQEREZOUYBoiIiKwcwwAREZGVYxggIiKycgwDREREVo5hgMgM+fv74/nnnxe7DCKyEAwDRCbkypUrmDp1Klq2bAl7e3u4urriySefxL///W8UFRWJXV6d3bx5E6NHj4a7uztcXV0xbNgwXL16VeyyiOhvcrELIKJ7oqOjMWrUKNjZ2WHixIlo3749SkpKcPz4cbzzzjtISkrCihUrxC5Tb4WFhYiIiEBeXh5mz54NGxsbLFmyBOHh4fjtt9/g5eUldolEVo9hgMgE/PXXXxgzZgweeeQRHDp0CE2aNCnfNm3aNKSmpiI6OlrECutu+fLlSElJwZkzZxAaGgoAGDBgANq3b4/Fixdj4cKFIldIRGwmIDIBX3zxBQoLC7Fq1aoKQaBMQEAA3njjjSq/n5OTgxkzZiAkJATOzs5wdXXFgAED8Pvvvz+07zfffIPg4GA4OjrCw8MDXbp0wYYNG8q3FxQU4M0334S/vz/s7Ozg4+ODvn374tdffy3fR6VS4c8//0R2dnaNP9vWrVsRGhpaHgQAoE2bNujTpw82b95c4/eJyPgYBohMwC+//IKWLVsiLCysTt+/evUqdu7cicGDB+Orr77CO++8g8TERISHhyMjI6N8vx9//BGvv/462rVrh6VLl2LevHno1KkT4uPjy/d56aWX8N133+H//u//sHz5csyYMQMODg64dOlS+T5nzpxB27ZtsWzZsmrr0ul0uHDhArp06fLQtieeeAJXrlxBQUFBnX5mIjIcNhMQiSw/Px83b97EsGHD6nyMkJAQJCcnQyr9J99PmDABbdq0wapVq/Dhhx8CuNcvITg4GFu2bKnyWNHR0ZgyZQoWL15c/tm7775bp7pycnKgVqsrfdtR9llGRgaCgoLqdHwiMgyGASKR5efnAwBcXFzqfAw7O7vyX2u1WuTm5sLZ2RlBQUEVXu+7u7vjxo0bSEhIqPDa/n7u7u6Ij49HRkYGmjZtWuk+vXr1giAINdZVNgLi/vrK2NvbV9iHiMTDZgIikbm6ugJAvV6X63Q6LFmyBK1bt4adnR28vb3RqFEjXLhwAXl5eeX7vffee3B2dsYTTzyB1q1bY9q0aThx4kSFY33xxRe4ePEiWrRogSeeeAJz586t8zBABwcHAIBarX5oW3FxcYV9iEg8DANEInN1dUXTpk1x8eLFOh9j4cKFeOutt9CzZ0+sW7cOMTExOHDgAIKDg6HT6cr3a9u2LS5fvoyNGzfiqaeewrZt2/DUU09hzpw55fuMHj0aV69exTfffIOmTZviyy+/RHBwMPbu3at3XZ6enrCzs8OtW7ce2lb2WVVvH4io4UiE2rzrIyKjmjp1KlasWIGTJ0+ie/fuNe7v7++PXr16Yc2aNQCATp06wdPTE4cOHaqwX/PmzREQEIDDhw9XepySkhKMGDEC+/btQ2FhYfmr+/tlZWWhc+fO8Pf3x/Hjx/X+2UJDQyGRSHDmzJkKn/fr1w9XrlzBlStX9D4mERkW3wwQmYB3330XTk5OePHFF5GZmfnQ9itXruDf//53ld+XyWQPteFv2bIFN2/erPCZQqGo8HtbW1u0a9cOgiCgtLQUWq22QrMCAPj4+KBp06YVXvXrM7Rw5MiRSEhIwNmzZ8s/u3z5Mg4dOoRRo0bV+H0iMj52ICQyAa1atcKGDRvw7LPPom3bthVmIDx58iS2bNlS7VoEgwcPxvz58/HCCy8gLCwMiYmJWL9+PVq2bFlhv379+qFx48Z48skn4evri0uXLmHZsmUYNGgQXFxckJubi+bNm2PkyJHo2LEjnJ2dERsbi4SEhAqjC86cOYOIiAjMmTMHc+fOrfZne+WVV/Djjz9i0KBBmDFjBmxsbPDVV1/B19cXb7/9dn3+2IjIUAQiMhnJycnClClTBH9/f8HW1lZwcXERnnzySeGbb74RiouLy/d75JFHhEmTJpX/vri4WHj77beFJk2aCA4ODsKTTz4pnDp1SggPDxfCw8PL9/vhhx+Enj17Cl5eXoKdnZ3QqlUr4Z133hHy8vIEQRAEtVotvPPOO0LHjh0FFxcXwcnJSejYsaOwfPnyCnXGxcUJAIQ5c+bU6udKT08XRo4cKbi6ugrOzs7C4MGDhZSUlDr/ORGRYbHPABERkZVjnwEiIiIrxzBARERk5RgGiIiIrBzDABERkZVjGCAiIrJyDANERERWjmGAiIjIyjEMEBERWTmGASIiIivHMEBERGTlGAaIiIisHMMAERGRlWMYICIisnIMA0RERFaOYYCIiMjKMQwQERFZOYYBIiIiK8cwQEREZOUYBoiIiKwcwwAREZGVYxggIiKycgwDREREVo5hgIiIyMoxDBAREVk5hgEiIiIrJxe7AGuiVGtwTaFEiUYHW7kU/l5OcLLjfwIiIhIX70RGlpJZgPXxaYi7nIW0HBWE+7ZJAPh5OiIiyAfjuvqhta+LWGUSEZEVkwiCINS8G+krPUeF2TsScSw1GzKpBFpd1X/MZdt7BHhj4fAQtPB0bMBKiYjI2jEMGMHGhDTM2ZUEjU6oNgQ8SCaVQC6VYN7QYIwJ9TNihURERP9gGDCwZXEpWLQ/ud7HmdEvEK9GtDZARURERNXjaAID2piQZpAgAACL9idjU0KaQY5FRERUHb4ZMJD0HBUilxyBWqOrdLugKUXusXVQJsVBV1wIm0b+cO85AQ6PPlblMe3kUsROD2cfAiIiMiq+GTCQ2TsSoammf0B29BLkJ+yEU7te8Ij8FyRSKbK2zEVxelKV39HoBMzekWiMcomIiMoxDBhASmYBjqVmV9lZUJ1xGapLR+EePgkevSfDpVN/+D63EHJXH+Qe/k+Vx9XqBBxLzUZqVoGxSiciImIYMIT18WmQSSVVblddPgFIpHDp1L/8M4ncFs4d+0J9809o8u9U+V2ZVIJ1p9l3gIiIjIdhwADiLmdVO4SwJPMqbDybQWpXse3ftklg+faqaHUC4pKzDFMoERFRJRgG6qlQrUFajqrafbSFOZA5ezz0uczZs3x7ddIUKijVmroXSUREVA2GgXq6rlCipuEYgqYEkNk89LlEbvvP9uq+D+CaQlnHComIiKrHMFBPJVUMJbyfRG4LaEsf+rwsBJSFgvqeh4iIqC4YBurJVl7zH6HM2RPawrsPfV7WPFDWXFDf8xAREdUF7zD15O/lhKrHEdxj69MSpTk3oVNX7FtQknFvtkJb35bVfl/y93mIiIiMgWGgnpzs5PCrYYZAxzZPAoIOBb/tK/9M0JSiMPEAbJsGQe7aqNrv+3k5wsmOq00TEZFx8A5jABFBPlgbf73K4YV2TYPg2OYp5B75CTpVLuQeTaFMPAhNXhZ8B7xR7bFlUgkiAn2MUTYREREArk1gECmZBei79Gi1+wiaEuQevbc2gba4ELY+/nDvMR4OLR+v8fix03siwMfFUOUSERFVwDBgIBNWxePkVUW1kw/pSyaVIKylF9ZGdTXYMYmIiB7EPgMGsnB4COTVTElcF3KpBAuHhxj0mERERA9iGDCQFp6OeCfiEYMec/7QYC5fTERERscwYCB5eXn4/t2JKDm7rZ5HutfM8E6/IDwb6lf/woiIiGrA0QQGoFQqMXjwYFy5cgWHD6/CpRIPzNmVBI1O0KsPgUTQQaspxZTObpgWEWDEiomIiP7BDoT1pFarMWTIEJw6dQqxsbHo2vVeZ7/0HBVm70jEsdRsyKSSakNB2fYnW3nhr80Lce3iWZw/fx6+vr4N9WMQEZEVYxioh9LSUowaNQoxMTHYu3cvevXq9dA+KZkFWB+fhrjkLKQpVBUWNZLg3oRCEYE+GN/NDwE+Lrh16xYee+wxBAcHY//+/ZDJZA314xARkZViGKgjrVaLiRMnYsuWLdi5cycGDhxY43eUag2uKZQo0ehgK5fC38up0pkFDx8+jD59+mD27NlYsGCBMconIiIqxzBQB4IgYOrUqVi1ahU2bdqEkSNHGvwcn376KWbPno09e/ZgwIABBj8+ERFRGYYBPQmCgBkzZuCrr77CmjVrMGnSJKOcR6fTYciQITh9+jTOnz8PPz+OLCAiIuNgGNDT3LlzMW/ePHz77bd45ZVXjHqunJwcdO7cGY0bN8bRo0dha2tr1PMREZF14jwDeli0aBHmzZuHzz77zOhBAAA8PT2xZcsW/Prrr3jnnXeMfj4iIrJODAO19P333+Odd97B+++/j/fee6/BzhsaGoolS5bg66+/xubNmxvsvEREZD3YTFAL69atw8SJE/Haa69h6dKlkEgMuwZBTQRBwNixY7F7926cPXsWQUFBDXp+IiKybAwDNdixYwdGjRqFSZMm4ccff4RUKs7LlIKCAjzxxBOQy+WIj4+HoyPXLCAiIsNgM0E1YmJi8Oyzz2LkyJFYsWKFaEEAAFxcXLB161ZcvXoVL7/8MpjhiIjIUBgGqnD06FEMHz4c/fv3x9q1a01iJsDg4GB8//33+O9//4tVq1aJXQ4REVkINhNUIiEhAX369METTzyB3bt3w97eXuySKpg6dSp++uknnD59Gp06dRK7HCIiMnMMAw9ITExEeHg42rZti5iYGDg7O4td0kOKi4sRFhaG/Px8nD17Fu7u7mKXREREZozNBPdJTk5G37594e/vj+joaJMMAgBgb2+PrVu3Ijs7Gy+88AL7DxARUb0wDPzt+vXriIyMhKenJ2JiYkz+abtly5b46aefsHPnTixZskTscoiIyIyxmQDArVu30LNnT+h0Ohw7dgxNmzYVu6Rae/fdd/HVV1/hyJEjePLJJ8Uuh4iIzJDVhwGFQoHw8HDk5ubi+PHj8Pf3F7skvZSWlqJ3797466+/8Ouvv8LHx0fskoiIyMxYVBhQqjW4plCiRKODrVwKfy8nONnJq9w/Ly8Pffr0QVpaGo4ePYo2bdo0YLWGk5GRgcceewwdOnTAvn37TGIYJBGRGPS9D9A9Zh8GUjILsD4+DXGXs5CWo8L9P4wEgJ+nIyKCfDCuqx9a+7qUb1Mqlejfvz8uXryIw4cPo2PHjg1euyEdPHgQ/fr1wwcffIB58+aJXQ4RUYOp632A/mG2YSA9R4XZOxJxLDUbMqkEWl3VP0bZ9h4B3lg4PAQ+TjIMGTIEJ0+eRGxsLLp169aAlRvPxx9/jI8++gh79+7F008/LXY5RERGVZ/7QAtPTul+P7MMAxsT0jBnVxI0OqHa//gPkkklkEsl8Ek/grOb/o29e/eiV69exiu0gel0OgwePBhnzpzB+fPn0aJFC7FLIiIyivreB+YNDcaYUD8jVmhezC4MLItLwaL9yXU/gCAAEgmGPAJ889IgwxVmIhQKBR577DE0a9YMR44cga2trdglEREZVL3vA3+b0S8Qr0a0NkBF5s+s5hnYmJBW/78Afy8//Mt1YFNCmgGqMi1eXl7YsmULzp07h/fee0/scoiIDMog94G/LdqfbJH3gbowmzcD6TkqRC45ArVGV+l2XUkR8uO3Q51xGSW3kqErLoTXwDfh3CGyymPayaWInR5ukW1H33zzDV5//XVs2bIFI0eOFLscIqJ6K7sPFKmUtbreZ+9eAuXFgw8dR+7ZHM3+9T0Ay74P6MNsxlvM3pEITTXtQjpVPvJO/AyZayPY+DwKdVpijcfU6ATM3pGItVFdDVmqSXj11Vdx7NgxTJ48GR06dEBgYKDYJRER1UvZfUCv673MBl4DXq/wkdTunxu/Jd8H9GEWYSAlswDHUrOr3Ufm7Inmr66FzNkD6lspuP3T9BqPq9UJOJaajdSsAgT4WNZwE4lEgpUrVyI0NBSjRo3CqVOn4Oho3cmXiMzX/fcBfa73EqkMzu0jqtxuyfcBfZhFn4H18WmQSSXV7iOR20Dm7KH3sWVSCdadtsw2I1dXV2zduhUpKSl49dVXq91XqdYgKSMP59PuIikjD0q1poGqJCKq2f33AX2v94JOC51aVeV2S74P1JZZvBmIu5yl19ARfWh1AuKSszAXwUY5vthCQkLw3Xff4fnnn8dTTz2FyZMnl2/jRB1EZC7qeh8QStVIXzIaQqkaUntnOLYLh0ev5yG1dSjfx9LvA7Vh8mGgUK1BWk7Vic4Q0hQqKNUai52yctKkSTh+/DimTZuGxx9/HJ4tWtc4UYcA4HqOCmvjr2PNqWucqIOIRFPX+4DM2QOu3f4Ptr6tAEGHoqu/ovDXaJRm/QXfsZ9CIv1n6nZLvw/UxOR/6usKJYw93EEAcE2hRHBTNyOfSTxff/01zp49ixEzFkHebWx5Z8yaknbZ9pNXFYhccoQTdRBRg6vrfcCj1/MVfu/ULhw2ns2Qe/S/UP15HE7twsu3WcN9oDom32egpIqhhOZ6HrE4ODhg+PvfQ9tlDNSlWr1ft2l1AtQaHWZuT8SyuBQjVUlE9DBDXp9dQocBEimKr/1u1POYG5MPA7byhimxoc4jlo0JaVh97u8RGZLqO2PWhBN1EFFDMuT1WWpjB6mDC7TFBUY9j7kx+WYCfy8nSACjNhVI/j6PpUrPUWHOrqRaTcwkCDooEw9BlXwSJZlXoSsugNzNF45te8Kt6whI5PemN/5oVxLCWnmzDwERGZ0h7wM6tQo6VT5kjhWbAyz9PlATk49BTnZy+Bn5huPn5WjRnUYenKijVJEOG59HK91XKFVDsWcptKo8uDw2AB59psC2SSDyjm9A5uY5KJuwsmyiDiIiY6vLfUDQlFQ6nDDv5EYAAhwe7Vzhc0u/D9TELH7yiCAfrI2/XmM7d/65X6ArVkJbmAMAKEo9A03BvVfjro8PgdT+4dQnk0oQEehj+KJNhL4TdUhkcviO/xL2zduWf+bSqT/kbr7IO74exdd/h4N/J07UQUQN6sH7QE3Xe11xIW7953U4tguHjVdzAEDxX7+i6MpZ2Ld8HA6B/yxdb+n3gdowizAwrqsf1py6VuN++fE7oM3PKv+9KvkkkHwSAOAcHFFpGNDqBIzvZrm948sm6tDqhFpN1CGR2VQIAmUcA7sj7/h6lGanw8G/E4B/JuqYO9R6x+YSUcN48D5Qm+u9Q8ATKL52HsqLByHodLDxaAL38IlwfWIEJJJ/Xoxb+n2gNswiDLT2dUGPAG+cvKqo9u1A81dW63VcmVSCsJZeFv1ka6gJm7TKuwAAmaPrP59xog4iaiAP3gdqc733HvJ2jftYw32gNky+z0CZhcNDIK9hSmJ9yaUSLBweYtBjmhJDTtiUH78NEjtHOLR8vMLnZRN1EBEZG+8DxmM2YaCFpyPmGfh19PyhwRbdG95QEzblndyM4mu/wSP8eUjtnStsK5uog4jI2HgfMB6zCQMAMCbUDzP6GWYp3nf6BeFZC59JzxATaCgvHUXu0bVw7tAPLp0HGu08RES1wfuAcZhFn4H7vRrRGt7OdpizKwkanaBXe7hMKoFcKsH8ocFW8RegvhNoFP11Htm7v4JDqy7w7D/NaOchItIH7wOGZ5ZX8TGhfoidHo6wll4AUOPyxmXbw1p6IXZ6uNX8BSibqKMu1BmXcWf7J7Br3Brez8yssKDH/ax9og4iEseD9wFBp612f2u9D9SW2b0ZKNPC0xFro7r+swxvchbSFBWX4YUg4BFvJ0QE+mB8Nz+r6y1aNlHHdT07EZZmpyNryzzI3XzQaNQcSG3sqtzX2ifqICLxlN0Hfti4C7NWRaNN7//DrfzSh5dj93K02vtAbZn9Vby1rwvmDg3GXARDqdbgmkKJEo0OH30wG3k3UnHk0AGxSxSVvhN1QCJB5uaPoCsuhGvXEShKTahwPBuPxrBrdm8eAk7UQUSm4MKxGHhcjcXJAz9UuA/YyqXw93LiA0stWNSfkJOdvHz5yc6P+uDH2P+JXJH49J2oAwC0+XcAALmH1zx0PKf2fcrDACfqICJTEBMTg6effhpAxfsA1Z5FhYH7BQYG4tatW8jPz4erq2vNX7BQdZmo45GZu2vchxN1EJEpuHLlCq5cuVIeBqhuzLIDYW0EBQUBAJKTk0WuRHycqIOILFVMTAzkcjkiIiLELsWsWWwYCAy8Nw6VYYATdRCR5YqJiUH37t2t+g2wIVhsGHB1dUXjxo1x+fJlsUsxCZyog4gsTUlJCQ4dOsQmAgOw2D4DwL2mAoaBf3CiDiKyJKdOnUJhYSHDgAFY7JsB4F5TAZsJKtJ3wiaJcG+qYU7UQUSmZv/+/fD29kbnzp3FLsXsWfybgQ0bNkAQBEgkhu1AZ85qM2GTBICzpBhZv8dh/7cfouOjvmKVS0RUqZiYGPTt2xdSqUU/1zYIiw8DSqUSN2/eRPPmzcUux+RUNWFT2UQdiswM+PuPwtlDj6FjVJTY5RIRlbtz5w5+/fVXvPbaa2KXYhEsOk5xREHtlU3U8ZifB4Kbut2bytjPD/369cOqVavELo+IqIIDBw5AEAT069dP7FIsgkWHgUcffRRyuZydCOth8uTJOHXqFC5duiR2KURE5WJiYtChQwc0adJE7FIsgkWHARsbG7Rq1YphoB6GDRsGLy8vrF5d88yFREQNQRAE7N+/n6MIDMiiwwDAEQX1ZWdnh/Hjx+O///0vSktLxS6HiAgXLlzA7du3GQYMyOLDAOcaqL+oqChkZWVh9+6a1ywgIjK2mJgYODo64qmnnhK7FIthFWHg2rVrUKvVYpditkJCQhAaGsqOhERkEmJiYtCrVy/Y2dmJXYrFsPgwEBgYCJ1OhytXrohdilmbPHky9u7di4yMDLFLISIrplQqcfz4cTYRGJjFh4Gy1QvZVFA/zz33HOzs7PDTTz+JXQoRWbHDhw+jpKSEYcDALD4M+Pj4wM3NjWGgntzc3DBy5EisXr0aglD7NQ2IiAwpJiYGfn5+5fPIkGFYfBiQSCQcUWAgUVFRSE1NxdGjR8UuhYisVExMDJ5++mlOMW9gFh8GAI4oMJSePXsiICCAHQmJSBTXrl1DcnIymwiMgGGAak0ikeCFF17A1q1bkZeXJ3Y5RGRlYmJiIJPJ0KdPH7FLsThWEQYCAwOhUCigUCjELsXsTZo0CWq1Ghs3bhS7FCKyMvv370fXrl3h7u4udikWxyrCQNmIAvYbqL9mzZphwIABbCogogal0Whw8OBBNhEYiVWEgYCAAAAcXmgoUVFRSEhIQGJiotilEJGViI+PR15eHsOAkVhFGHByckKLFi34ZsBABg8eDB8fH74dIKIGExMTA09PT3Tp0kXsUiySVYQBgJ0IDcnGxgYTJkzAunXrOM0zETWImJgYREZGQiaTiVqHUq1BUkYezqfdRVJGHpRqjaj1GIpc7AIaSmBgII4cOSJ2GRYjKioKixcvxq5duzBq1CixyyEiC6ZQKJCQkICpU6eKcv6UzAKsj09D3OUspOWocP+0axIAfp6OiAjywbiufmjt6yJKjfUlEaxkOrmvv/4a7777LpRKpejJ0lKEhYXB1dUV+/btE7sUIrJgmzZtwpgxY5Ceno7mzZs32HnTc1SYvSMRx1KzIZNKoNVVfbss294jwBsLh4eghadjg9VpCFbVTKBWq5GWliZ2KRYjKioK+/fv558pERlVTEwMgoODGzQIbExIQ+SSIzh59d6Q9OqCwP3bT15VIHLJEWxMMK/rotWEgbJ5rNlvwHBGjx4NR0dHLl5EREYjCEL5FMQNZVlcCmZuT4Rao6sxBDxIqxOg1ugwc3silsWlGKlCw7OaMODn5wc7OzuOKDAgFxcXjB49GqtXr4ZOpxO7HCKyQElJScjIyGiwMLAxIQ2L9hvmPrFofzI2mckbAqsJAzKZDK1bt+abAQOLiorCtWvXEBcXJ3YpRGSBYmJiYG9vjx49ehj9XOk5KszZlQRdSRFyj61H5qaPkL50DK5/NhiFF2Kr/a6g1SDjx5dx/bPByIvfXv75R7uSkJ6jMnbp9WY1YQC411TAMGBYYWFhCAoK4pwDRGQUMTExCA8Ph4ODg9HPNXtHIjQ6ATpVPvJO/IxSRTpsfB6t1XcLzv0CTf6dhz7X6ATM3mH6E7RZVRgICgpiM4GBSSQSREVFYfv27bh7967Y5RCRBVGpVDh69Cj69etn9HOlZBbgWGo2tDoBMmdPNH91LZq/8h94REyu8btaZS5yT2yEa7f/e3ibTsCx1GykZhUYo2yDsbowkJ6eDqVSKXYpFmXixInQaDTYsGGD2KUQkQU5evQo1Gp1g/QXWB+fBplUAgCQyG0gc/ao9XfvHl4DG89mcAqOqHS7TCrButOm3XfAqsJA2YiC1NRUkSuxLL6+vhg8eDCbCojIoGJiYtCsWTO0a9fO6OeKu5yl98gBAFBnXIby4iF4Rk6BBJJK99HqBMQlZ9W3RKOyqjBQtnoh+w0YXlRUFM6fP4/z58+LXQoRWYiyIYUSSeU3WUMpVGuQVodOfoIgIOfAD3Bs2wN2zdpWu2+aQmXSUxdbVRjw9PSEt7c3w4ARDBgwAE2aNOHbASIyiPT0dFy6dKlBmgiuK5Soy1S8ysRYlN65Do9ez9e4rwDgmsJ0m6itKgwA95oK2InQ8ORyOSZNmoT169ejqKhI7HKIyMzt378fUqkUkZGRRj9XiUb/eVJ0ahXuHvkJrl1HQO7ayGjnaShWFwa4eqHxTJ48Gbm5udi5c6fYpRCRmYuJiUFoaCg8PT2Nfi5buf63wvz47YBWA8e2PaDJzbz3v4JsAICuuBCa3EwI2tJ6n6ehWM2qhWUCAwOxbds2CIJg9HYoa9O6dWv06NEDq1atwnPPPSd2OURkprRaLWJjY/Haa681yPn8vZwgAfRqKtDk34GuuBC3Vr7y0Lb8U5uRf2ozmrzwNWx9WwK4t7qhv5eTQeo1BqsLA0FBQcjPz0dWVhZ8fX3FLsfiREVF4fnnn8dff/2FRx+t3WQdRET3S0hIwN27dxtsCmInOzn8PB1xXY9OhC5dhsAxsFuFz7SqPOTsWwankEg4tu4Kuds/9xg/L0c42ZnuLdd0KzOS+0cUMAwY3siRI/Haa6/hP//5D+bPny92OURkhmJiYuDm5oYnnniiQc537do1CBkXIdg+Aon03hL3+ed+ga5YCW1hDgCgKPVMeTOA6+NDYNc4AGgcUOE4mtxMAICNtx8cA7uXfy6TShAR6NMQP0qdmW4DhpG0atUKUqmU/QaMxMnJCc899xzWrFkDrVYrdjlEZIZiYmIQGRkJudy4z6upqamYPHkyWrdujT93rywPAgCQH78DecfWofD8HgCAKvkk8o6tQ96xddAVF+p1Hq1OwPhufgat3dCs7s2AnZ0d/P39OaLAiKKiorBixQrExsY26LKjRGT+7t69i/j4eHz//fdGO8eff/6JTz75BBs2bICPjw8+//xzTJ06FS9tvIiTVxXQ6gQ0f2W13seVu/vikZm7K3wmk0oQ1tILAT4uhirfKKzuzQDAEQXGFhoaiuDgYM45QER6O3jwIHQ6nVEeJBITE/Hss8+iXbt2iIuLw9KlS3H16lW89dZbcHJywsLhIZBLDduxXC6VYOHwEIMe0xisMgxw9ULjKlu8aOfOncjOzha7HCIyYUq1BkkZeTifdhdJGXmIjolFmzZt4OdnuNfqv/76K4YPH44OHTogPj4e3333Ha5cuYLXXnutwmqILTwdMW9osMHOCwDzhwajhaejQY9pDBJBEOoy8ZJZ++677/D6669DpVLBxsZG7HIsUnZ2Npo2bYovvvgCb775ptjlEJEJSckswPr4NMRdzkJajqrCkD5BEOAsFGHUk+0wrqsfWvvW/fX66dOnsWDBAuzZswcBAQGYPXs2xo8fX+N1f1lcChbtr39T8jv9gjAtIqDmHU2AVYaBQ4cOoU+fPrh8+XL54kVkeKNGjcLly5fx+++/c04HIkJ6jgqzdyTiWGo2ZFJJtQsDlW3vEeCNhcND9Hq6Pnr0KBYsWIDY2Fi0bdsW77//Pp599lm9OiRuTEjDnF1J0OgEvRYwkkklkEslmD80GM+GmnanwftZbTMBwAWLjC0qKgqJiYk4e/as2KUQkcg2JqQhcskRnLyqAIAab7Bl209eVSByyRFsTKh+CWBBEHDw4EH06tUL4eHhyMzMxObNm5GYmIhx48bpPTJhTKgfYqeHI6ylFwCUL29clbLtYS29EDs93KyCAGCFowkAoFmzZnBycuKIAiPr27cvmjdvjlWrViE0NFTscohIJPV57a79+8l85vZEZBeq8WpE6wrbBUHAvn37sGDBApw6dQqdO3fGjh07MHToUEil9XvebeHpiLVRXf9p1kjOQpqiYrOGBPcmFIoI9MH4bn4mP2qgKlbZTAAAnTt3RpcuXbBixQqxS7FoH374Ib7++mvcunULjo6m34mGiAxrY0IaZm5PNNjxPh8RgmdD/SAIAnbt2oWPP/4YZ8+eRbdu3fDhhx9iwIABRm2WVKo1uKZQokSjg61cCn8vJ5OeWbC2rDYMjBkzBrdu3cKRI0fELsWiXb16Fa1atcJPP/2EiRMnil0OETWg9BwVIpccQZFKifz47VBnXEbJrWToigvhNfBNOHeouCJhwW/7oEw6jFLFDejUhZA5e8HeLwTuTz4Hufu9GWPt5FK8FaTE8i8X4MKFC+jZsyc+/PBD9OnTh32T6sH840wdBQUF4ciJ00jKyLO4hGdKWrZsid69e2P16tUMA0RWZvaORGh0AnSqfOSd+Bky10aw8XkU6rTK3xSUZF6F3M0XjgFPQGrvDE1eJgp+j0FR6hk0mfwN5C5eUJeU4qPdlxDSqBEOHz6M8PDwBv6pLJPVvRkoa/vZde4KFGpphSQpAeDn6YiIIJ96D2mhf2zYsAHjxo1DSkoKAgLMY5gNEdVPSmYB+i49CgAQNKXQFRdC5uwB9a0U3P5peqVvBiqjvp2K22vehHv4JLh1H1X+eez0nmbbPm+KrGY0QXqOChNWxaPv0qNYG38dOSWyh14pCQCu56iwNv46+i49igmr4pGuxypWVLnhw4fDzc0Nq1frP70nEZmn9fFp5T3sJXIbyJw96nQcudu9BX50amX5ZzKpBOtOVz+6gPRjFWHA2ENaqHoODg4YN24c1qxZA41GI3Y5RNQA4i5n6TU+/37aonxolblQ30qBInopAMD+kY7/bNcJiEvOMkSZ9DeLbyA35pAWqr2oqCgsX74cMTExGDRokNjlEJERFao1SKvHW9UbyyYB2lIAgNTBFR6RU+Hw6GMV9klTqKBUa9jPy0As+s3AxoQ0g0wpCQCL9idjE98Q1Fnnzp3RqVMnLl5EZAWuK5SoT2c039Hz4DNqLjx6R0Hu2ghCafFD+wgArimUD3+Z6sRiI1V6jgpzdiVVuk19KxnKxIMoTkuEJi8TUgdX2DUNgnvPCbDxbFblMT/alYSwVt5mseiEKYqKisL06dORmZkJX19fscshIiMp0ejq9X37RzoAABxadYFD6264tWoaJLb2cH18iEHPQ/+w2DcDZUNaKpN/eitUl0/C/pGO8Ij8F5w7Po3i9Iu49Z83UHLnWpXH1OgEzN5huMkzrM3YsWMhk8mwdu1asUshIiOylRvu1mLj0QS2vi2hTDps1PNYO4v8k0zJLMCx1OwqO6+4hA5Hs1dWw7PvVLh0fBruT45B43GfQ9BpkX96a5XH1eoEHEvNRmpWgbFKt2ienp4YPnw4Vq1aBSsb0UpkVfy9nGDI6X90pSUQ1BX7IEj+Pg8ZhkWGgfuHtFTGvnlbSGQVl7C08WwGW28/lGanV3tsDmmpn6ioKPz55584ffq02KUQkZE42cnhp2dzqqDTQltc+NDn6ozLKL1zDbaNK85R4uflyM6DBmSRf5J1GdIiCAK0qlzYeFe/0lTZkJa5CK5PiVard+/e8Pf3x6pVq9C9e3exyyEiI4kI8sHa+Ovl1+L8c79AV6yEtjAHAFCUegaagmwA+LsvgICb3z4Px7Y9YOvtB4mNPUrvXENhYiykdk5we3JM+bFlUgkiAn0a/GeyZBYXBuo6pEWZdBjaAgXcnxpX474c0lJ3UqkUL7zwAr788kssXboUzs7OYpdEREYwrqsf1py6Vv77/Pgd0Ob/MzeAKvkkkHwSAOAcHAGZiyecO/ZD8fULUF0+AaG0BDJnTzi1DYdb2LPlaxMA9x7KxnczryWCTZ3FTUeclJGHQd8c1+s7pYp03Prv27D19oPvuM8hkcpq/E70a08huKlbXcu0amlpafD398fKlSsxefJkscshIiOZsCoeJ68q6jz5UGVkUgnCWnphbVRXgx2TLLDPgL5DTbSFd5G1ZR6kdk7wfmZWrYJAXc5D//Dz80Pfvn0rzDmgVGuQlJGH82l3kZSRB6WaMxUSmbuFw0Mgr6b/Vl3IpRIsHB5i0GOSBTYT6DPURFesRObmOdAVK+E7/nPIXbyMch56WFRUFMa/MgOv/3Qcv2eVIi1HVWGSEi4aRWT+Wng6Yt7QYMzcbrgh2fOHBnOuFyOwuDtabYe0CJoSZG2dD83dm/AZ9RFsa+g4eD8Oaamf9BwV/pfXAk2nfIdfLt3F9QeCAMBFo4gsxZhQP7QrTTXIsd7pF4RnQ9lXwBgsLgzUZkiLoNPizs7Poc74E42emQm7Zm31OgeHtNRd2aJRp6/dBQAIkur/CnLRKCLz9vPPP2Pv4jfRz/0O7OTSaod9V0YmlcBOLsXnI0IwLYJLoBuLRd7RHhzS8qC7h1ahKDUeDgFPQFtUiMKLcRW2O7ePqPLYHNJSd1w0isi6JCYm4sUXX8T48ePxw7uTcONuEWbvSMSx1GzIpJJqOxaWbQ9r6YWFw0PYNGBkFjeaALg3A2HfpUer3H57/Uyo0y9Wuf2RmburPX7s9J4I8GEbtj42JqQZtN3w8xEhfF1IZMJyc3PRpUsXODs74+TJk3B0/OdmnpJZgPXxaYhLzkKaopL+Ql6OiAj0wfhufrzWNhCLDAMAh7SYkvQcFSKXHEGRSon8+O1QZ1xGya1k6IoL4TXwTTh3iKywvzrjMgoTD6Ik4/K9tSJ02ocCmp1citjp4XxaIDJBOp0Ow4YNw/Hjx3H27Fm0atWqyn2Vag2uKZQo0ehgK5fC38uJzbAisLg+A2U4pMV0lC0apVPlI+/EzyhVpMPG59Eq9y+6chaFv+8HJBLI3RtXug8XjSIyXZ988gmio6OxYcOGaoMAcK+fV3BTNzzm54Hgpm4MAiKx2DBQNqTFkDikRX/3Lxolc/ZE81fXovkr/4FHRNWTDbl0HogW0zehyfNL4eDfqdJ9uGgUkWnau3cv5syZg7lz52LAgAFil0O1ZLFhALg3pGVGv8B6HuVeM0OkTxHbqOvg/kWjJHIbyJw9avyOzMkDUhu7mvfjolFEJuXq1asYO3YsBg0ahA8++EDsckgPFh0GAODViNb4bERIPYa0yNAm7xz++95YxMXF1fwlqqAui0bVVtmiUUQkPpVKhREjRsDLywtr166FVGrxtxeLYhX/tcaE+iF2ejjCWt6bYbCmUFC2PaylF2Knh+OXf89Cr169MHz4cCQlJRm9XktR10Wj9FG2aBQRiUcQBLz00ktISUnB9u3b4e7uLnZJpCer6anRwtMRa6O61nlIy9atW9GjRw8MGDAAp0+fRtOmTRv8ZzA31xXKh2YWNDQBwDWFkotGEYno22+/xdq1a7FhwwZ06NBB7HKoDqwmDJRp7euCuUODMRfBeg1pcXV1RXR0NLp164bBgwfjyJEjcHHh+NfqNNRiTlw0ikg8J06cwPTp0/HGG2/gueeeE7scqiOraCaoir5DWpo3b449e/YgNTUVo0ePhkbD19PVaajFnLhoFJE4bt++jVGjRqF79+748ssvxS6H6oFXUT116NAB27ZtQ2xsLF5++WVY6JxNBlHbRaPqg4tGEYmjtLQUo0aNAgBs3rwZNjY2IldE9cEwUAd9+/bFjz/+iJUrV+LTTz8VuxyTVZtFo+qLi0YRiWPGjBmIj4/H1q1b0bhx5ZODkfngVbSOnn/+eVy/fh3vv/8+/Pz8MH78eLFLMkkPLhqVf+4X6IqV0BbmAACKUs9AU5ANAHB9fAik9k7Q5GWh8OIhAID69r2lT3NPbAQAyN184Ny+NwAuGkUklg0bNuDrr7/GsmXLEBYWJnY5ZAAWuzZBQxAEAZMnT8b69esRExODiIiqVzu0Vg8uGnVj+WRo8yufG6DZS6sgd/dF8fULyPx5dqX72LVoj8bjPiv/PReNImpYFy5cQLdu3TBy5Ej89NNPkEiM3RhIDYFhoJ5KS0sxaNAgnDlzBidOnEBwsGGnQLYEXDSKyDLcvXsXoaGhcHFxwYkTJyqsREjmjX0G6snGxgZbt27FI488goEDB+LWrVtil2RyFg4PgRSCwTpbCoIAKQQuGkXUgHQ6HcaPH4+cnBxs376dQcDCMAwYQNkcBFqtFoMGDUJhYaHYJZmUi/FHoIj5zmCvEyUSCbL2LMPeresMcjwiqtmCBQuwd+9ebNiwAY8+WvWqo2SeGAYMhHMQVG7Hjh0YNmwYejST4c3e1S9lWltv9QnA+LCWmDp1Kl5//XX+WRMZ2Z49ezBv3jzMmzcP/fv3F7scMgL2GTCwAwcOYODAgXjhhRfwww8/WHXnmo0bN2L8+PEYMWIE1q9fDxsbG2xMSMOcXUnQ6AS9+hDIpBLIpRLMHxpcvnrkd999h9deew0RERHYvHkzPDxqXhGRiPRz5coVdOnSBT169MDOnTu5AJGFYhgwgjVr1uCFF17AwoULMWvWLLHLEcV//vMfREVFYcKECVi1ahXk8n9GsabnqDB7RyKOpWZDJpVUGwrKtvcI8MbC4SFo8cC8BXFxcRg5ciS8vLzwyy+/ICgoyGg/E5G1UalU6N69O1QqFRISErgAkQVjGDCSuXPnYt68eVi3bh3GjRsndjkNavny5Zg2bRqmTp2K5cuXV/kkUddFox505coVDBkyBBkZGdi0aROefvppw/5ARFZIEARMmDABO3bsQHx8PNq3by92SWRMAhmFTqcTJk2aJNjY2AiHDh0Su5wGs3jxYgGA8MYbbwg6na7W3yssLhUu3swVfr2eI1y8mSsUFpfqdd7c3Fxh4MCBglQqFZYsWaLXuYnoYV9//bUAQPj555/FLoUaAMOAEanVaiEyMlJwc3MTLl68KHY5RrdgwQIBgDBr1ixRbsYajUaYMWOGAECIiooS1Gp1g9dAZAmOHTsmyOVyYfr06WKXQg2EzQRGlpeXhx49eiAvLw+nT59GkyZNxC7J4ARBwAcffICFCxdiwYIF+OCDD0St56effsK//vUvPPHEE9i2bRt8fDhlMVFt3bp1C507d0ZgYCBiY2O5AJGVYBhoADdu3EC3bt3g4+ODo0ePwtnZWeySDEYQBLz11ltYunQpFi1ahLffflvskgAAp06dwjPPPAMHBwfs2rULHTp0ELskIpNXUlKC3r1746+//sKvv/4KX19fsUuiBsIxIg2gefPmiI6Otrg5CHQ6HV5++WUsXboU3377rckEAQDo3r07EhIS4OHhgbCwMOzcuVPskohM3owZM3DmzBls3bqVQcDKMAw0kI4dO2Lr1q04cOAAXnnlFYNNzSsWjUaDF154AStWrMDq1avxyiuviF3SQ/z8/HD8+HH0798fw4cPx8KFC83+z53IWNatW4dvvvkGS5cuRffu3cUuhxqaeN0VrNPq1asFAMLChQvFLqXOSkpKhNGjRwsymUzYsGGD2OXUSKvVCnPmzBEACM8995ygUqnELonIpPz222+Cg4ODMGnSJI7EsVLsMyCCsjkI1q9fj7Fjx4pdjl7UajVGjx6NvXv3YtOmTRg+fLjYJdXali1bMGnSJAQHB2Pnzp1o1qyZ2CURie7u3bvo0qUL3NzccOLECTg4OIhdEomAYUAEgiDghRdewM8//4z9+/cjPDxc7JJqRaVSYfjw4Th69Ci2b9+OAQMGiF2S3n799VcMGzYMWq0WO3fuxBNPPCF2SUSi0el0GDJkCE6dOoVz585xASIrxj4DIpBIJFixYgV69uyJZ555Bn/88YfYJdWooKAAgwYNwvHjxxEdHW2WQQAAOnfujISEBDzyyCPo2bMnNmzYIHZJRKKZP38+9u7di59//plBwMoxDIjE1tYWW7duRYsWLTBw4EDcunVL7JKqlJubi6effhrnzp3D/v370bt3b7FLqpfGjRsjLi4Oo0ePxrhx4/D+++9Dp9OJXRZRg9q9ezfmzZuHBQsWcApvYjOB2NLT09GtWzc0btwYR44cMbk5CBQKBfr164e//voLMTExCA0NFbskgxEEAYsWLcJ7772HoUOHYu3atXBxqXoNBCJLkZqaii5duiA8PBw7duzgSoTEMGAKfv/9d/To0QM9evTA//73vwor/IkpMzMTkZGRyMzMxIEDB9CxY0exSzKK3bt3Y+zYsfD398euXbvg7+8vdklERqNUKtG9e3cUFxcjISEBbm5uYpdEJoBx0ASUzUEQExODadOmmcRY+Bs3bqBnz55QKBQ4cuSIxQYBABg8eDBOnToFpVKJ0NBQHD16VOySiIxCEAT861//wpUrV7B9+3YGASrHMGAi+vXrhxUrVmDFihX4/PPPRa3l2rVr6NmzJ4qLi3H06FG0bdtW1HoaQnBwcPkyrZGRkVi1apXYJREZ3Ndff40NGzZg9erVXJKYKmAYMCGTJ0/GRx99hFmzZonWyz0lJQU9evSAVCrF0aNHERAQIEodYvD29sb+/fsRFRWFF198EW+++abFTB1NdOzYMcyYMQNvvfUWnn32WbHLIRPDPgMmRhAEPP/889i4cWODz0GQlJSEyMhIuLu74+DBg2jatGmDndvULF++HK+//jr69OmDTZs2wd3dXeySiOosIyMDnTt3Rps2bRAbG2sy/ZLIdDAMmKCSkhIMHDgQ586dw8mTJxvkNf358+fRr18/NG3aFAcOHOCyvwAOHjyIUaNGoVGjRvjll18QGBgodklEeispKUFERASuX7+Oc+fOcQEiqhSbCUyQra0ttm3bhubNm2PAgAG4ffu2Uc8XHx+P3r17w9/fH3FxcQwCf+vTpw/i4+MhlUrRtWtXHDhwQOySiPT21ltvISEhgSsRUrUYBkyUm5sb9uzZg9LSUgwaNAiFhYVGOc/Ro0cRGRmJ4OBgxMbGwtPT0yjnMVetW7fG6dOn0a1bNwwYMABff/21SYz2sBZKtQZJGXk4n3YXSRl5UKrZh0Mfa9euxbfffouvv/4a3bp1E7scMmFsJjBxZXMQ9OzZEzt37qyyrU+p1uCaQokSjQ62cin8vZzgZFd9u2BsbCyGDh2K7t2743//+5/JTXhkSrRaLd577z0sXrwYU6ZMwbJly2Brayt2WRYpJbMA6+PTEHc5C2k5Ktx/gZIA8PN0RESQD8Z19UNrX04SVZXffvsN3bt3x5gxY7B69WpIJBKxSyITxjBgBmJiYjBo0CBMmTIFy5cvL/9HXZ+L5u7duzFy5Ej07t0b27Zt40pltfSf//wHU6dORffu3bF161Y0atRI7JIsRnqOCrN3JOJYajZkUgm0uqovTWXbewR4Y+HwELTwdGzASk1fTk4OunTpAg8PDxw/fpz/vqlGDANmYtWqVXjxxRfx2WefYeyU1+p10dy6dSuee+45DBkyBD///DPs7Owa8CcxfydOnMCIESPg6OiIXbt2ISQkROySzN7GhDTM2ZUEjU6o9u/zg2RSCeRSCeYNDcaYUD8jVmg+tFotBg8ejDNnzuDcuXOcUZNqhWHAjHz00UdY8r94+A56DYJEWqeL5gCfAnzz5nN49tln8dNPP8HGxsaIFVuu69evY9iwYbhy5QrWr1+PoUOHil2S2VoWl4JF+5PrfZwZ/QLxakRrA1Rk3j766CN8/PHH2LdvH/r16yd2OWQmGAbMyDeHUrD4QDIEQahj+58AQIKWhRdxYOkMyGQyQ5doVQoLCzFx4kTs3LkTn3zyCWbOnMl2WT1tTEjDzO2JBjve5yNC8KwVvyH45ZdfMHToUHzyySeYPXu22OWQGWEYMBNVXTRL7lxH3vENKLmdCq0yFxIbO9h4tYBr1xFwbN21yuNZ+0XTUHQ6HebOnYsFCxZg7NixWLlyJdtnayk9R4XIJUeg1lS+fLT6dipyj/wX6puXAAB2TdvAI+IF2Pq2rPKYdnIpYqeHW1Qfgtp2Di5bibBXr17Yvn07VyIkvTAMmIHqLppFVxKQf/YX2DVrA5mzJ4RSNVSXT0J9Iwme/V+FS6f+lR7TEi+aYtq0aROef/55hISEYOfOnVY9e2NtTVgVj5NXFZU2d6lvpyJz3buQuXjDpVN/CBBQ8Ose6IoL0GTiV7Dxal7pMWVSCcJaemFtVNVB2Bzo2zlYqVSiW7duKCkpwZkzZ7gAEemNYcAMVHfRrIyg0+LWmjchaErR7F/fV7qPpVw0Tcm5c+cwbNgwCIKA//3vf+jSpUutv1uXoaHmLCWzAH2XVr06ZNaWuVDf/BNNp66AzMEVAKApzEHGiqlw8H8MjUZU/wo8dnpPBPiY37DDuoyoeCrAG8rDP+LAjo04c+YM2rVr14AVk6Ww3KuNhUjJLMCx1Gy9viORyiB38Yb6dkqV+2h1Ao6lZiM1q8AsL5qm6PHHH0dCQgKeeeYZ9OjRA2vWrKl2QRhrHk+/Pj6t2ptdcXoSHFo+Xh4EAEDu7An7Fu2hunIGupIiSG0rb46RSSVYdzoNc4cGG6V2Y7l/RAWAGsN/2fYTqXeg9R2Il74cyCBAdcZGJRNXdtGsia6kGFpVHkrv3kL+mZ0ounoO9o90rPY7ZRdNMpwmTZrgyJEj+L//+z+MGTMGH374IXS6is076TkqTFgVj75Lj2Jt/HVcfyAIAPe6el7PUWFt/HX0XXoUE1bFIz1H1WA/h7HFXc6q9mYnaEshkT88qZPExg7QalB653qV39XqBMQlZxmkzoayLC4FM7cnQq3R6TVKCAAESCCV22Jrmh2WxVX9AEBUHb4ZMHE1XTTL3D20EoW/7bv3G4kUjoHd4dnv5Wq/U3bRnAvzeoIydfb29li7di3at2+P2bNnIykpCf/973/h7Oxc56e/k1cViFxyxCLG0xeqNUirIdjYeDaHOuMyBJ0WEum9US+CthTqjMsAAE2BAtXNjpGmUEGp1phFU8vGhLQqh1aW5txE7rF1UN/4A7qiQshcG8GpXThcuw6H1Mb+nx3/HsWyaH8yGjnbsXMw6c30/6VYsdpcNMu4hg6DY5unoC1QQPXncQiCDtCW1vg9c7pomhOJRIKZM2ciODgYY8eOxZNPPolRc1ZiZULdnli1f0/GM3N7IrIL1WY9nv66QvnQm5AHuXQeiJyY5VDs+Rqu3f4PEHTIO7kJ2sK7AABBU1Lt9wUA1xRKBDc17Y506TkqzNmVVOk2Tf4d3P7pLUjsnODSeTCkDi5Q3/wTecfXo+R2KnxGfljp9z7alYSwVt7sHEx6YTOBCavNRbOMjVcLOPh3gnNIH/iMmgOhpBhZW+fXuKhO2UWTjGPIkCE4deoU8ryD6xwEHrRofzI2JZhv805JFUMJ7+fy2EC4dh8N5R9HcGvlK7i16lVo7t6+FwwASG3tazgCkPTnZdy8eRPFxcX1rtlYZu9ILH9L9CDlxTjo1Er4jJoDt+6j4NKpP7wHvQmn9r1RlBoPbXHli5dpdAJm7zDc3A1kHfg4aMJqc9GsimObJ5Gzbxk0OTerHIZliPNQzdyatoRd2HgUKZXIP7MD6ozLKLmVDF1xIbwGvgnnDpEPfac0Ox05B3+E+sYfkMjkcGgVCo8+L0LmeO9J15Sf/nQ6HbKysnDjxg3cvHmz/P/Lfp1eKAC9367xOB7hE+HadQRK71yH1M4Jtj7+uHvkJwCA3LNZjd8f/9wYlGb9BQBwcnKCt7c3vL294eXlVf7rqn7v5eUFe/uaA0d91NQ5WFdy762gzMm9wucyZ09AIoVEWvnlm52DqS4YBkyYrbzuL26EUjUAQKeu+am/Puehmt17+gN0RQXIO/EzZK6NYOPzKNRplT+9afKzcXv9e5DaOcE9fCKEkmLkn9mOkjvX0GTSV5DIbMqf/hp6aGhxcTEyMjIqvcmX/f+tW7eg0fyz1LCNjQ2aNm2KZs2aoXnz5gju6Iddf8+GWROZvTNkLf7p01J87TfIXLxrDLgAELvjZxTmKpCdnY3s7GwoFP/8+saNG/j999/Lf6/Vah/6vrOzc62Cw/2/1medj5pGVNj7hSD/9FYo9nwN9x7j/m4muISC83vg8viQat+OmOuIChIPw4AJ8/dyggSotqlAq8x96MlB0GqgvHgIErkdbLyr70gk+fs8ZBz3P/3JnD3R/NW1kDl7QH0rBbd/ml7pd/JObYZQqobv80shd/MBANg2DUTWxg9QmHgQLp36G/zpTxAE5OXlPXRjf/Bmn51d8UnWxcUFzZs3R7NmzRAYGIjevXujWbNm5Tf+Zs2aoVGjRg/Nhvf7l3G4rufoCOWloyi5lQKPiMmQSKoPsI94OaJnWO2CkiAIyM/PLw8GDwaHst+np6fj/Pnz5b+vKkDU9g3EgT9uV9uB1KHl43DrMR75p7bgVmp8+eeuYc/Co+eEan8mdg4mfTEMmDAnOzn8PB2rvWgq9i2DUKKCXYv2kLl4QVt4F8o/DkOjuAGP3lFVjsUu4+flyM6DRnT/059EbgOZs0eN31FdPgmHgNDyIAAADv6dIPdsBtWlY+WzStb26U+r1SIzM7PSp/j7P1OpKv498/X1Lb+xh4WFVbjBl/3P1dW1irNWLyLIB2vjr1c9z0DaReSd+Bn2jz4GqYMrSjL+ROGFWNi3fBwuocOqPbZMKkFEoE+1+9xPIpHAzc0Nbm5uaNWqVa2+UxaeqgoOZb9OS0urNEBIbB3QYvrmGteykLv5wq5FMByDwiBzcIXqSgLyT26GzMkdro8Pqfa77BxM+uDfEhNX00XTqW0PFF44gILze6ArKoDU1gG2jQPg0euFatcmAPS/aJL+ajs0tIymIBs6VS5sGwc8tM2uSSCKrpwt/71WJ+DQ5UxMuGJf6VP8/a/t73+KtbGxqfD0/thjj1W4yTdv3hxNmjSBre3D4/wNZVxXP6w5da3K7TIXL0AqRX78duhKiiB394V7zwlwfeKZ8qGGVdHqBIzvZtyhdRKJBO7u7nB3d0dAwMP/rSqj0+nK30CcvXIbMw/nVbu/8o8jyNm3DE3/9QPkrt4AAMegMEAQkHt4DZzahVeYlOlB5jKigkwDw4CJq+mi6dQuHE7twut07Ia4aFozfYaGlikbOidz9nxom8zZA7riAgiaUkjk95aevq5QoXXb9hBK7/WYd3V1Lb+ht2nTBpGRkQ+9tvf29hZ9EZvWvi7oEeBd5TTbNh5N4PvsAr2PWzbNtil2nJNKpeUBosDWCzh8str9C37dA1vfluVBoIxjwBNQJsaiJPMqHPw7VXsMdg6m2mIYMHE1XTTrypQvmpZCn6GhZQTNvY6fEpnNQ9skMtvyfcrCgEQiwarNu9C9TQs0a9YMLi7m899z4fAQRC45YtC/13KpBAuHhxjseMZSm067WlUupPbOD30u6P5+y6N7uM9CXc5DBHCeAbOwcHgI5LWYklgf5nLRNGd1eSqTyO/1RhcqmTBK0JZU2KdMp85d0KZNG7MKAgDQwtMR8wzc233+0GCTHG75oLLOwdWx8WiKkswrKM25WeFz5R9HAIkUNo38q/0+OweTPhgGzIA1XzTNWV2eyso6GGoLcx7api28C6m9S/lbgfqcx1SMCfXDjH6BBjnWO/2CzGYa3rLOwdVx7fp/gE6H2+veQ+6Jn1HwazQyN89BUcppOHeIhNzFq9rvs3Mw6cN8ryJWxlovmuasNk9/D5K7eEPq6IaS26kPbVPfSoat76MVPrOEp79XI1rjsxEhsJNLa7Uo1/1kUgns5FJ8PiIE0yJq15HPVEQE+VT789r7tUfjCV/CtnErFP66BzmxP0KTexvuPSfC8+lp1R6bnYNJX4yNZuTViNbwdrYrX+hGn7ZWQauBrY0MHz/TgUGggdRmaGhlHIPCoEw8BE3+HchdGwEAiq79Bk3OTbg+MKzOUp7+xoT64clW3pi9IxHHUrOrnYwHQPn2sJZeWDg8xCzfctXUORgA7JoGwXf0PL2Pzc7BpC/zv4pYmbpeNB0Lb0J7eh1GzD3SgNXSg0ND88/9Al2xsrwZoCj1DDQF9ybycX18CKT2TnDrPhqqP08gc8NsuHQZCqG0CPnx22HTyB/OIX3Lj21pT38tPB2xNqorUjILsD4+DXHJWUhTVFzeWYJ7ASgi0Afju/mZdQdYdg4mUyIRalrJhkyWPhdN1e2/0KlTJ/z73//Ga6+9JlbJViclswB9lx4t//2N5ZOhza98waJmL62C3N0XAFBy5zruHlp5b20CqRwOAaHw6B0FmVPFSYtip/e06Iu+Uq3BNYUSJRodbOVS+Hs5WcSbkDLpOSpELjkCtQGHANrJpYidHm6Wb0tIPAwDFqI2F80pU6Zg+/btSE1NhYdHzTPhkWFMWBVvtKe/hl6bgAxvY0IaZm433CqDn48IYVMg6Y1hwIrcvn0brVu3xpQpU/DVV1+JXY7V4NMf1WRZXAoW7U+u93He6Rdkdh0pyTRwNIEVady4MWbNmoVly5YhJSVF7HKsBoeGUk2sdUQFmQ6+GbAyRUVFaNOmDTp37owdO3aIXY5V4dMf1SQ9R1XeOVjQaatdh6Gsc3CPAG+zHVFBpoNhwAr9/PPPGDt2LA4dOoSIiAixy7EqGxPS6jQ0VCaVQC6VYP7QYLYHW4Hdx85h4vwVaNVjKLKLYLEjKsh0MAxYIUEQ0L17d6jVapw9exYyWfWrwJFh3f/0V9uhoXz6sy6ffvopPvnkEygUCmggs+gRFWQaGAas1KlTpxAWFobVq1fjhRdeELscq3T/0NDr2UrgvrXt+fRn3Z566il4e3tj586dYpdCVoJhwIqNGTMGR48eRXJyMpydH14djRpO734D4ODjh48XfsanPyunUCjg4+OD77//HlOmTBG7HLISHE1gxT777DPk5OTgiy++ELsUq5d9+yYedbfBY34eCG7qxiBgxfbt2wedToeBAweKXQpZEYYBK+bv74/p06dj0aJFSE9PF7scq5aVlQUfH8uZWpjqLjo6Gp06dUKzZs3ELoWsCMOAlZs1axZcXFwwe/ZssUuxWjqdDnfu3GEYIGg0Guzbtw+DBw8WuxSyMgwDVs7V1RULFizAunXrkJCQIHY5ViknJwc6nY5hgHD69GncvXsXgwYNErsUsjIMA4SoqCiEhIRg+vTpYH/ShpeVdW/hIoYBio6Ohre3N0JDQ8UuhawMwwBBJpNh8eLFOHHiBLZt2yZ2OVaHYYDKREdHY8CAAZz7gxocwwABAPr27YtBgwbh3XffRXFxsdjlWBWGAQKAtLQ0JCYmsomARMEwQOW+/PJLpKWl4ZtvvhG7FKty584d2NjYwM3NTexSSETR0dGQyWR4+umnxS6FrBDDAJVr27YtXn75ZXz88cflT6tkfGXDCiUS/VarI8sSHR2Np556Cu7u7mKXQlaIYYAqmDNnDiQSCebOnSt2KVaDcwxQUVERDh06xCYCEg3DAFXg7e2Njz76CD/88AOSkpLELscqMAxQXFwcioqKGAZINAwD9JBp06bh0UcfxYwZM8QuxSowDFB0dDT8/f3Rtm1bsUshK8UwQA+xs7PDF198gX379mHfvn1il2PxsrKy0KhRI7HLIJEIgoDdu3dj0KBB7DdComEYoEoNHz4cPXv2xNtvvw2NRiN2ORaNbwasW1JSEtLS0jgFMYmKYYAqJZFI8NVXX+HSpUtYuXKl2OVYrJKSEuTm5jIMWLHo6Gg4OjqiV69eYpdCVoxhgKr0+OOPY+LEifjoo4+Ql5cndjkW6c6dOwA44ZA1i46ORp8+fWBvby92KWTFGAaoWp988gmUSiUWLlwodikWibMPWre7d+/i5MmTHEVAomMYoGo1a9YM7777LpYuXYqrV6+KXY7FYRiwbjExMdBqtRg4cKDYpZCVYxigGs2YMQPe3t6YOXOm2KVYnLIwwNEE1mn37t3o2LEjWrRoIXYpZOUYBqhGTk5O+PTTT7FlyxYcP35c7HIsSlZWFpydneHo6Ch2KdTAtFot9u3bxyYCMgkMA1Qr48ePx+OPP4633noLOp1O7HIsBocVWq/4+HgoFAqGATIJDANUK1KpFF999RUSEhKwYcMGscuxGHfu3GEYsFLR0dHw8vJC165dxS6FiGGAaq9nz54YMWIEZs2aBZVKJXY5FoFvBqxXdHQ0+vfvD5lMJnYpRAwDpJ8vvvgCmZmZWLx4sdilWASGAet048YN/P7772wiIJPBMEB6adWqFV5//XV89tlnyMjIELscs8cwYJ2io6Mhk8nw9NNPi10KEQCGAaqDDz74AA4ODvjggw/ELsWsCYLAMGCloqOjERYWBk9PT7FLIQLAMEB14O7ujnnz5mHNmjU4f/682OWYLaVSiaKiIoYBK1NcXIyDBw+yiYBMCsMA1cnUqVPRpk0bvPXWWxAEQexyzBInHLJOhw8fhkqlYhggk8IwQHUil8uxaNEiHD58GLt27RK7HLPEqYitU3R0NPz8/BAcHCx2KUTlGAaozgYMGIC+fftixowZKCkpEbscs8MwYH0EQUB0dDQGDx4MiUQidjlE5RgGqM4kEgkWL16Mq1evYvny5WKXY3bKwoC3t7fIlVBDuXTpEv766y82EZDJYRigegkJCcGLL76IefPmQaFQiF2OWcnKyoKXlxfkcrnYpVADiY6OhoODAyIiIsQuhagChgGqt/nz50Or1WL+/Plil2JWOKzQ+kRHR6N3795wcHAQuxSiChgGqN58fX0xe/ZsLF++HJcvXxa7HLPBMGBdcnNzcfz4cTYRkEliGCCDePPNN9GsWTO88847YpdiNhgGrMv+/fuh1WoZBsgkMQyQQdjb2+Pzzz/HL7/8goMHD4pdjllgGLAu0dHRCAkJgZ+fn9ilED2EYYAMZvTo0ejevTveeustaLVascsxeVy+2HpotVrs2bOHbwXIZDEMkMFIJBIsWbIEFy5cwJo1a8Qux6TpdDqGASuSkJCA7OxshgEyWQwDZFBdu3bF2LFj8f7776OgoKDCNqVag6SMPJxPu4ukjDwo1RqRqhTf3bt3odVqGQasRHR0NDw8PNCtWzexSyGqFAc4k8F9+umnCAoKwueff45Jr72H9fFpiLuchbQcFe5fxUACwM/TERFBPhjX1Q+tfV3EKrnBcfZB6xIdHY3+/ftzTgkyWRKBq8yQEbz5/gJs+ksKu0c6QSaVQKur+q9Z2fYeAd5YODwELTwdG7BScRw5cgS9evXC5cuXERgYKHY5ZEQZGRlo1qwZ1q9fj7Fjx4pdDlGl2ExABrcxIQ17pV1g2yIEAKoNAvdvP3lVgcglR7AxIc3oNYqNKxZajz179kAqlaJ///5il0JUJb6zIoNaFpeCRfuTAQASqUyv72p1ArQ6ATO3JyK7UI1XI1obo0STkJWVBblcDnd3d7FLISPbvXs3unfvDk9PT7FLIaoS3wyQwWxMSCsPAvW1aH8yNlnwG4KyOQa4cp1lU6vViI2N5SgCMnkMA2QQ6TkqzNmVVOv9805uwvXPBiNj5StV7vPRriSk56gMUZ7J4YRD1uHIkSNQKpUMA2TyGAbIIGbvSISmhr4BZTT52cg7tRkSG/vq99MJmL0j0RDlmRyGAesQHR2N5s2bIyQkROxSiKrFMED1lpJZgGOp2TV2FCxzN24V7JoGwbZxQLX7aXUCjqVmIzWroNr9zBHDgOUTBAHR0dEYPHgwm4PI5DEMUL2tj0+DTFq7i11x2kWo/jwBjz7/qtX+MqkE605bXt8BhgHLl5ycjCtXrrCJgMwCwwDVW9zlrFq9FRB0WuQc+B7OHfvB1se/VsfW6gTEJWfVs0LTwzBg+Xbv3g17e3v07t1b7FKIasQwQPVSqNYgrZad/ArP74Um/w7ce07Q6xxpCpVFTV1cUlKC3NxchgELFx0djYiICDg6Wv4kWmT+GAaoXq4rlKhNTwFtUT5yj62He9izkDm66XUOAcA1hbJO9ZmiO3fuAOBUxJYsLy8Px44dYxMBmQ2GAaqXEo2uVvvlHl0LqYMzXLoMMep5zAHDgOU7cOAANBoNwwCZDc5ASPViK685T5bm3EThbzHw6DMF2oKc8s8FbSkEnRaa3ExI7Bwhc6h6oaLanMdccJEiyxcdHY3g4GD4+/uLXQpRrTAMUL34ezlBAlTbVKAtUACCDndjf8Dd2B8e2n7z+yi4dBkKz8jKRxhI/j6PpeC6BJZNp9Nhz549eP7558UuhajWGAaoXpzs5PDzdMT1ajoR2jR6BI1GvP/Q57lH10JXUgTPyH9B7t6kyu+X5GSgb0RPDBw4EIMGDUKnTp3Metx2VlYWnJ2d2bHMQp09exZZWVlsIiCzwjBA9RYR5IO18derHF4oc3SDY2D3hz7PT/gfAFS6rfy7EuCJR92hzW6CL774Ah9++CGaNGlSHgwiIyPh4lJ184Ip4rBCy6JUa3BNoUSJRgdbuRQ7d++Fu7s7wsLCxC6NqNYYBqjexnX1w5pT14xybK0ALJzcHwEzR6GkpATHjx9HdHQ09uzZg1WrVsHGxgY9e/bEoEGDMGjQILRu3drk3xpkZWWxicDMpWQWYH18GuIuZyEtR1WxmUx4HD5R3+HjPZcxrqsfWvuaV1gl6yQRBKF2c8gSVWPCqnicvKqo9ZTEtSGTShDW0gtro7pWuv3q1avYs2cPoqOjERcXB7VajVatWmHQoEEYOHAgwsPDYW9f/foHDansCfK1N6ZDJhGwc+2PcLJjHjcn6TkqzN6RiGOp2ZBJJdX+fS/b3iPAGwuHh6CFJ5uFyHQxDJBBpOeoELnkCNQGHAJoJ5cidnp4rS6iSqUScXFxiI6ORnR0NNLT0+Ho6IjIyEgMHDgQAwcORIsWLQxWW21V9wQpAeDn6YiIIB8+QZqBjQlpmLMrCRqdoFfolUklkEslmDc0GGNC/YxYIVHdMQyQwWxMSMPM7YZbZfDzESF4tg4XT0EQkJSUVN6ccOLECWi1WnTo0KG8r0G3bt0glxvvqZxPkJZlWVwKFu1PrvdxZvQLxKsRrQ1QEZFhMQyQQRnqovlOvyBMi6h+VcPaunv3Lvbv3489e/Zg7969uHPnDjw8PPD0009j0KBB6N+/P7y9vQ1yLoBPkJbGVEIukTExDJDB1fdmOH9osNEuljqdDgkJCeV9Dc6dOweJRIKuXbuW9zV47LHH6twJkU+QlqW65q/s3UugvHiwyu82m7YGcpeHQ6Y+zV9EDYVhgIzCXF6T37p1C/v27UN0dDT279+PgoKC8qGLAwcORN++fWs9dJFPkJanuo6x6puXUHr39gOfCsiJ+RZyN180fXF5pcesqWMskRgYBsioyjvQJWchTVFJBzovR0QE+mB8Nz8E+Ijbga6kpAQnTpwo72tw6dKl8qGLZX0NAgMDK31rUN0TZPH1C8j8eXal52w8YRHsmrWpdBufIMWVklmAvkuP6vWd4vQkZK5/D+49J8ItbHS1+8ZO7yn633miMgwD1GAenJzF38vJpIfWlQ1d3LNnDw4dOlQ+dLEsGNw/dLG6J8iyMODy+BDYNgmssM2hZecqV3HkE6S45u5KqnYyrcooYpaj8PxeNHtpJeTuvlXuJ5NKMKHrI5g7NNgQpRLVG8MAUS2oVCocOnSovK9BWloaHB0d0adPH3TtOww/ZjSu8rtlYcD7mZlwavOU3ufmE6Q4wr+Mq3aa7QcJWg1uLJsIG6/maDz+ixr3f8TLEUdmRNSnRCKDsZyl4IiMyNHREYMHD8by5ctx7do1JCYmYs6cOcjLy8NXv5yFoNPW6jg6tarW+wL3niDXnU6ra9lUR4VqDdL0CAIAUPTXr9AV5cOpXa9a7Z+mUEGp1tShOiLDM913tEQmSiKRoH379mjfvj3effddPPX5QdzILa7xe4o9/4ZQUgRIpLBrEQyPiMmwa1L9iAGtTkBcchbmgq+TG9J1hbLalTgro/zjCCCVw7Ft7d7+CACuKZQIblp5MxFRQ2IYIKqHQrUGN2sKAjIbOAaFwaFlF0gd3VCanYb8MzuQuf49NB7/JWwbt6r262VPkKbcv8LSlOg5k6aupAhFKafh8OhjkDm4Gu08RMbCqwtRPdTmCdK+eVvYN2/7zwetu8KxzZO4teo13D3yE3yfnV/t9/kE2fBs5fq1oKqST0MoVcMpuJdRz0NkLPybSFQPdX2ys/FoCofWXVGcdqFWfQj4BNmw/L2coM+0U8o/DkNi6wCH1rUf+SH5+zxEpoBhgKge6vNkJ3f1BrQaCKVqo56H9OdkJ4dfLed30KryUHztNzi27gapTe1XyfTzcmTTD5kMXmGI6kHfJ8j7aXJvQyK3hcS2+hsInyDFERHkA5m05v+6yktHAZ1WryYCmVSCiECfelRHZFgMA0T1UJsnSK0q76HPSjKvQpVyBvb+j0Eiqf6fIZ8gxTEwyKVWEw4pkw5D6ugOe/9OtT62VidgfDdONU2mg1cYonqKCPKpdqa6Ozs/h9TGFnbN2v49miAdhb/vg8TGDh69nq/22HyCbHiCIGDjxo14/fXXYff027Bt0R66at7/NJm4WK/jl80syYmkyJTwzQBRPY3r6lftE6RjYDdoVfnIP7MTOfu/g+rPY3AMDEOT55fAxrtFtcfW6gR09SoxdMlUhRs3bmDo0KEYO3YsevfujV8+Ggsbucyg55BLJVg4PMSgxySqL05HTGQA1a1NUFdSCNDdvoz0/76LqKgozJ8/H40bVz3tMdWdTqfDihUr8O6778LZ2RnLly/HM888A4CrUZJ14JsBIgNYODwE8lp0NtOHjVyGg5+9iK+++grbtm1DQEAAFixYAKVSadDzWLvk5GRERETg5ZdfxpgxY/DHH3+UBwEAGBPqhxn9Aqs+gB7e6RfEIEAmiWGAyABaeDpinoFXoJs/NBitfN3wxhtvIDU1FS+99BI+/vhjBAYGYs2aNdBqa7/GAT1Mo9Hgiy++QMeOHXHjxg0cPHgQK1asgLu7+0P7vhrRGp+NCIGdXFqrEQb3k0klsJNL8fmIEEyLCDBQ9USGxWYCIgNaFpeCRfuT632cd/oFVXrjuHr1KmbNmoXNmzejY8eOWLRoESIjI+t9Pmvz22+/ISoqCr/99hveeustzJs3D46ONc8rkJ6jwuwdiTiWmg2ZVFJts1DZ9h4B3lg4PAQtajlvAZEYGAaIDGxjQhrm7EqCRifo1YdAJpVALpVg/tDgGl8lnzp1Cm+//TZOnTqFgQMH4osvvkBwMBczqklxcTEWLFiAzz//HO3atcOqVasQGhqq93FSMguwPj4NcclZSFOoKkxJLcG94aARgT4Y382PowbILDAMEBlBQzxBCoKAbdu24b333sO1a9cwZcoUzJs3D76+vob6MSzK8ePH8eKLL+Lq1av48MMP8d5778HW1rbex1WqNbimUKJEo4OtXAp/LyfOC0Fmh2GAyIga4glSrVZj+fLlWLBgAUpLS/Hee+/hrbfeqtVrb2tQUFCAWbNm4dtvv0X37t2xcuVKtGvXTuyyiEwKwwBRAzH2E2ROTg4+/vhjLFu2DD4+Pvjkk08wYcIESKXW20947969mDp1KhQKBT799FNMmzYNMplh5w0gsgQMA0QW5sqVK5g5cya2bt2KTp06YfHixejdu7fYZTWo7OxsTJ8+HevWrUO/fv3www8/wN/fX+yyiEyW9T4yEFmoVq1aYcuWLThx4gTs7e3Rp08fDBkyBJcuXRK7NKMTBAGbNm1Cu3btEB0djTVr1mDfvn0MAkQ1YBggslBhYWE4efIkNm/ejKSkJISEhODll19GZmam2KUZxY0bNzBs2DCMGTMGvXr1wh9//IFJkyZBIjHsZFBElohhgMiCSSQSjBo1CpcuXcIXX3yBjRs3onXr1li4cCGKiorELs8gyqYSDg4ORkJCArZv347Nmzdz6mYiPbDPAJEVUSgU+Pjjj/Htt9/C19cXCxcuxLhx48y2k2FKSgqmTJmCI0eOICoqCl9++SU8PDzELovI7JjnFYCI6sTLywtLlizBH3/8ga5du2LixIkIDQ1FXFyc2KXpRaPR4Msvv0SHDh2Qnp6O2NhYrFy5kkGAqI4YBoisUEBAALZu3Yrjx4/DxsYGvXv3xtChQ/Hnn3+KXVqNfv/9d3Tr1g0zZ87EK6+8ggsXLqBPnz5il0Vk1hgGiKzYk08+iVOnTmHjxo1ITExE+/btMW3aNGRlZRnsHEq1BkkZeTifdhdJGXlQqjV1Ok5xcTE++OADdOnSBWq1GqdOncLixYvh5ORksFqJrBX7DBARgHs322XLluHjjz+GTqfDrFmz8Oabb8LBwUHvY5XPvHg5C2k5lcy86OmIiCAfjOvqh9a+Nc+8eOLECbz44ou4cuUKPvjgA8ycOdMgUwkT0T0MA0RUgUKhwPz587F8+XI0adIECxcuxNixY2vVydDQazIUFBRg9uzZ+Pbbb9G1a1esXLmSCzIRGQHDABFVKiUlBTNnzsT27dvx+OOPY/HixQgPD69y//qu1jhvaDDG3Lda4759+zB16lRkZ2dj4cKFePXVVzmVMJGRsM8AEVWqdevW2LZtG44ePQqpVIpevXph2LBhuHz58kP7LotLwcztiVBrdHoFAQDQ6gSoNTrM3J6IZXEpUCgUmDhxIgYMGIDAwEBcvHgRb7zxBoMAkRHxzQAR1Uin02HTpk2YNWsWbty4gZdeeglz5sxBo0aNsDEhDTO3JxrsXOpjq1GcdAhLlizhDIJEDYRhgIhqrbi4GN988w0++eQTCIKAV9+bg61FbaHW6CrdX307FXnHN0B94w8ImlLI3X3h3Kk/XLsMrXR/QRAgFbTYNrkTOgc9YswfhYjuwzBARHrLzs7G/PnzsfmOD+z8OkAiffgVftFfvyJr63zY+raCU5sekNjaQ5N7GxB08IiYXOWxZVIJwlp6YW1UV2P+CER0H4YBIqqTlMwC9F16tNJtOrUKN1f8C3bN2qLR8FmQSPTvnhQ7vScCfGoedkhE9ccOhERUJ+vj0yCTVt6er/zjMHTKXHj0nAiJRApdSTEEofKmhMrIpBKsO51mqFKJqAZysQsgIvMUdzmrypEDxdd+g8TOEZpCBbK2fwxNzk1IbOzh1D4Cnn2mQCKvfsIgrU5AXHIW5oJzChA1BIYBItJboVqDtBxVldtLczIAnRZ3ti2Ac4d+sA+fhOK0RBSc+wW6YiUaDXu3xnOkKVRQqjVwsuNlisjY+K+MiPR2XaFEdZ2NhNJiCKVqOD82AJ59pwIAHIPCIGhLUfjbPpT2GAcbz2bVnkMAcE2hRHBTN8MVTkSVYp8BItJbSRVDCcuUNQM4ta04Y6FTu14AAPXN2q2OWNN5iMgwGAaISG+28uovHTJnr3v/7+Re8XOne0/5uuJCg5yHiAyD/9KISG/+Xk6obl5A28atAACaAkWFzzUFOQAAmWPNr/4lf5+HiIyPYYCI9OZkJ4dfJasMlm9v0wMAUHhhf4XPCy/sB6Qy2PmF1HgOPy9Hdh4kaiD8l0ZEdRIR5IO18dcrHV5o27gVnDr0hfLCAdzR6WDv1x7FaYlQ/Xkcrt1HQe7iVe2xZVIJIgJ9jFU6ET2AMxASUZ1UNwMhAAhaDfJObUbhhVhoC3Mgd2sEl86D4Ro6rFbH5wyERA2HYYCI6mzCqnicvKrQe9ni6nBtAqKGxz4DRFRnC4eHQF7FlMR1JZdKsHB4zX0KiMhwGAaIqM5aeDpi3lDDThk8f2gwWlTTOZGIDI9hgIjqZUyoH2b0CzTIsd7pF4RnQ/0Mciwiqj32GSAig9iYkIY5u5Kg0Ql69SGQSSWQSyWYPzSYQYBIJAwDRGQw6TkqzN6RiGOp2ZBJJdWGgrLtPQK8sXB4CJsGiETEMEBEBpeSWYD18WmIS85CmkJVYVEjCe5NKBQR6IPx3fw4fJDIBDAMEJFRKdUaXFMoUaLRwVYuhb+XE2cWJDIxDANERERWjqMJiIiIrBzDABERkZVjGCAiIrJyDANERERWjmGAiIjIyjEMEBERWTmGASIiIivHMEBERGTlGAaIiIisHMMAERGRlWMYICIisnIMA0RERFaOYYCIiMjKMQwQERFZOYYBIiIiK8cwQEREZOUYBoiIiKwcwwAREZGVYxggIiKycgwDREREVo5hgIiIyMoxDBAREVk5hgEiIiIrxzBARERk5RgGiIiIrBzDABERkZVjGCAiIrJy/w9s2m6zIZafKwAAAABJRU5ErkJggg==\n"
                },
                "metadata": {}
              }
            ]
          }
        },
        "5713ef75aaa049d6a2d52672895966e5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0ed7d65b4c2420a87e66fbcd8975abc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdb2ae1e627e4a188978c0e481be4fd8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "SliderStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "SliderStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": "",
            "handle_color": null
          }
        },
        "b30d88594e2540ecb30e5e318468bbaf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}