First example draft

examples/mnist-pytorch-studio/.gitignore

@@ -0,0 +1,6 @@
+data
+*.npz
+*.tgz
+*.tar.gz
+.mnist-pytorch
+client.yaml

examples/mnist-pytorch-studio/API_Example.ipynb

@@ -0,0 +1,272 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "622f7047",
+   "metadata": {},
+   "source": [
+    "## FEDn API Example\n",
+    "\n",
+    "This notebook provides an example of how to use the FEDn API to organize experiments and to analyze validation results. We will here run one training session using FedAvg and one session using FedAdam and compare the results.\n",
+    "\n",
+    "When you start this tutorial you should have a deployed FEDn Network up and running, and you should have created the compute package and the initial model, see the README for instructions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "743dfe47",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from fedn import APIClient\n",
+    "import time\n",
+    "import uuid\n",
+    "import json\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "import collections"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1046a4e5",
+   "metadata": {},
+   "source": [
+    "We make a client connection to the FEDn API service. Here we assume that FEDn is deployed locally in pseudo-distributed mode with default ports."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "1061722d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "DISCOVER_HOST = '127.0.0.1'\n",
+    "DISCOVER_PORT = 8092\n",
+    "client = APIClient(DISCOVER_HOST, DISCOVER_PORT)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07f69f5f",
+   "metadata": {},
+   "source": [
+    "Initialize FEDn with the compute package and seed model. Note that these files needs to be created separately by follwing instructions in the README."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "5107f6f9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client.set_package('package.tgz', 'numpyhelper')\n",
+    "client.set_initial_model('seed.npz')\n",
+    "seed_model = client.get_initial_model()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e26c50b",
+   "metadata": {},
+   "source": [
+    "Next we start a training session using FedAvg and wait until it has finished:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "f0380d35",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "session_id = \"experiment_fedavg\"\n",
+    "\n",
+    "session_config_fedavg = {\n",
+    "                    \"helper\": \"numpyhelper\",\n",
+    "                    \"session_id\": session_id,\n",
+    "                    \"aggregator\": \"fedavg\",\n",
+    "                    \"model_id\": seed_model['model_id'],\n",
+    "                    \"rounds\": 2\n",
+    "                 }\n",
+    "\n",
+    "result_fedavg = client.start_session(**session_config_fedavg)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8cc709c2",
+   "metadata": {},
+   "source": [
+    "We wait for the session to finish: "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "897451fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "while not client.session_is_finished(session_id):\n",
+    "    time.sleep(2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "16874cec",
+   "metadata": {},
+   "source": [
+    "Next, we retrive all model validations from all clients, extract the training accuracy metric, and compute its mean value accross all clients"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4e8044b7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "models = client.list_models(session_id)\n",
+    "\n",
+    "validations = []\n",
+    "acc = collections.OrderedDict()\n",
+    "for model in models[\"result\"]:\n",
+    "    model_id = model[\"model\"]\n",
+    "    validations = client.list_validations(modelId=model_id)\n",
+    "\n",
+    "    for _ , validation in validations.items(): \n",
+    "        metrics = json.loads(validation['data'])\n",
+    "        try:\n",
+    "            acc[model_id].append(metrics['training_accuracy'])\n",
+    "        except KeyError: \n",
+    "            acc[model_id] = [metrics['training_accuracy']]\n",
+    "            \n",
+    "mean_acc_fedavg = []\n",
+    "for model, data in acc.items():\n",
+    "    mean_acc_fedavg.append(np.mean(data))\n",
+    "mean_acc_fedavg.reverse()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "42425c43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "x = range(1,len(mean_acc_fedavg)+1)\n",
+    "plt.plot(x, mean_acc_fedavg)\n",
+    "plt.legend(['Training Accuracy (FedAvg)'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4eeea5aa",
+   "metadata": {},
+   "source": [
+    "Let's try another aggregation algorithm. We start another session using FedOpt (FedAdam)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4f70d7d9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "session_config_fedopt = {\n",
+    "                    \"helper\": \"numpyhelper\",\n",
+    "                    \"session_id\": \"experiment_fedopt\",\n",
+    "                    \"aggregator\": \"fedopt\",\n",
+    "                    \"model_id\": seed_model['model_id'],\n",
+    "                    \"rounds\": 10\n",
+    "                 }\n",
+    "\n",
+    "result_fedopt = client.start_session(**session_config_fedopt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ce8a89a3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "while not client.session_is_finished(\"experiment_fedopt\"):\n",
+    "    time.sleep(2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "900eb0a7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "models = client.list_models(session_id = \"experiment_fedopt\")\n",
+    "\n",
+    "validations = []\n",
+    "acc = collections.OrderedDict()\n",
+    "for model in models[\"result\"]:\n",
+    "    model_id = model[\"model\"]\n",
+    "    validations = client.list_validations(modelId=model_id)\n",
+    "    for _ , validation in validations.items(): \n",
+    "        metrics = json.loads(validation['data'])\n",
+    "        try:\n",
+    "            acc[model_id].append(metrics['training_accuracy'])\n",
+    "        except KeyError: \n",
+    "            acc[model_id] = [metrics['training_accuracy']]\n",
+    "            \n",
+    "mean_acc_fedopt = []\n",
+    "for model, data in acc.items():\n",
+    "    mean_acc_fedopt.append(np.mean(data))\n",
+    "mean_acc_fedopt.reverse()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "40db4542",
+   "metadata": {},
+   "source": [
+    "Finally, plot the resulting accuracy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d064aaf9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "x = range(1,len(mean_acc_fedavg)+1)\n",
+    "plt.plot(x, mean_acc_fedavg, x, mean_acc_fedopt)\n",
+    "plt.legend(['FedAvg', 'FedAdam'])"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}