Reproducible-Data-Science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reproducible Data Science\n",
    "\n",
    "## Tory W. Clasen\n",
    "## Chief Data Scientist\n",
    "### 18 JUL 19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# My Backstory\n",
    "\n",
    "\n",
    " - 20+ years of software development (Python < 0.9!)\n",
    " -  8+ years of cyber stuff\n",
    " -  4 years of data science\n",
    " \n",
    " \n",
    " ** Remind Tory to go on a tangent, time permitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What does it mean for research to be reproducible?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " - <B>Reproducibility:</B>\n",
    "\n",
    "A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study.\n",
    "\n",
    "** <B>functional purity</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " - <B>Replicability:</B>\n",
    "\n",
    "This is the act of repeating an entire study, independently of the original investigator without the use of original data (but generally using the same methods). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reproducicble research has 4 components:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Source Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b439982f30b5b47fde89ff1384b671e0\n",
      "de6fdc009fda2de3cf4dab4bfd4529c2\n",
      "62028945d0ab974e183756ebbb1ca07f\n",
      "f6febaf70042268a17efc985875131f5\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 4 malware_hash_list.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash\n",
      "\n",
      "source config.env\n",
      "\n",
      "if [ ! -d samples ]; then\n",
      "    mkdir samples\n",
      "\n",
      "    for sample in $( cat malware_hash_list.txt ); do\n",
      "\techo \"Downloading $sample\"\n",
      "        curl -s -o ./samples/$sample  \"https://malshare.com/api.php?api_key=$MALSHARE_API_KEY&action=getfile&hash=$sample\"\n",
      "    done\n",
      "fi\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat 1_download_samples.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Source Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash\n",
      "\n",
      "source config.env\n",
      "\n",
      "if [ ! -d hashes ]; then\n",
      "    mkdir hashes\n",
      "\n",
      "    docker run --rm -it \\\n",
      "        -v \"$PWD\"/samples:/archive \\\n",
      "        -v \"$PWD\"/hashes:/hashes \\\n",
      "        -w /hashes \\\n",
      "        --name pharos \\\n",
      "        eschwartz/pharos:$PHAROS_DOCKER_TAG \\\n",
      "        /bin/bash -c \"\\\n",
      "            for i in /archive/*; do \\\n",
      "                fn2hash \\$i; \\\n",
      "            done > hashes.csv;\\\n",
      "            for i in /archive/*; do \\\n",
      "\t        fn2yara \\$i; \\\n",
      "\t    done\"\n",
      "fi\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat 2_hash_samples.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Configuration Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "JUPYTER_PASSWORD=\"\"\n",
      "JUPYTER_DOCKER_TAG=\"6c3390a9292e\"\n",
      "\n",
      "MALSHARE_API_KEY=\"<INSERT API KEY HERE>\"\n",
      "\n",
      "PHAROS_DOCKER_TAG=\"latest\"\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat config.env | sed -e 's/MALSHARE_API_KEY=.*$/MALSHARE_API_KEY=\"<INSERT API KEY HERE>\"/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "** <B>Pets vs. Cattle</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Intermediate Data and results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Show your work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reproducible Data Science with 10 easy rules, you won't believe #9!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 1: For Every Result, Keep Track of How It Was Produced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " - Again, show your work!\n",
    " - \"OP Notes\"\n",
    " - Jupyter Notebooks\n",
    " - \"script\" command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 2: Avoid Manual Data Manipulation Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " - Don't use excel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " - Instead use:\n",
    " \n",
    "`zcat conn.gz.log | bro-cut id.orig_h id.orig_p id.resp_h id.resp_p orig_bytes | awk '{ a[\"\\t\", $1, \"\\t\", $2, \"\\t\", $3, \"\\t\", $4] +=$5 } END { for ( i in a ) { print i, \"\\t\", a[i] } }' | sort -rnk6 | head -n $size 2>/dev/null`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 3: Archive the Exact Versions of All External Programs Used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "`docker run --rm -it \\\n",
    "        -v \"$PWD\"/samples:/archive \\\n",
    "        -v \"$PWD\"/hashes:/hashes \\\n",
    "        -w /hashes \\\n",
    "        --name pharos \\\n",
    "        eschwartz/pharos:$PHAROS_DOCKER_TAG`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But don't do this:\n",
    "\n",
    "`PHAROS_DOCKER_TAG=\"latest\"`\n",
    "\n",
    "** <B>CI/CD Pipelines in Data Science</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 4: Version Control All Custom Scripts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "commit 36e75683d565a7a0c2785c825f65acce104a1b69\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Thu Jul 18 20:42:17 2019 -0400\n",
      "\n",
      "    Cleaned up formatting\n",
      "\n",
      "commit e1ed103bf3bd093dd4608322d0c7610913a0e456\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Tue Jul 16 23:03:55 2019 -0400\n",
      "\n",
      "    Unhid all slides required, added a summary slide.\n",
      "\n",
      "commit d1dbcdb79af5bc7cf1fc79b34d7a616560bdc306\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Tue Jul 16 22:51:56 2019 -0400\n",
      "\n",
      "    Renamed notebook to match repo\n",
      "\n",
      "commit 164d104847e9bfd2e7cee625d2c9d38736cd7ab2\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Tue Jul 16 22:49:17 2019 -0400\n",
      "\n",
      "    Unhide slide for collection\n",
      "\n",
      "commit 2cdc24ca3dd98b5aae8fa859e804d12d41cc92b9\n",
      "Merge: a2c2327 97355e5\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Tue Jul 16 22:46:20 2019 -0400\n",
      "\n",
      "    Merge branch 'master' of github.com:tclasen/Reproducible-Data-Science\n",
      "\n",
      "commit a2c2327597ce712baa01ff72173e172d02714aa8\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 02:46:01 2019 +0000\n",
      "\n",
      "    Added product folders to ignore\n",
      "\n",
      "commit 97355e58c5f66833d63f44a85e0dc5365b714c73\n",
      "Author: tclasen <ToryClasen@Meertecha.com>\n",
      "Date:   Tue Jul 16 22:44:18 2019 -0400\n",
      "\n",
      "    Create README.md\n",
      "\n",
      "commit e7ffbf614b6c7734512f7912e1b96ed82a5d2f96\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Tue Jul 16 22:41:52 2019 -0400\n",
      "\n",
      "    Renamed to match repo\n",
      "\n",
      "commit 0fe2eecbc74fabe16825da5657b0868699105dd9\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 02:38:08 2019 +0000\n",
      "\n",
      "    Completed rough draft\n",
      "\n",
      "commit fefee2021781b566a8707320c156a7727f12ce69\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:39:16 2019 +0000\n",
      "\n",
      "    Added shell of talk\n",
      "\n",
      "commit c91ba5fbba22d213d816590fe89611c4e088b58f\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:39:02 2019 +0000\n",
      "\n",
      "    Added notebook checkpoints\n",
      "\n",
      "commit 34a4b40f9a4cfcc1cd44931fbc8b9d10f9f2a70b\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:36:51 2019 +0000\n",
      "\n",
      "    Added script to run fn2hash and fn2yara on all samples downloaded\n",
      "\n",
      "commit 871d4d4ef5015b93c77181e1707c330ba10adb33\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:36:26 2019 +0000\n",
      "\n",
      "    Added script to download malware samples from a given list of hashes\n",
      "\n",
      "commit 89b387cf8c7461c90486dddc854bdf4c365d7ab9\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:36:03 2019 +0000\n",
      "\n",
      "    Fixed folders to ignore. I need to learn how this gitignore works.\n",
      "\n",
      "commit 9a84c4890e32918695f9329496cdce7ca7558f60\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:35:15 2019 +0000\n",
      "\n",
      "    Added config file and malware folders\n",
      "\n",
      "commit ab5e947cb79d62727479348a0ee6d151fb357ec6\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:32:10 2019 +0000\n",
      "\n",
      "    Added standard jupyter server file\n",
      "\n",
      "commit b3123820bd524ca41fcf4ea5a9a887e81d4f875a\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:31:41 2019 +0000\n",
      "\n",
      "    Initial malware pull\n",
      "\n",
      "commit 1f798ef26559a660c1edaf5b65837a5edbb9dbfe\n",
      "Author: Tory Clasen <ToryClasen@gmail.com>\n",
      "Date:   Wed Jul 17 01:31:20 2019 +0000\n",
      "\n",
      "    Add product build script for slides and pdf reports\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "git log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 5: Record All Intermediate Results, When Possible in Standardized Formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00051B00402ACCB8A685FD77374DA817,0x00401000,2,2,53,147,51ADB788A659ACCFC01A55FD5BB296FB,51ADB788A659ACCFC01A55FD5BB296FB,5857DB86AA1F27EAED5A1A4FD65DC474,22FAAD875C868C0046084CFB6DC3846B,D40ADD2AF2612EAC4E7E67502D7D19BD,EBC72393A33D247D9C427660B00EE26A,D7ACE67CFAC1F836388E40C780749510,adc:1;add:28;call:1;cmp:1;cmpsb:1;cwde:1;dec:1;hlt:1;imul:1;in:1;lodsb:1;mov:4;or:1;push:4;sbb:1;sub:1;test:1;xchg:2;xlatb:1,BR:1;CMP:2;CRYPTO:0;FLT:0;I/O:1;LOGIC:1;MATH:33;SIMD:1;STR:2;SYS:0;UNCAT:0;VMM:0;XFER:12\n",
      "00051B00402ACCB8A685FD77374DA817,0x00401093,1,1,42,93,88F2BEE97B748C0E5F5F9EE6A4291ACC,88F2BEE97B748C0E5F5F9EE6A4291ACC,D3FD500560A6FD3B31F81C8345D11A0B,E3A24B0C644A5957FBFAF8DAE0E34950,14CD661AA379D889E154407AC70D4822,2E9D3D8E4E55CCA23E50060555D17A6F,44B22C487D842AADAC6B54246E621367,adc:1;add:24;cdq:1;enter:1;inc:1;jmp:1;mov:2;or:1;out:1;pop:1;popad:1;popfd:1;push:2;pushad:1;stosd:1;test:1;xor:1,BR:1;CMP:1;CRYPTO:0;FLT:0;I/O:1;LOGIC:2;MATH:26;SIMD:0;STR:1;SYS:0;UNCAT:0;VMM:0;XFER:10\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat ./hashes/hashes.csv | head -n 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Include headers with your CSVs, or instead use JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from random import randint\n",
    "def rolldice(): return randint(1,20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[20, 10, 6, 17, 9, 17, 16, 10, 9, 11]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[ rolldice() for i in range(10) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 14, 14, 13, 9, 12, 10, 6, 13, 2]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[ rolldice() for i in range(10) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from random import seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 1, 9, 8, 8, 5, 4, 18, 3, 19]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seed(42)\n",
    "[ rolldice() for i in range(10) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 1, 9, 8, 8, 5, 4, 18, 3, 19]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seed(42)\n",
    "[ rolldice() for i in range(10) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "** <B>functional purity</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 7: Always Store Raw Data behind Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "names = \"filemd5,fn_addr,num_basic_blocks,num_basic_blocks_in_cfg,num_instructions,num_bytes,exact_hash,pic_hash,composite_pic_hash,mnemonic_hash,mnemonic_count_hash,mnemonic_category_hash,mnemonic_category_counts_hash,mnemonic_count_string,mnemonic_category_count_string,opt_basic_block_data,opt_bb_cfg\".split(\",\")\n",
    "df = pd.read_csv(\"./hashes/hashes.csv\", names=names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdYAAAFvCAYAAAD33+rnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXlWd5/HPl7QSkISAC6JRApgBESFAiMPmAnQLNouCDiDIqgwKEmS0m9Z+aY8ONu0y3WwtHZXQoKCIwoBDCzaKLAEhQEII4IgJSMQNgRAIAlX1mz/uKermSaWeJ8lNnXvrft+87ivPPXf7pajUr85yz1FEYGZmZtVYL3cAZmZmY4kTq5mZWYWcWM3MzCrkxGpmZlYhJ1YzM7MKObGamZlVyInVzMysQk6sZmZmFXJiNTMzq5ATq5mZWYX+IncA1hwvPr7I818C8dyy3CHUxoSpB+QOoTYun7RX7hBq5eDfXaq1vUevP3Ne9qqt1vpZVXJiNTOzehrozx3BGnFiNTOzeoqB3BGsESdWMzOrpwEnVjMzs8pEf1/uENaIE6uZmdWTm4LNzMwq5MFLZmZmFWpojdUTRJiZWT0NDPS29UDSfpJ+IekhSWcMc3wTSVdKulfSHZK2Lx2bKek+SQslndbtWU6sZmZWS9Hf19PWjaRxwPnA/sB2wBGStus47dPAvIjYATgaODtduz3wEWAGsCNwgKSpIz3PidXMzOopBnrbupsBPBQRiyLiBeA7wMEd52wH3AAQEQ8CUyRtBrwZuD0ilkdEH/Az4H0jPcyJ1czM6mmgv7etu9cDj5b2l6SysvnAIQCSZgBbAJOB+4C3S3qlpA2B9wBvGOlhHrxkZmb11OPgJUknAieWimZFxKzyKcPdvWP/LOBsSfOABcA9QF9EPCDpn4AfA89QJOAR25+dWM3MrJ56HJiUkuisEU5Zwoq1zMnAYx33eBo4DkCSgMVpIyK+CXwzHftiut8quSnYzMzqqbo+1juBqZK2lPRy4HDg6vIJkialYwAfBm5KyRZJr0l/vpGiufiykR7mGquZmdVS9L9YzX0i+iSdAlwHjAMujIiFkk5Kxy+gGKR0saR+4H7ghNItvi/plcCLwMkR8eRIz3NiNTOzeqpwgoiIuBa4tqPsgtLn24BhX6OJiNVabNeJ1czM6smr25iZmVWooVMaOrGamVk9eRJ+MzOzCnk9VjMzswq5KdjMzKxCHrxkZmZWoYYmVs+8tAqSbpQ0vYL7zKnq2ZKOlXTeGtxviqT7Vvc6M7OcIvp72urGNdZ1LCJ2zx2DmVkjuca67qWa1wOSvp5Wcr9e0gblGp6kV0l6OH0+VtJVkq6RtFjSKZJOl3SPpNslbdrlkUdJmpNWjp+R7jkjld2T/twmlb8lrTo/L61APzWVP1OK/28kLZA0X9JZq/vsjq/FFpJuSM+6Ic1hiaTNJF2ZnjFf0u4d122VYt91VTGbmdVCf19vW800KrEmU4HzI+ItwFPAoV3O3x74IMVCt2cCyyNiJ+A2ilXiR/KKVOP8GHBhKnsQeHu6x2eBL6byk4CzI2IaMJ2O1Q8k7Q+8F3hbROwIfGkNnl12HnBxWu3+28A5qfwc4GfpGTsDC0sxbAN8HzguIu7sFnO65kRJcyXN/cbFI847bWZWreom4R9VTWwKXhwR89Lnu4ApXc7/aUQsA5ZJWgpck8oXADt0ufYygIi4SdJESZOACcC/p9pdAC9L594GfEbSZOAHEfHLjnvtC8yOiOXpnk+swbPLdiMtygtcwlCi3pv0C0MUnQ9LJW0CvBr4P8ChETGYbLvFvMJyTC8+vqhz/UIzs3XHTcGj5vnS536KXw76GPq7jB/h/IHS/gDdf7HoTCQBfIEiWW8PHDj4vIi4FDgIeA64TtLeHddqmPut7rNX5/xOS4FHgT1euqB7zGZm+TS0xtrExDqch4Fd0uf3V3jfwwAk7QksjYilwMbAb9LxYwdPlLQVsCgizqFY56+zNnw9cLykDdP53fp3h3t22RyKNQUBjgRuSZ9vAD6arh0naWIqf4GiKfpoSR/sMWYzs3wGBnrbaqaJTcHD+QpwuaQPAT+p8L5PptdlJgLHp7IvUTQFn97xrMMoBhy9CPwO+Hz5RhHxI0nTgLmSXqBYvujTq/nsslOBCyV9CvgjFCvfAzOBWZJOoKjRfxT4bYrhWUkHAD+W9Cyw3Ugxm5llVcOk2QtFuNvMeuM+1kI8tyx3CLUxYeoBuUOojcsnrdaSnWPewb+7VGt7j+eu/kpPP3M2OOiTa/2sKo2VGquZmY01New/7UXrE6uk8ykN6EnOjojZY/nZZma119Cm4NYn1og4uY3PNjOrPddYzczMKuQaq5mZWYX66zfBfi+cWM3MrJ5cYzUzM6uQE6uZmVmFPHjJzMysQq6xmpmZVaihMwM6sZqZWT311W8R8144sZqZWT01tI91rCwbZ2ZmY0wMRE9bLyTtJ+kXkh6SdMYwxzeRdKWkeyXdIWn70rFJkq6Q9KCkByTtNtKznFjNzKyeKlqPVdI44Hxgf4rlMo+QtF3HaZ8G5kXEDsDRwNmlY2cDP4qIbYEdgQdGep4Tq5mZ1VMM9LZ1NwN4KCIWRcQLwHeAgzvO2Q64ASAiHgSmSNpM0kTg7cA307EXIuKpkR7mxGpmZvU0EL1t3b0eeLS0vySVlc0HDgGQNAPYApgMbAX8EZgt6R5J35D0ipEe5sRqZmb11NfX0ybpRElzS9uJHXcabiH0zox8FrCJpHnAx4F7gD6KQb47A1+LiJ2AZ4GV+mjLPCrYzMzqqcf3WCNiFjBrhFOWAG8o7U8GHuu4x9PAcQCSBCxO24bAkoj4eTr1CrokVtdYzcysnioavATcCUyVtKWklwOHA1eXT0gjf1+edj8M3BQRT0fE74BHJW2Tju0D3D/Sw1xjNTOzeurxVZpuIqJP0inAdcA44MKIWCjppHT8AuDNwMWS+ikS5wmlW3wc+HZKvItINdtVcWK1nsVzy3KHUBvaYELuEGqhv6Fzua4LzZx8r+YqnCAiIq4Fru0ou6D0+TZg6iqunQdM7/VZTqxmq8lJ1Wx0RJ8XOjczM6tORU3Bo82J1czM6qmhcwU7sZqZWT25xmpmZlahhg6Oc2I1M7N6co3VzMysQv0eFWxmZlaZcFOwmZlZhdwUbGZmViEnVjMzswr5PVYzM7MKucZqZmZWnehzjdXMzKw6HhVsZmZWITcFm5mZVciJ1czMrDoRTqxmZmbV8eAlMzOz6oSbgs3MzCrkxGpmZlahZrYEO7GamVk9uSnYzMysSk6sZmZm1Ym+ZibW9XIHUCbpRknTK7jPnKqeLelYSeetwf2mSLpvda/ruMdq/z263O8ySfdK+kSV9zUzWycGetxqZkzWWCNi99wxVKHKv4ek1wK7R8QWVd3TzGxdamofa9caa6p5PSDp65IWSrpe0gblGp6kV0l6OH0+VtJVkq6RtFjSKZJOl3SPpNslbdrlkUdJmiPpPkkz0j1npLJ70p/bpPK3SLpD0rxUE5uayp8pxf83khZImi/prNV9dsfXYgtJN6Rn3SDpjal8M0lXpmfMl7R7x3Vbpdh3XVXMq/jaP5P+fGf6el8h6UFJ35akEa7bNf095qdnTQCuB16TnrtXOudeSbdJ+vLa1q7NzCrX0Bprr03BU4HzI+ItwFPAoV3O3x74IDADOBNYHhE7AbcBR3e59hWppvYx4MJU9iDw9nSPzwJfTOUnAWdHxDRgOrCkfCNJ+wPvBd4WETsCX1qDZ5edB1wcETsA3wbOSeXnAD9Lz9gZWFiKYRvg+8BxEXFnt5hHsBNwGrAdsBWwx3AnSXo58F1gZopnX+A54CDgVxExLSJuBmYDJ0XEbkD/qh4q6URJcyXN/ca3r+gxVDOztRcDvW1102tT8OKImJc+3wVM6XL+TyNiGbBM0lLgmlS+ANihy7WXAUTETZImSpoETAD+PdXuAnhZOvc24DOSJgM/iIhfdtxrX2B2RCxP93xiDZ5dthtwSPp8CUOJem/SLwwR0Q8slbQJ8Grg/wCHRsRgsu0W86rcERFLACTNo/h/cMsw520D/DYlcSLi6XTNSycMfk0jYrAP91LggOEeGhGzgFkALzw6v5ntMmbWSNFX3b0k7QecDYwDvhERZ3Uc34SiQrU18Gfg+Ii4T9J44CZgfYqceUVEfG6kZ/VaY32+9Lk/3byvdP34Ec4fKO0P0D2Zd/7wDuALFMl6e+DAwedFxKUUNbHngOsk7d1xrYa53+o+e3XO77QUeJRS7bKHmFdluP8Hw+nl77zKZmQzs9qoqClY0jjgfGB/ila/IyRt13Hap4F5qUXyaIokDMXP3r1TC+A0YD9J/3Wk563NqOCHgV3S5/evxX06HQYgaU9gaUQsBTYGfpOOHzt4oqStgEURcQ5wNSvXhq8Hjpe0YTq/W//ucM8umwMcnj4fyVCN8Qbgo+nacZImpvIXKJqij5b0wR5jXlsPAq+TtGt63gRJKyThiHiSojVh8JvjcMzMaqbCpuAZwEMRsSgiXgC+Axzccc52FD/LiYgHgSmSNovC4Lidl6VtxMrL2iTWrwAfTa+EvGot7tPpyXTPC4ATUtmXgH+UdCtFNX7QYcB9qWl0W+Di8o0i4kcUyWtuOueTa/DsslOB4yTdC3wImJnKZwLvkrSAoqn8LaUYnqVoZv2EpIO7xby20jfNYcC5kuYDP2blFgUo/n6zJN1GUYPt/CXCzCyrChPr6ylaDwctSWVl80ldfWnw6hbA5LQ/Lv3M/gPw44j4+UgPU1PXu7O1I2mjwd/CJJ0BbB4RM0e6xn2sBW0wIXcItbHB6/bKHUJtXLHpO3KHUCvv/d2la93l9Pt3vaOnnzmvvfGm/w6cWCqalcaHACDpA8C7I+LDaf9DwIyI+HjpnIkUzb87UYwH2hb4cETML50zCbgS+HhErPJNijH5Hqv15K8l/R3F98AjlJrYzcxqIXrLzeVBlquwBHhDaX8y8FjHPZ4GjgNIrzMuTlv5nKck3QjsB9QrsUo6n5VfFzk7ImaP5WevIp5Xktr1O+wTEX/qcu2VwJYdxX8bEdd1e25EfJfitRwzs1oa6KtsnOWdwFRJW1KM1zmc4pXQl6Ta6PLUnfZh4KaIeFrSq4EXU1LdgOJtk38a6WFZEmtEnJzjubmfPZyUPKet4bXvqzgcM7PaqOod1Yjok3QKcB3FOJ0LI2KhpJPS8QuANwMXS+oH7mdonM3mFK97jqMYl3R5RPxwpOe5KdjMzGopemwK7u1ecS1wbUfZBaXPt1FMhtR53b0U/a49c2I1M7NaquOsSr1wYjUzs1qKgWbOZePEamZmtdTUt0GdWM3MrJYG+mq1ZHjPnFjNzKyWXGM1MzOrkPtYzczMKlTl6zajyYnVzMxqya/bmJmZVah/wIOXzMzMKuM+VjMzswp5VLCZmVmFXGM1MzOr0IBHBZuZmVXHr9uYmZlVqN9NwWZmZtVxjdXMzKxCHhVsY96EqQfkDqEW+gcaOh3MOvDcYzfnDqE2+ub8IHcIY44HL5mZmVXITcFmZmYVco3VzMysQv1OrGZmZtVxU7CZmVmFmjpM0InVzMxqKXCN1czMrDIDfo/VzMysOv14oXMzM7PKuI/VzMysQk3tY21mPdvMzMa8gR63XkjaT9IvJD0k6Yxhjm8i6UpJ90q6Q9L2vV7byYnVzMxqqarEKmkccD6wP7AdcISk7TpO+zQwLyJ2AI4Gzl6Na1fgxGpmZrUUqKetBzOAhyJiUUS8AHwHOLjjnO2AGwAi4kFgiqTNerx2BU6sZmZWS31ST1sPXg88WtpfksrK5gOHAEiaAWwBTO7x2hU4sZqZWS1Fj5ukEyXNLW0ndtxquOzb+ZbsWcAmkuYBHwfuAfp6vHYFHhVsZma11OvApIiYBcwa4ZQlwBtK+5OBxzru8TRwHIAkAYvTtmG3azu5xmpmZrU0IPW09eBOYKqkLSW9HDgcuLp8gqRJ6RjAh4GbUrLtem0n11jNzKyWqprRMCL6JJ0CXAeMAy6MiIWSTkrHLwDeDFwsqR+4HzhhpGtHep4Tq5mZ1VKVMy9FxLXAtR1lF5Q+3wZM7fXakTixmplZLfU44rd2nFjNzKyWGrq4jROrmZnV00AzK6xOrGZmVk9e3cbMzKxCTW0K9nusPZL0eUn7rsF1UyR9sMI4pkl6T2n/oF5WWzAza5o+9bbVjRNrjyLisxHxn2tw6RRg2MQqaU1aDKYBLyXWiLg6Is5ag/uYmdValcvGjaZGJ9ZUG3xA0tclLZR0vaQNJN0oaXo651WSHk6fj5V0laRrJC2WdIqk0yXdI+l2SZuO8KyLJL0/fX5Y0v+UdLekBZK2TeXvkDQvbfdImkAx/+ReqewTKYbvSboGuF7SOyX9sPSc8yQdmz7vKmmOpPlpfcCNgc8Dh6X7HZbud146fwtJN6T1BG+Q9MZS7Oekey0q/T02l3RTutd9kvaq+H+RmdkaC/W21U2jE2syFTg/It4CPAUc2uX87SlqkDOAM4HlEbETcBvFGny9ejwidga+BnwylX0SODkipgF7Ac8BZwA3R8S0iPjndN5uwDERsfeqbp6mzvouMDMidgT2BZ4FPgt8N93vux2XnQdcnNYT/DZwTunY5sCewAEUyZ70dbguxbsjMG+YOF6a3Lq//5nuXxUzs4q4xprP4ogYTAh3UTS9juSnEbEsIv4ILAWuSeULeri27AfDPPNW4H9LOhWYFBF9q7j2xxHxRJf7bwP8NiLuhGKC6BHuN2g34NL0+RKKRDroqogYiIj7gc1S2Z3AcZL+AXhrRCzrvGFEzIqI6RExfdy4jbo83sysOk6s+Txf+txPMdK5j6G/2/gRzh8o7Q+weqOkB68bfCapr/PDwAbA7YNNxMN4tvS5HGs5XrH2g+LK15f/3gKIiJuAtwO/AS6RtDo1djOzdarXZePqZiwk1uE8DOySPr9/tB4qaeuIWBAR/wTMBbYFlgETRrjsEWA7SeunPtR9UvmDwOsk7ZruPSENdhrpfnMoVl4AOBK4pUu8WwB/iIivA98Edu72dzQzGy1NHRU8Vt9j/QpwuaQPAT8ZxeeeJuldFLXY+4H/oKgJ90maD1wEPFm+ICIelXQ5cC/wS4rFdYmIFyQdBpwraQOK/tp9gZ8CZ6TFeP+x4/mnAhdK+hTwR9LagiN4J/ApSS8Cz7B6fcxmZutUHZt5e6GIOlakrY7WH/8Gf7MA/QNN/edeveceuzl3CLXRN+cH3U9qkQ0O+uRa1yW/8sajevqZ88lff6tW9daxWmM1M7OG81zBY4Sk84E9OorPjojZOeIxM2urprYNObF2iIiTc8dgZmb1HPHbCydWMzOrpb6GplYnVjMzq6VmplUnVjMzqyn3sZqZmVXIo4LNzMwqNNDQxmAnVjMzq6X+3AGsISdWMzOrJddYzczMKtTMtOrEamZmNeVRwWZmZhVyU7CZmVmFmplWx+5C52Zm1nD9RE9bLyTtJ+kXkh6SdMYwxzeWdI2k+ZIWSjoulW8jaV5pe1rSaSM9yzVWMzOrpar6WCWNA84H/hJYAtwp6eqIuL902snA/RFxoKRXA7+Q9O2I+AUwrXSf3wBXjvQ811jNzKyWBoieth7MAB6KiEUR8QLwHeDgjnMCmCBJwEbAE0Bfxzn7AL+KiEdGepgTq5mZ1VL0uPXg9cCjpf0lqazsPODNwGPAAmBmRHRWmg8HLuv2MCdWMzOrpV5rrJJOlDS3tJ3YcavhZh3uzMnvBuYBr6No+j1P0sSXbiC9HDgI+F63uN3HamZmtdTrwKSImAXMGuGUJcAbSvuTKWqmZccBZ0VEAA9JWgxsC9yRju8P3B0Rv+8WjxOr9ezySXvlDqEWmvoKwLrQN+cHuUOojb/Y/ZDcIYw5FU4QcScwVdKWFIOPDgc+2HHOryn6UG+WtBmwDbCodPwIemgGBidWMzOrqajo19iI6JN0CnAdMA64MCIWSjopHb8A+AJwkaQFFE3HfxsRjwNI2pBiRPF/7+V5TqxmZlZLVU5pGBHXAtd2lF1Q+vwY8FeruHY58Mpen+XEamZmtTQQzex4cWI1M7NaamZadWI1M7Oa6m/o+jZOrGZmVkvNTKtOrGZmVlNeNs7MzKxCVb1uM9qcWM3MrJbcFGxmZlah8Os2ZmZm1elzU7CZmVl13MdqZmZWIY8KNjMzq5D7WM3MzCrkUcFmZmYV8pSGZmZmFXJTsJmZWYU8eMnMzKxCft3GzMysQl7o3MzMrELNTKtOrGZmVlN9HhVsZmZWHY8KNjMzq1BTRwWvlzsA607SjZKmr8b575W03bqMycxsXYse/6sbJ9ax6b2AE6uZNVpE9LTVjRPrGpI0RdIDkr4uaaGk6yVtUK5dSnqVpIfT52MlXSXpGkmLJZ0i6XRJ90i6XdKmXR55lKQ5ku6TNEPSepJ+KenV6f7rSXpI0tuBg4AvS5onaeu0/UjSXZJulrRtuuYD6X7zJd207r5aZmarb4DoaasbJ9a1MxU4PyLeAjwFHNrl/O2BDwIzgDOB5RGxE3AbcHSXa18REbsDHwMujIgB4FvAken4vsD8iLgJuBr4VERMi4hfAbOAj0fELsAngX9N13wWeHdE7EiRjFci6URJcyXNvW75Q11CNDOrTn8M9LTVjRPr2lkcEfPS57uAKV3O/2lELIuIPwJLgWtS+YIerr0MICXOiZImARcylJCPB2Z3XiRpI2B34HuS5gH/BmyeDt8KXCTpI8C44R4aEbMiYnpETH/3hm/qEqKZWXWa2sfqUcFr5/nS535gA6CPoV9Yxo9w/kBpf4Du/y86v3siIh6V9HtJewNvY6j2WrYe8FRETFvphhEnSXob8NfAPEnTIuJPXeIwMxsVTZ15yTXW6j0M7JI+v7/C+x4GIGlPYGlELE3l36BoEr48IvpT2TJgAkBEPA0slvSBdL0k7Zg+bx0RP4+IzwKPA2+oMF4zs7VSZY1V0n6SfpHGopwxzPGN0xiY+WnczHEdx8elMTE/7PYsJ9bqfQX4qKQ5wKsqvO+T6Z4XACeUyq8GNmLFZuDvAJ9K3wRbU9RkT5A0H1gIHJzO+7KkBZLuA24C5lcYr5nZWhmI6GnrRtI44Hxgf4o3Jo4Y5pXEk4H705iTdwJflfTy0vGZwAO9xO2m4DUUEQ9TDEYa3P9K6fAOpc9/n45fBFxUOn9K6fMKx4Z51jtHCGVHikFLD5bOv5WVX7fZb5j7HjLCfc3MsqpwYNIM4KGIWAQg6TsUFYz7S+cEMEGSKCorT1B07SFpMkWX2ZnA6d0e5sTaYKk546MM37dqZtZoFQ5Mej3waGl/CcW4lLLzKFoAH6PoSjssvX0B8C/A36TyrtwUXCOSzk/vnpa341Z1fkScFRFbRMQtoxmnmdlo6LUpuPxaYNpO7LiVhrl9Z9Z+NzAPeB0wDThP0kRJBwB/iIi7eo3bNdYaiYiTc8dgZlYXvdZYI2IWxfv6q7KEFQdnTqaomZYdB5wVxVROD0laDGwL7AEcJOk9FG96TJT0rYg4alUPc43VzMxqKWKgp60HdwJTJW2ZBiQdTtHsW/ZrYB8ASZsB2wCLIuLvImJyGhdzOPCTkZIquMZqZmY1VdV0hRHRJ+kU4DqKyXAujIiFkk5Kxy8AvkAxYc4Ciqbjv42Ix9fkeU6sZmZWS1VOVxgR1wLXdpRdUPr8GPBXXe5xI3Bjt2c5sZqZWS3VceWaXjixmplZLTV1SkMnVjMzq6U6TrDfCydWMzOrJTcFm5mZVaiOi5j3wonVzMxqqX+gfouY98KJ1czMaslNwWZmZhVyU7CZmVmFXGM1MzOrkN9jNTMzq1CVUxqOJidWMzOrJTcFm5mZVcgzL5mZmVXINVYzM7MKNTWxqqmBWztJOjEiZuWOow78tRjir8UQfy3yWy93AGar6cTcAdSIvxZD/LUY4q9FZk6sZmZmFXJiNTMzq5ATqzWN+46G+GsxxF+LIf5aZObBS2ZmZhVyjdXMzKxCTqxmZmYVcmI1MzOrkBOrNYKkrSWtnz6/U9KpkibljsvykrSHpFekz0dJ+t+StsgdV26SNpG0Q+442sqDl6wRJM0DpgNTgOuAq4FtIuI9OePKQdICWGl28qXAXOB/RcSfRj+qPCTdC+wI7ABcAnwTOCQi3pE1sAwk3QgcRDFV7Tzgj8DPIuL0nHG1kWus1hQDEdEHvA/4l4j4BLB55phy+Q/g/wJHpu0a4Cbgd8BF+cLKoi+K2sHBwNkRcTYwIXNMuWwcEU8DhwCzI2IXYN/MMbWSJ+G3pnhR0hHAMcCBqexlGePJaY+I2KO0v0DSrRGxh6SjskWVxzJJfwccBbxd0jja+33xF5I2B/4b8JncwbSZa6zWFMcBuwFnRsRiSVsC38ocUy4bSXrb4I6kGcBGabcvT0jZHAY8D5wQEb8DXg98OW9I2XyeopvkVxFxp6StgF9mjqmV3MdqjSBpl4i4q6PswIi4JldMuUiaDsxmKJkuA04A7gf+OiIuzxWbmbnGas3xdUlvHdxJzcJ/nzGeUSdpZvq4fkS8FZgG7BQRO0TEnRHxbNuSqqRDJP1S0lJJT0taJunp3HHlIOm/SLpB0n1pfwdJrfo3UheusVojpGatKygG6+wJHA0cEBFLswY2iiTNi4hpku6OiJ1zx1MHkh4CDoyIB3LHkpuknwGfAv4tInZKZfdFxPZ5I2sfD16yRoiIRZIOB64CHgX+KiKeyxzWaHtA0sPAq9NrJoMERES08b3F3zupvmTDiLhDUrmsbX3uteDEarU2zDubmwLjgJ9Lok3JJCKOkPRaigEqB+WOpybmSvouxS9czw8WRsQP8oWUzeOStib9e5H0fuC3eUNqJzcFW611m0UnIh4ZrVjqQtJGwPKIGEj76wHjI2J53shGn6TZwxRHRBw/6sFklrpLZgG7A08Ci4Ej2/hvJDcnVmsESf8VWBgRy9L+BGC7iPh53shGn6TbgX0j4pm0vxFwfUTsnjcyy0nSlulVtFcA60XEssGy3LG1jUcFW1N8DXimtP9sKmuj8YNJFSB93jBjPNlImizpSkl/kPR7Sd+XNDl3XJl8HyCNDl+Wyq7IGE9ruY/VmkJRal6JiAFJbf22c+kIAAAOu0lEQVT+fVbSzhFxNxTv+AJtG8g1aDZwKfCBtH9UKvvLbBGNMknbAm8BNpZ0SOnQRGB8nqjara0/mKx5Fkk6laFa6seARRnjyek04HuSHkv7m1PMQNRGr46Icj/rRZJOyxZNHtsABwCTGJruE4qJQz6SJaKWcx+rNYKk1wDnAHunov8ETouIP+SLKh9JL6P4gSrgwYh4sXTsLyPix9mCG0WS/pNi4YHLUtERwHERsU+2oDKRtGdE3JI7DnNiNRtz2jSBhKQ3AudRzCMdwBxgZhtHwkr6JcVycbOB/wj/cM/GidUaIQ1IORfYg+IH6C0UP0CXZA2shiTdMzjzjrWHipkh9gWOB2YA3wUuioj/lzWwFnJitUaQ9GOKQSqXpKKjKN7Ra80glV61ocYq6W8i4kuSzmXlRd+JiFMzhFUbkt5FsfrTK4D5wBkRcVveqNrDg5esKTxIxcoGpzGcmzWKGpH0SopfOD8E/B74OHA1xWIN3wO2zBdduzixWlM8nhbxLg9S+VPGeOrs4dwBrGul5QKXR8T3ysckfWCYS9rgNooWnfd2dJHMlXRBpphayU3B1ggepLIiSbsDUyj9chwRF2cLKJPhmr3b0BQ+HEmKiJA0kWJax2VdL7J1wjVWa4SI+DWeeB4ASZcAW1OMAO1PxQG0JrFK2h94D/B6SeeUDk2kvSu67JLmTp5AkWefAo6PiLsyx9U6TqxWa6sanDKopYNUplPMk9zm5qbHKPpXDwLKiWMZ8IksEeV3IfCxiLgZivdaKV69ac0KUHXhxGp158EpK7sPeC0tXhIsIuYD8yVdCTwbEf0AksYB62cNLp9lg0kVICJukeTm4Azcx2qN4v4jkPRTipGed7DiGqStayr3Sj8gabA/+UMUizFcRtHKcxjwZER8JldsbeUaqzWCpOkUzVruP4J/yB1Ajay00o+ktq3089WO/c+VPrvmlIETqzWF+4+SiPhZ7hhqpPUr/UTEu3o5T9IxEfHv6zoec1OwNYSkWyNij25lY5mkWyJiz9RvVv6HK4rm8YmZQstG0q7AdygGM0Fa6aelLRkjautrSDk4sVqtuf9o9UnaJCKezB3HaBlppR8b4jmkR48Tq9VaGqizKhERe49wvJXaVDORdPRw5W2cLKObNn1f5OY+Vqs19x+tEeUOYBTtWvo8HtgHuJsWTZaxGtr0fZGVE6uNFTMBJ9ZCa5qhIuLj5X1JGzO0ApKt6NbcAbSFE6uNFf5t3ACWA1NzB5GDpEnA0aw8h/Sp6c9T8kTWPk6sNla0ppbWg9b8kiHpGob+368HbAdcni+irK4FbgcWAAOZY2k1J1YbK9qUTLYGlkTE85LeSfEu78UR8VQ6ZZ9swY2+r5Q+9wGPdCyZ1ibjI+L03EGYRwXbGCHpvLY0dUmaRzER/xTgOorFrLeJiPfkjGu0pXmBr4uIfXPHUgeSPgE8A/yQFae6fCJbUC21Xu4AzHoh6YupD2lwfxNJ/2twvy1JNRmIiD7gfcC/RMQnKCZGaJU08f7yNGDJ4AXgyxQLnt+VNi9ikYGbgq0p9o+ITw/uRMSTkt4D/H3GmHJ5UdIRwDHAgansZRnjyenPwAJJPwaeHSxs6XKCpwNviojHcwfSdk6s1hTjJK0fEc8DSNqA9i4PdhxwEnBmRCyWtCXwrcwx5fJ/01bW1v6thRSjoi0zJ1Zrim8BN0iaTfGD83ja+97qBuUaWUquC3MGlNGkiDi7XCBpZq5gMusH5qXZysp9rG2svWflwUvWGJL2pxjxKoo1N6/LHFIWku4GjomIBWn/COC0iHhb3shG33DT9LV1TlxJxwxX7hnJRp8Tq1nDSNoKuAI4EtiTYlKAAyJiadbARlH6ZeKDFH//m0uHJgJ9HilsObkp2GrNS6WtLCIWSTocuAp4FPiriGjVGqTAHOC3wKtYcaHvZcC9WSLKTNJihulfjoitMoTTaq6xmjWEpAWs+IPzNcBSUn9aRLRu0XdJrwCei4gBSf8F2Bb4jzYuHSfplaXd8cAHgE0j4rOZQmotJ1ZrhB5mGxrzJG0x0vGIeGS0YqkLSXcBewGbUEznNxdYHhFHZg2sJgZbfHLH0TaeIMKa4vtAv6Q3Ad8EtgQuzRvS6IqIR1Ly3Bx4orT/BPDavNFlo4hYDhwCnBsR76OYL7h1JO1c2qZLOgmYkDuuNnIfqzXFQET0SRqcbehcSffkDiqTrwHlkbDPDlPWFpK0G8VArhNSWVt/rn2Voa6CPuBhiuZgG2Vt/Qa05vFsQ0MUpT6c1L/Y1n/LM4G/A66MiIVpxPRPM8eUy/7Aoay4bNzhwOdzBdRWbf3HaM3j2YaGLJJ0KkUtFeBjwKKM8WQTETcBN5X2FwFtnRDhKuAp4G6KqR4tEw9eMmsYSa8BzgH2TkX/STFBxB/yRZVHGgn8SVZe3HvvVV0zVkm6LyK2zx2HObFaQ0iaCvwjxcCU8YPlfkev3STNBy6gWMmlf7A8Iu7KFlQmkmZRDOBakDuWtnNTsDXFbOBzwD8D76JoGm7N4uZlkiYD5wJ7UAxWuQWY2dIFvvsi4mvdT2uFPYFj00QRzzM0iUrr3m/OzTVWawRJd0XELpIWRMRbU9nNEbFX7thGW1oi7VLgklR0FHBkRPxlvqjykPQPwB+AK2n54t6res+5je835+bEao0g6VaKiQCuAH4C/AY4KyK2yRpYBpLmRcS0bmVtkGpnncJdBJaTm4KtKU4DNqQY8fkFioE7w67m0QKPSzoKuCztHwH8KWM82UTElrljMOvkGqs1iqSJFDWSZbljyUXSG4HzgN0o+ljnUPSxtqbJT9LeEfETSYcMdzwifjDaMZkNco3VGkHSdIoBTBPS/lLg+DaO/oyIXwMH5Y4js3dQdAkcOMyxAJxYLRvXWK0RJN0LnBwRN6f9PYF/bdOIR0nnMsyyYIMioq0TI6ySpGO80LeNNtdYrSmWDSZVgIi4Ja3R2iZzcwfQQDMBJ1YbVU6sVmuSBieWv0PSv1EM2AngMODGXHHl0Fnzcn9zT1r5rrPl5aZgqzVJI02oHi2duq7c3yyK+WFb2d/cjaS7I6KNq/5YRq6xWq1FxLt6Oa9lfWkXAh/r6G+eTbH4u63INVYbdV7o3MaKmbkDGEUr9TcDbg4e3q25A7D2cY3VxooxXzNxf/PKJE0Cjmbl1W1OTX+ekicyazMnVhsr2jBY4Ksd+58rfW7D33841wK3AwuAgcyxmAFOrDZ2jPkaq/ubhzU+Ik7PHYRZmftYbaxwX9qQNvU3XyLpI5I2l7Tp4JY7KGs311itEdyXtlrGfO295AXgy8BnGGoOD8Cr21g2TqzWFO5L612b+ltPB94UEY/nDsRskBOrNYX70nrXphrrQmB57iDMypxYrSkukfQR4IfA84OFEfFEvpBqq039zf3AvDRDV/n7wgsSWDae0tAaQdLJwJkU0/e91JcWEa3rS5P0ReBLEfFU2t8E+B8R8fd5Ixt9koZd7L5Fo6KthpxYrREk/Qp4m/vSQNI9EbFTR5nnxDWrCTcFW1O4L23IOEnrR8TzAJI2ANbPHFMWkhYzzGCtNrZkWH04sVpTuC9tyLeAGyTNpkgqx9PeNUenlz6PBz4A+D1Wy8pNwdYI7ktbkaT9gX0oRgBfHxHXZQ6pNiTdEhF75o7D2suJ1cwaq7QwARQzyU0HPhoRO2YKycxNwdYM7ksbqolJWsaKXwtRjJCemCm0nL7K0NeiD3iYojnYLBvXWK0RJL2ytPtSX1pEfDZTSFYDksYDh7LiVJcREZ/PFpS1nhOrNVZb+9IkbQ0siYjnJb0T2AG4ePC91jaR9COKd5vvphjgBkBEdC6xZzZq3BRsjbCKvrQJmcLJ7fvAdElvAr4JXA1cCrwna1R5TI6I/XIHYVbmxGpN4b60IQMR0SfpfcC/RMS5ku7JHVQmcyS9NSIW5A7EbJATqzXF/qzcl3Y40Ma+tBclHQEcAxyYyl6WMZ6c9gSOTYPbnmdoINcOecOyNnNitaa4iqG+tD9njiW344CTgDMjYrGkLSkmjWij/XMHYNbJg5esESTdFxHb547DzKyb9XIHYNajOZLemjuIOpA0VdIVku6XtGhwyx2XmRXcFGxN4b60IbOBzwH/DLyLomm4TYubm9Wam4KtESRtMVx5RDwy2rHkJumuiNhF0oKIeGsquzki9sodm5m5xmoN0cYEOoI/S1oP+KWkU4DfAK/JHJOZJa6xmjWMpF2BB4BJwBeAjYEvRcTtWQMzM8CJ1ayxJE2k6GdeljsWMxviUcFmDSNpuqQFwL3AAknzJe2SOy4zK7jGatYwku4FTo6Im9P+nsC/tnSEtFntuMZq1jzLBpMqQETcArg52KwmPCrYrCFKK/zcIenfgMsoFiY4DLgxV1xmtiI3BZs1hKSfjnA4ImLvUQvGzFbJidVsjJF0TET8e+44zNrKidVsjJF0d0Ts3P1MM1sXPHjJbOzxvMFmGTmxmo09boYyy8iJ1WzscY3VLCMnVrOx59bcAZi1mQcvmTWMpEnA0cAUSu+iR8SpuWIysyGeIMKsea4FbgcWAAOZYzGzDq6xmjWMX6cxqzcnVrOGkfQJ4Bngh8Dzg+UR8US2oMzsJW4KNmueF4AvA59h6NWaALbKFpGZvcQ1VrOGkfQr4G0R8XjuWMxsZX7dxqx5FgLLcwdhZsNzU7BZ8/QD89JqN+U+Vr9uY1YDTqxmzXNV2syshtzHamZmViHXWM0aRtJihploPyI8KtisBpxYzZpneunzeOADwKaZYjGzDm4KNhsDJN0SEXvmjsPMXGM1axxJ5ekM16OowU7IFI6ZdXBiNWuerzLUx9oHPEzRHGxmNeCmYLOGkTQeOJQVl42LiPh8tqDM7CWusZo1z1XAU8DdwJ8zx2JmHVxjNWsYSfdFxPa54zCz4XmuYLPmmSPprbmDMLPhucZq1jCS7gfeBCymmCtYFH2sO2QNzMwAJ1azxpG0xXDlEfHIaMdiZitzYjUzM6uQ+1jNzMwq5MRqZmZWISdWMzOzCjmxmpmZVciJ1czMrEL/H2VayMa9uGjMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(df[[\"num_basic_blocks\", \"num_basic_blocks_in_cfg\", \"num_instructions\", \"num_bytes\"]].corr());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>num_basic_blocks</th>\n",
       "      <th>num_basic_blocks_in_cfg</th>\n",
       "      <th>num_instructions</th>\n",
       "      <th>num_bytes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>16</td>\n",
       "      <td>10</td>\n",
       "      <td>65</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>44</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   num_basic_blocks  num_basic_blocks_in_cfg  num_instructions  num_bytes\n",
       "0                 2                        2                53        147\n",
       "1                 1                        1                42         93\n",
       "2                16                       10                65        157\n",
       "3                 5                        5                44        100\n",
       "4                 1                        1                 2          3"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[[\"num_basic_blocks\", \"num_basic_blocks_in_cfg\", \"num_instructions\", \"num_bytes\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Collect\n",
    " - Collection\n",
    " - Formatting\n",
    " - Cleaning\n",
    " - Normalizing\n",
    " - Feature Engineering\n",
    " \n",
    " ** <B>Data Engineer / Data Architect</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Describe\n",
    " - Measures of central tendency\n",
    " - Measures of variability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Discover\n",
    " - Anomaly / Outlier Detection\n",
    " - Trend Analysis\n",
    " - Dimensionality Reduction\n",
    " - Cluster Analysis\n",
    "\n",
    "** <B>Data Exploration</B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 9: Connect Textual Statements to Underlying Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Why are you even doing this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Start with an analytic research question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rule 10: Provide Public Access to Scripts, Runs, and Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "https://github.com/tclasen/Reproducible-Data-Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary\n",
    "1. For Every Result, Keep Track of How It Was Produced\n",
    "2. Avoid Manual Data Manipulation Steps\n",
    "3. Archive the Exact Versions of All External Programs Used\n",
    "4. Version Control All Custom Scripts\n",
    "5. Record All Intermediate Results, When Possible in Standardized Formats\n",
    "6. For Analyses That Include Randomness, Note Underlying Random Seeds\n",
    "7. Always Store Raw Data behind Plots\n",
    "8. Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected\n",
    "9. Connect Textual Statements to Underlying Results\n",
    "10. Provide Public Access to Scripts, Runs, and Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Questions?\n",
    "\n",
    "** <B>Show me the notebook!<B>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "https://www.amstat.org/asa/files/pdfs/POL-ReproducibleResearchRecommendations.pdf\n",
    "\n",
    "https://data-ken.org/how-to-build-reproducable-data-science-workflow.html\n",
    "\n",
    "https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285#s9"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}