0-Graph-Eager-Mode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Eager Mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything is eager by default in tensorflow 2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=2, shape=(2,), dtype=float64, numpy=array([3., 7.])>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([1., 2.])\n",
    "b = np.array([2., 5.])\n",
    "\n",
    "tf.add(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph Model : AutoGraph\n",
    "As tensorflow suggest (https://www.youtube.com/watch?v=jh4ITuOytE4), we can use Autograph to automatically convert  eager function to graph function by using the tf.function decorator\n",
    "\n",
    "The following code first compiles inside the function into graph code before to execute it. Note that if you remove the @tf.function decorator the code will work perfectly but will not be executed in graph mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([3. 7.], shape=(2,), dtype=float64)\n"
     ]
    }
   ],
   "source": [
    "@tf.function\n",
    "def add_fc(a, b):\n",
    "    return tf.add(a, b)\n",
    "\n",
    "print (add_fc(a, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code show the code generated to create the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def tf__add_fc(a, b):\n",
      "  do_return = False\n",
      "  retval_ = ag__.UndefinedReturnValue()\n",
      "  do_return = True\n",
      "  retval_ = ag__.converted_call('add', tf, ag__.ConversionOptions(recursive=True, force_conversion=False, optional_features=(), internal_convert_user_code=True), (a, b), None)\n",
      "  cond = ag__.is_undefined_return(retval_)\n",
      "\n",
      "  def get_state():\n",
      "    return ()\n",
      "\n",
      "  def set_state(_):\n",
      "    pass\n",
      "\n",
      "  def if_true():\n",
      "    retval_ = None\n",
      "    return retval_\n",
      "\n",
      "  def if_false():\n",
      "    return retval_\n",
      "  retval_ = ag__.if_stmt(cond, if_true, if_false, get_state, set_state)\n",
      "  return retval_\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def add_fc(a, b):\n",
    "    return tf.add(a, b)\n",
    "\n",
    "print(tf.autograph.to_code(add_fc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph mode and eager mode  with a keras model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten\n",
    "model = tf.keras.models.Sequential()\n",
    "# Add layers\n",
    "model.add(tf.keras.layers.Dense(256, activation=\"relu\"))\n",
    "model.add(tf.keras.layers.Dense(128, activation=\"relu\"))\n",
    "model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling model.predict you get a numpy array but the output is not differentiable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_output = model.predict(np.zeros((1, 30)))\n",
    "model_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling model(), eager execution is used and the output is differentiable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=161, shape=(1, 10), dtype=float32, numpy=array([[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]], dtype=float32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_output = model(np.zeros((1, 30)))\n",
    "model_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, you can execute the model in graph mode by using the @tf.function decorator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]], shape=(1, 10), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "@tf.function\n",
    "def predict(x):\n",
    "    return model(x)\n",
    "\n",
    "model_output = predict(np.zeros((1, 30)))\n",
    "print(model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}