notebook_fresh.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to actually build a neural network from blocks?\n",
    "\n",
    "The webinar is a part of the [Intel Software Student Developer Program for AI](https://software.intel.com/en-us/experts/ambassadors).\n",
    "\n",
    "## Webinar\n",
    "\n",
    "* sign up: https://www.crowdcast.io/e/neural-network-blocks/register (6 July 2017)\n",
    "* speaker: Jakub Czakon, moderator: Dr. Piotr Migdał\n",
    "\n",
    "## Installation\n",
    "\n",
    "* Python 3 with Jupyter Notebook and respective libraries (the easiest way is with [Anaconda distribution](https://www.continuum.io/downloads)).\n",
    "* [Keras](https://keras.io/) (2.x) with [TensorFlow](https://www.tensorflow.org/) backend.\n",
    "* [keras-sequential-ascii](https://github.com/stared/keras-sequential-ascii) for diagrams\n",
    "\n",
    "```bash\n",
    "$ pip install tensorflow\n",
    "$ pip install keras\n",
    "$ pip install git+git://github.com/stared/keras-sequential-ascii.git\n",
    "$ wget http://yaroslavvb.com/upload/notMNIST/notMNIST_small.mat\n",
    "```\n",
    "\n",
    "## Data\n",
    "\n",
    "Data source: [notMNIST](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html) (you need to download `notMNIST_small.mat` file):\n",
    "\n",
    "> some publicly available fonts and extracted glyphs from them to make a dataset similar to MNIST. There are 10 classes, with letters A-J taken from different fonts.\n",
    "\n",
    "> Approaching 0.5% error rate on notMNIST_small would be very impressive. If you run your algorithm on this dataset, please let me know your results.\n",
    "\n",
    "## More info\n",
    "\n",
    "For additional information, including [some context for notMNIST](http://p.migdal.pl/2017/04/30/teaching-deep-learning.html#notmnist), see [Learning Deep Learning with Keras](http://p.migdal.pl/2017/04/30/teaching-deep-learning.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "from scipy import io\n",
    "import numpy as np\n",
    "from keras.utils import np_utils\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from keras_sequential_ascii import sequential_model_to_ascii_printout\n",
    "from live_loss_plot import PlotLosses\n",
    "\n",
    "# Keras layers\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Flatten\n",
    "from keras.layers import Conv2D, MaxPool2D, Dropout, BatchNormalization, GlobalMaxPool2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load data\n",
    "data = io.loadmat(\"../resources/notMNIST_small.mat\")\n",
    "\n",
    "# transform data\n",
    "X = data['images']\n",
    "y = data['labels']\n",
    "resolution = 28\n",
    "classes = 10\n",
    "\n",
    "X = np.transpose(X, (2, 0, 1))\n",
    "\n",
    "y = y.astype('int32')\n",
    "X = X.astype('float32') / 255.\n",
    "\n",
    "# channel for X\n",
    "X = X.reshape((-1, resolution, resolution, 1))\n",
    "\n",
    "# 3 -> [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]\n",
    "Y = np_utils.to_categorical(y, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# random letters\n",
    "rows = 6\n",
    "fig, axs = plt.subplots(rows, classes, figsize=(classes, rows))\n",
    "for letter_id in range(10):\n",
    "    letters = X[y == letter_id]\n",
    "    for i in range(rows):\n",
    "        ax = axs[i, letter_id]\n",
    "        ax.imshow(letters[np.random.randint(len(letters)),:,:,0],\n",
    "                  cmap='Greys', interpolation='none')\n",
    "        ax.axis('off')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# splitting data into training and test sets\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y,\n",
    "                                                    test_size=0.20,\n",
    "                                                    random_state=137)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_losses = PlotLosses(figsize=(8, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Flatten(input_shape=(resolution, resolution, 1)))\n",
    "model.add(Dense(classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "sequential_model_to_ascii_printout(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# train model\n",
    "model.fit(X_train, Y_train,\n",
    "          epochs=10,\n",
    "          batch_size=32,\n",
    "          validation_data=(X_test, Y_test), callbacks=[plot_losses])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# example predictions\n",
    "predictions = model.predict(X_test)\n",
    "\n",
    "rows = 5\n",
    "fig, axs = plt.subplots(rows, 2, figsize=(8, 1.5 * rows))\n",
    "for i in range(rows):\n",
    "    ax = axs[i,0]\n",
    "    idx = np.random.randint(len(X_test))\n",
    "    ax.imshow(X_test[idx,:,:,0],\n",
    "              cmap='Greys', interpolation='none')\n",
    "    ax.axis('off')\n",
    "        \n",
    "    pd.Series(predictions[idx], index=list(\"ABCDEFGHIJ\")).plot('bar', ax=axs[i,1], ylim=[0,1])\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: Grid Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid_plot import score_model_grid, plot_complexity, plot_loss_grid, plot_acc_grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "webinar",
   "language": "python",
   "name": "webinar"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}