{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Cooking a simple neural network library\n", "\n", "##### written by Alexandre Boucaud – " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mise en bouche\n", "\n", "To get started, let's introduce a canonical problem in ML called XOR and solve it with the Keras library." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### XOR\n", "\n", "XOR is a logical gate (like AND or OR) that takes to binary inputs and outputs the following values\n", "\n", "![](https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Ftse3.mm.bing.net%2Fth%3Fid%3DOIP.ZfIfrU13bRMppqcMi3nTRgHaEK%26pid%3DApi&f=1)\n", "\n", "Such gate is interesting because there are no linear combinations of the input values that can obtain all of the outputs. This introduces the need for non-linearities in algorithms, for instance activation layers in the case of neural networks.\n", "\n", "\n", "A way to code the problem in supervised learning is to define the input and output vectors $X$ and $y$ such as\n", "\n", "```\n", "[0, 0] => 0 \n", "[0, 1] => 1 \n", "[1, 0] => 1 \n", "[1, 1] => 0 \n", "```\n", "\n", "Because of the extremely small size of the dataset, we will **forget** about the prescriptions on _training, validation and test sets_ for this example, which **you shouldn't do in practice**. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Imports and set up" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])\n", "y = np.array([[0], [1], [1], [0]])" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "def print_xor_results(X, y_true, y_pred):\n", " print('\\nX => y => y_pred => round(y_pred)')\n", " for a, b, c in zip(X, y_true, y_pred):\n", " print(f'{a} => {b} => {c} => {c.round()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To help visualise the final decisions to which the network has converged, the decision contours can be drown from it using a grid of parameters in the [0, 1] plane in 2D." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "# Create an array of points to plot the decision regions\n", "bounds = [0, 1, 0, 1]\n", "x_min, x_max, y_min, y_max = bounds\n", "rows, cols = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]\n", "X_grid = np.c_[rows.ravel(), cols.ravel()]" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "y_grid_true = np.zeros_like(rows)\n", "y_grid_true[np.logical_and(rows > 0.5, cols < 0.5)] = 1\n", "y_grid_true[np.logical_and(rows < 0.5, cols > 0.5)] = 1" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def plot_decision_contours(predictions):\n", " plt.figure(figsize=(5, 5))\n", " # Plot decision region\n", " plt.pcolormesh(rows, cols, predictions > 0.5, cmap='Paired')\n", " plt.grid(False)\n", " # Plot decision boundaries\n", " plt.contour(rows, cols, predictions, \n", " levels=[.25, .5, .75],\n", " colors=['k', 'k', 'k'], \n", " linestyles=['--', '-', '--'])\n", " \n", " plt.xlim(x_min, x_max)\n", " plt.ylim(y_min, y_max)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEzCAYAAABJzXq/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGltJREFUeJzt3XlwHOed3vHvb2ZwEwBBAjwBErwl6liJpmQda1nWYZFaW1zHXhfpcrzOKmZ5s9pKlbeS0pZTypZc+WOzFW/VppRsuLUqr51YWtlJHNqmROu0bEmUSEsUxVvgCfACCeI+B8CbP/o1DcyAwoicQTfYz6eK1TPzNgZPY4bP9Ex3T5tzDhERgUTYAUREokKFKCLiqRBFRDwVooiIp0IUEfFUiCIi3qSFaGZPm1mrme29zLiZ2d+ZWZOZ7TGzNfmPKSJSeLmsIX4PWPcR4+uBFf7fZuC/X30sEZGpN2khOudeBy5+xCwbgO+7wA5gppnNz1dAEZGpko/PEBcCzWOut/jbRESmldRU/jIz20zwtprSpH1iYXXJVP56mYZae9J0D41QUlrGgiXLw44j08CxAx9ccM7VXcnP5qMQTwENY67X+9uyOOe2AFsAls8uc999qDEPv16uZf/17TO8dLST+uXX8Z3vbw07jkwDX1nTcOJKfzYfb5m3Al/zW5vvADqdc2fycL8iIlNq0jVEM3sGuBeoNbMW4D8CRQDOub8HtgEPA01AH/CvChVWRKSQJi1E59ymScYd8Gd5SyQiEhIdqSKRtrQm2PC2dPXNISeROFAhSqTdMKc8mN52V8hJJA5UiBJpwyPBN7qnh4ZCTiJxoEKUSHu+qQOAF555OuQkEgcqRBERT4UoIuKpEEVEPBWiiIinQpRIWzm7NJj+3idCTiJxoEKUSFtVWxZMb7kt5CQSBypEibSB9CgA/b09ISeROFAhSqS9eLQTgJd+/D9DTiJxoEIUEfFUiCIingpRRMRTIYqIeCpEibTVdcFuN6vX3hFyEokDFaJE2rJZwY7Zy264JeQkEgcqRIm0nsGRYNrZEXISiQMVokTaq8e7gulPng05icSBClFExFMhioh4KkQREU+FKCLiqRAl0m6eG5yG9OY77wk5icSBClEibfHM4ET1i1euDjmJxIEKUSKtY2A4mF5oDTmJxIEKUSLtVye6g+nP/0/ISSQOVIgiIp4KUUTEUyGKiHgqRBERT4UokXbr/GA/xDWfuj/kJBIHKkSJtPqqYD/EhUtXhJxE4kCFKJF2oS8dTM+cCjmJxIEKUSLtrebgBPVv/eKnISeROFAhioh4KkQREU+FKCLiqRBFRDwVokTabQsrgul9D4WcROJAhSiRNm9GcTBtWBJyEokDFaJE2tmeoWDafCzkJBIHKkSJtJ2neoPpK9tDTiJxoEIUEfFUiCIiXk6FaGbrzOyQmTWZ2eMTjC8ys1fN7D0z22NmD+c/qohIYU1aiGaWBJ4C1gOrgU1mlnkKtP8APOecuxXYCPy3fAcVESm0XNYQbweanHNHnXNDwLPAhox5HFDlL1cDp/MXUeLszoYZwfSznw85icRBKod5FgLNY663AJ/MmOevgF+Y2Z8DFcADE92RmW0GNgPUlefyqyXuasuLgun8hSEnkTjI10aVTcD3nHP1wMPAD8ws676dc1ucc2udc2urSlWIMrmWrkEATh39MOQkEge5FOIpoGHM9Xp/21iPAs8BOOfeAkqB2nwElHh770wfAO/+6uWQk0gc5FKIO4EVZrbEzIoJNppszZjnJHA/gJldT1CI5/MZVESk0CYtROfcMPAYsB04QLA1eZ+ZPWlmj/jZ/gL4hpm9DzwDfN055woVWkSkEHL6IM85tw3YlnHbE2Mu7wfuzm80EZGppSNVREQ8FaJE2qcWVwbTP/gXISeROFAhSqTN9LtnzaydE3ISiQMVokTaiY5gP8QTh/eHnETiQIUokbbnXLAf4p63Xg85icSBClFExFMhioh4KkQREU+FKCLiqRAl0j7TGHzN5mf+cGPISSQOVIgSaTNKksG0embISSQOVIgSaUcuDgTTfbtDTiJxoEKUSNt/vj+Y7toRchKJAxWiiIinQhQR8VSIIiKeClFExFMhSqQ9uLQagAe+9NWQk0gcqBAl0kqLgqdoWcWMkJNIHKgQJdIOXQh2uzm0e2fISSQOVIgSaYfbgh2zD7//m5CTSByoEEVEPBWiiIinQhQR8VSIIiKeClEibf3y4Gu/1m36k5CTSByoECXSUkkDoKi4OOQkEgcqRIm0fa3BaUj37Xwz5CQSBypEibSj7cGJ6o/u3xNyEokDFaKIiKdCFBHxVIgiIp4KUUTEUyFKpH1+VU0w/eNvhpxE4kCFKCLiqRAl0t4/2xtM33wt1BwSDypEibSTnUPB9MODISeROFAhioh4KkQREU+FKCLiqRAl0pL+GZoqSoUbRGJBhSiR9vCKYD/E9V/51yEnkThQIYqIeCpEibR3T/cE01++FHISiQMVokTaqe50MD3eFHISiYOcCtHM1pnZITNrMrPHLzPPl81sv5ntM7Mf5jemiEjhTbrpzsySwFPAg0ALsNPMtjrn9o+ZZwXwl8Ddzrl2M5tTqMAiIoWSyxri7UCTc+6oc24IeBbYkDHPN4CnnHPtAM651vzGFBEpvFwKcSHQPOZ6i79trJXASjN7w8x2mNm6fAWUeCtJBWfdKy0rDzmJxEG+NqqkgBXAvcAm4B/MbGbmTGa22cx2mdmuroHhPP1quZZ9dlnwNHrwy18LOYnEQS6FeApoGHO93t82Vguw1TmXds4dAw4TFOQ4zrktzrm1zrm1VaU68kBEoiWXQtwJrDCzJWZWDGwEtmbM8xOCtUPMrJbgLfTRPOaUmHqnJdgP8Z2Xnw85icTBpIXonBsGHgO2AweA55xz+8zsSTN7xM+2HWgzs/3Aq8C/c861FSq0xMe53mA/xHMtJ0JOInGQ0/tW59w2YFvGbU+MueyAb/l/IiLTko5UERHxVIgiIp4KUSKtojh4ilZUVYecROJAhSiRdt+SoAjv+8KmkJNIHKgQRUQ8FaJE2psnu4Pp9v8XchKJAxWiRFpbf3CIZ9vZMyEnkThQIYqIeCpEERFPhSgi4qkQJdKqS5LBdFZtyEkkDlSIEmn3NFYF089/KeQkEgcqRBERT4Uokfb68a5g+tMfh5xE4kCFKJHWOTgSTC9eCDmJxIEKUUTEUyGKiHgqRBERT4UokTa7LDjLxex580NOInGgQpRIu2tRZTB9aEPISSQOQjs58tDwKEcvDly6Pq+yiPKiJD1DI7T2pLPmn19ZTFlRgq7BES70Zo8vrCqmJJWgc2CYtr7hrPGG6mKKkgk6+oe52J89vmhmCamE0daXpnNgJGu8saaEhBkX+tJ0TTC+dFYpAK29aXoGx48nDBprgvFzPUP0Do2OG08mjMUzSwA40z1Ef3r8eFHSaKgOxk91DTE4PH68OGXUVwXjzZ2DpEfcuPGyogTzK4sBONk5yHDGeHlxgnkzgvHjHQOMjr97ZpQkmVNRBMDR9gEY/+NUliSpqyjCOcex9sHMPw3VpUlmlxcxMuo40ZE9XlOWoqYsRXrE0dw5fry1dwgItjIfP7h33Nic+sWUz6ikp6uDC6dbsu533uKllJaV093RTtvZzFOJw4LGZRSXltF58QLtrWezxuuXrSRVVEzHhVY6LrRmjTcsv45kKsXF1jN0Xcw+yeTiVTdgZrSdPU13x8Xxg2Y0rroBgPOnW+jt6hg3nEgmWbTieiA442B/T/e48VRREfXLVgFw9uQxBvp6x40XFZewcGlwavTTx48wNNA/brykrJz5i5cCcOroh6SHxv/dS8srmLdoCQDNTQcZGR7/f6a8soo5CxcBcPLwAUZHxz/nK6pmUregHiDrcQOorJnN7LnzGR0d5eTh/Vnj1bNrqambx3A6TcuRQ1njNXVzqZ5dR3pokFNHP8wav1IWnDBv6pnZuF9cXZKkojhBz+AIXRmFAVBTmqSsKEH34AjdE4zPKktRmjI6B0boTWeP15anKE4aHf0j9A1nj8+pSJFKGO39w/QPZ/9N5lWkSCSMtr5hBkeyxxdUBoVxoW+YoY8YP9+bJjOeAfP9eGtvmsx4CYN5M4Lxcz1pMu8+aTDXj5/tSTOaMZ5KcKnQznSnM/uMogTUfcR4ccKorQheO093Z78YlSSN2eUpRkcdZ3uzX2xKU8asshQjo45zE4yXpRLUlCVJjzjOT/BiFjAym7hqVi3lMyrp7erMLhxgZu1cSsvL6elopyejcABmzZlHcWkZXe1t9HV3ZY3PnreQouJiOtvO09/bkzVet6CBZCpF+4VzDPb1ZY3PqV9MIpGgvfUsgxmFBFwqnIvnzjA0ODB+0Ix5DY0AXDhziuH0UMZwgrkNiwE4f6aFkfT4xyWRSDCn3o+famZkZPzfNZlMUbewAQgK12W8CiaLiqibHxTauebjZPZEqriE2nkLADjbfBwyxotLSpk1N/iY4+zJY1nLXlJWRk3dPEZHR2md4BSzpeUVzKydw0g6zfkz2S92ZTNmUD2rjvTQ0EQvdr9xzq3N+qEchLaGmGnOqt+jbkED55pP0LVvd9b43NW3MWvOXE4da6L70L6s8QU33UnVrFmc/PAgvRO8oiy89R4qKis5dnAvfcePZI033HYfJaVljOzdTf8ED9DiOx8imUox+P4uBs9kr2003v0wAP2/eZuh8+PXNhLJ5KXx3p1v0Nk2fp+6VFExjXevB6Drrdfp6WwfP15SRuPdnwWg/devZK0tFJfPoPHu+wFoe+1FBgfG/+csmTGTxrs+DcD5V17IWhsomzmbxtt/H4CzL/4cl/Gfp3z2HBo/cQcAp7dvzXryz6hbQOMtaxkZGeHsiz/L+ttUzWug8cZbGRoY4Nxr27PGZy5spPH6m+jr6eb8r18ZN5ZMFTEynCaVcFkvFAsSPcyxIU4nhhj/FwnUWxezrJ/mxCDZdQaLEp1UWR/HEwOcnGC80TqosCRNiUGy6wyWWjslluCwpclev4QV1kbSEhxIpDmfMWbASgvWKvfaMJl1nsRdGu9PjNCZMZ6y34332EjW8hUnfjfelRilL+NNTUli9NJ4uzky19vLbeTSeJtBOuNVckZi+NL4eRyZ75nGjk/0t6myNCutjREbJXvdG2YmhlhpbQwmRrL+dgCzbJBl1kavjZDPE8CHtoa4dPXN7j/9r22TzyixduLwfv5y40P8+7vnc/cinWhKJrfhmYNXvIaojSoSaWY2bipSSCpEibQXn/s+AD891D7JnCJXT4UokfbbrZvDmVuKRApAhSgi4qkQRUQ8FaKIiKdClEirX7YSgEZ/JI9IIakQJdJu/GSww/it8ytCTiJxoEIUEfFUiBJpL/zwaQD+74HsY5VF8k2FKCLiqRBFRDwVooiIp0IUEfFUiBJpjdcF3yq93H8juUghqRAl0q5b80kAbppbHnISiQMVokTasP9q/MzzxIgUggpRIu2lH/0AgJ8d1vchSuGpEEVEPBWiiIinQhQR8XIqRDNbZ2aHzKzJzB7/iPm+aGbOzK7ojFciImGatBDNLAk8BawHVgObzGz1BPNVAv8WeDvfISW+lt10CwDX1ZaFnETiIJc1xNuBJufcUefcEPAssGGC+b4D/DUwkMd8EnMrbloDwPV1KkQpvFwKcSHQPOZ6i7/tEjNbAzQ4536ex2wiDPT1AdCfHg05icTBVW9UMbME8F3gL3KYd7OZ7TKzXd3t+n47mdxrP3kWgBeaOkJOInGQSyGeAhrGXK/3t/1WJXAj8JqZHQfuALZOtGHFObfFObfWObe2smbWlacWESmAXApxJ7DCzJaYWTGwEdj620HnXKdzrtY51+icawR2AI8453YVJLGISIFMWojOuWHgMWA7cAB4zjm3z8yeNLNHCh1QRGSqpHKZyTm3DdiWcdsTl5n33quPJSIy9XSkikTaqltvA+DGOfr6Lyk8FaJE2pLrbwJgxWx9QawUngpRIq23qxOA7sGRkJNIHKgQJdJ+9bP/DcBLRztDTiJxoEIUEfFUiCIingpRRMRTIYqIeCpEibQbbr8LgFvmaT9EKTwVokRaw/LrAFhSo/0QpfBUiBJpnRcvANDePxxyEokDFaJE2lsvBF+s9NrxrpCTSByoEEVEPBWiiIinQhQR8VSIIiKeClEi7ea7Pg3A2gUVISeROFAhSqQtaFwGQEN1SchJJA5UiBJpbefOAHC+Nx1yEokDFaJE2s6Xnwfg1ye7Q04icaBCFBHxVIgiIp4KUUTEUyGKiHgqRIm0Nfc8AMAd9TNCTiJxoEKUSJtTvwiA+ZXFISeROFAhSqS1tpwE4Ez3UMhJJA5UiBJp777+EgA7WnpCTiJxoEIUEfFUiCIingpRRMRTIYqIeCpEibTb7l8PwO8vqgw5icSBClEibfbc+QDUVRSFnETiQIUokXb6+BEAmjsHQ04icaBClEjb8+YvAdh1ujfkJBIHKkQREU+FKCLiqRBFRDwVooiIp0KUSLtz3SMA3NtYFXISiQMVokRa9axaAGrKUiEnkThQIUqkNTcdBOBY+0DISSQOVIgSafveeROA3Wf7Qk4icaBCFBHxVIgiIl5OhWhm68zskJk1mdnjE4x/y8z2m9keM3vZzBbnP6qISGFNWohmlgSeAtYDq4FNZrY6Y7b3gLXOuZuBHwP/Od9BRUQKLZc1xNuBJufcUefcEPAssGHsDM65V51zv/3UewdQn9+YElef+twXAXhgaXXISSQOcinEhUDzmOst/rbLeRR4fqIBM9tsZrvMbFd3+8XcU0psVVQFRVhZkgw5icRBXjeqmNlXgbXA30w07pzb4pxb65xbW1kzK5+/Wq5Rxw58AMCHbdoPUQovl93/TwENY67X+9vGMbMHgG8Dn3bO6ds8JS8OvbcTgL2t2g9RCi+XNcSdwAozW2JmxcBGYOvYGczsVuB/AI8451rzH1NEpPAmLUTn3DDwGLAdOAA855zbZ2ZPmtkjfra/AWYAPzKz3Wa29TJ3JyISWTkdMe+c2wZsy7jtiTGXH8hzLhGRKacjVUREPBWiRNq9f7gRgHXLZ4acROJAhSiRVlpeDkBZkZ6qUnh6lkmkffjBuwAcON8fchKJAxWiRNqRD3YDcPCCClEKT4UoIuKpEEVEPBWiiIinQhQR8VSIEmkP/NG/BOBzK2tCTiJxoEKUSEsVFQFQlLSQk0gcqBAl0g6++zYAH5zT139J4akQJdKOH9wHQNNFfUGsFJ4KUUTEUyGKiHgqRBERT4UoIuKpECXS1n3lTwD4wvU6S6MUngpRRMRTIUqk7X371wC8d6Y35CQSBypEibSWI4cBON6hU31L4akQRUQ8FaKIiKdCFBHxVIgSaclUCoBUQt92I4WnQpRIe/DLXwPg86v0fYhSeCpEERFPhSiRtvvNVwHYeaon5CQSBypEibSzx48B0NI1FHISiQMVooiIp0IUEfFUiCIingpRIq24rAyA0pSeqlJ4epZJpN33hU0ArF8xM+QkEgcqRBERT4UokfabX/4CgLeau0NOInGgQpRIO3+qBYCzPemQk0gcqBBFRDwVooiIp0IUEfFUiBJp5VVVAMwo1lNVCk/PMom0ez73JQAeXKb9EKXwVIgiIp4KUSLt7Ze2AfCrE10hJ5E4UCFKpLW3ngXgQt9wyEkkDnIqRDNbZ2aHzKzJzB6fYLzEzP7Zj79tZo35DioiUmiTFqKZJYGngPXAamCTma3OmO1RoN05txz4W+Cv8x1URKTQcllDvB1ocs4ddc4NAc8CGzLm2QD8k7/8Y+B+M9N5I0VkWsmlEBcCzWOut/jbJpzHOTcMdAKz8xFQ4q2mbi6JRJK68qKwo0gMpKbyl5nZZmCzvzr4lTUNe6fy90+xWuBC2CEKaEqX743mbjY8c3Cqfp0eu+lt1ZX+YC6FeApoGHO93t820TwtZpYCqoG2zDtyzm0BtgCY2S7n3NorCT0daPmmr2t52SAey3elP5vLW+adwAozW2JmxcBGYGvGPFuBP/aXvwS84pxzVxpKRCQMk64hOueGzewxYDuQBJ52zu0zsyeBXc65rcA/Aj8wsybgIkFpiohMKzl9huic2wZsy7jtiTGXB4A/+pi/e8vHnH+60fJNX9fysoGW77JM72xFRAI6dE9ExCt4IV7rh/3lsHzfMrP9ZrbHzF42s8Vh5LwSky3bmPm+aGbOzKbVlstcls/Mvuwfv31m9sOpzng1cnhuLjKzV83sPf/8fDiMnFfCzJ42s1Yzm3DXPQv8nV/2PWa2Jqc7ds4V7B/BRpgjwFKgGHgfWJ0xz78B/t5f3gj8cyEzhbB8nwHK/eU/nS7Ll8uy+fkqgdeBHcDasHPn+bFbAbwH1Pjrc8LOnefl2wL8qb+8Gjgedu6PsXz3AGuAvZcZfxh4HjDgDuDtXO630GuI1/phf5Mun3PuVedcn7+6g2A/zukgl8cO4DsEx64PTGW4PMhl+b4BPOWcawdwzrVOccarkcvyOaDKX64GTk9hvqvinHudYI+Wy9kAfN8FdgAzzWz+ZPdb6EK81g/7y2X5xnqU4FVrOph02fzbkAbn3M+nMlie5PLYrQRWmtkbZrbDzNZNWbqrl8vy/RXwVTNrIdiL5M+nJtqU+Lj/N4EpPnQvzszsq8Ba4NNhZ8kHM0sA3wW+HnKUQkoRvG2+l2DN/nUzu8k51xFqqvzZBHzPOfdfzOxOgn2Jb3TOjYYdLCyFXkP8OIf98VGH/UVULsuHmT0AfBt4xDk3OEXZrtZky1YJ3Ai8ZmbHCT6n2TqNNqzk8ti1AFudc2nn3DHgMEFBTge5LN+jwHMAzrm3gFKC45yvBTn938xS4A8+U8BRYAm/+2D3hox5/ozxG1WeC/sD2zwv360EH26vCDtvvpctY/7XmF4bVXJ57NYB/+Qv1xK8BZsddvY8Lt/zwNf95esJPkO0sLN/jGVs5PIbVf6A8RtV3snpPqcg9MMEr6xHgG/7254kWFuC4FXpR0AT8A6wNOw/dJ6X7yXgHLDb/9saduZ8LVvGvNOqEHN87IzgY4H9wAfAxrAz53n5VgNv+LLcDXw27MwfY9meAc4AaYI1+UeBbwLfHPPYPeWX/YNcn5s6UkVExNORKiIingpRRMRTIYqIeCpEERFPhSgi4qkQRUQ8FaKIiKdCFBHx/j8zzoNN7IUWCwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plot_decision_contours(y_grid_true)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise #0 - Solving this problem using Keras\n", "\n", "***10 min*** - Based on the `Keras` examples given in the [lecture](https://aboucaud.github.io/slides/2020/neural-networks-mml), solve the XOR problem using `Keras` methods." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "# Write down the keras model below\n", "#---------------------------------\n", "# Star by the necessary imports\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense\n", "from tensorflow.keras.layers import Activation\n", "from tensorflow.keras.optimizers import SGD\n", "\n", "# Write the model architecture\n", "model = Sequential()\n", "model.add(Dense(4, input_dim=2))\n", "model.add(Activation('tanh'))\n", "model.add(Dense(1))\n", "\n", "# Compile the model\n", "model.compile(loss='mse', optimizer='sgd')\n", "\n", "# Train the model (no validation_split required here)\n", "model.fit(X, y, epochs=2000, verbose=0)\n", "#---------------------------------\n", "\n", "# Once trained, this will then predict the values (equivalent of `.forward()`)\n", "y_pred_keras = model.predict(X)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "X => y => y_pred => round(y_pred)\n", "[0 0] => [0] => [0.12707123] => [0.]\n", "[1 0] => [1] => [0.8336644] => [1.]\n", "[0 1] => [1] => [0.83080935] => [1.]\n", "[1 1] => [0] => [0.24702188] => [0.]\n" ] } ], "source": [ "print_xor_results(X, y, y_pred_keras)" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEzCAYAAABJzXq/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8VFX+//HXmZbeew8ISFGkREBUQAjYVrGXVdfCin3dVTpKlyaK7oqruOuKrmvZtSy76s8AUix0VJQqJZWQhHTSy/n9kQnfyAoZyCR3ZvJ5Ph4+HpnkMvO5Jnnnzj3nfI7SWiOEEAJMRhcghBCuQgJRCCHsJBCFEMJOAlEIIewkEIUQwk4CUQgh7FoNRKXU60qpfKXUj6f4ulJK/VEpdUAptVMpNcD5ZQohRPtz5ArxDeCK03z9SqC7/b/xwJ/bXpYQQnS8VgNRa70BKDrNIWOBN3WTTUCwUirGWQUKIURHccY9xDggq8XjbPvnhBDCrVg68sWUUuNpeluNt1kNjAvy6siXb1flNQ3kV9QR7G0mzMcCShldkmgLrckqq6WhUROd3B2bt7fRFYlWFBfkUXIsH+CY1jribJ7DGYGYAyS0eBxv/9z/0FovB5YDdAvz0c9fnuyEl3cNDY2aV7fn8fmBEgbHB/DAwCjMJglFd5ZVWsOsdVkU5efyxPN/pc+FQ40uSZxGbU01K19fxoevvZBxts/hjLfMK4Hf2EebhwClWutcJzyvWzGbFA+lRHFjr1A+P1DC0o1HqGuQxhnuLCHIi0Wjkwg317Lo0bvYsuZTo0sSp2Hz8uamh55s03M4Mu3mHWAjcK5SKlspNU4p9aBS6kH7IZ8Ch4ADwGvAw22qyI0ppfhNv0juviCCzTnHySitMbok0UbhvlYWpCbRLcjMixMfYMP4FKNLEu1IGdX+y9PeMp+soKKOCD8r0PR2Wt4+u7ea+kYWf53DtiMV3PjAE9ww/vcouU/skn49IGG71vqs/nLJSpV20hyGXxwqZWJaOiXV9QZXJNrCy2Ji6qXxjOwSyAevPs/fFk6nsaHB6LKEk0kgtrNgbzNZZbVMXZ1JQUWd0eWINrCYFL8bHMMNvUJZ/c+3WHbD+dTVym0RTyKB2M4GxPoz+7IESqvrmbI6g5yyWqNLEm2glOLufpHc1z+SjVnlLLm2L5XlZUaXJZxEArED9I7wZd6oROoaNFNXZ1BWI2+f3d3YnqH84aIYdhdUMnf8Lc3z34Sbk0DsIF1DvFmQmsTNfcII9OrQ+fCinYxIDuKpYfEcPbCbmfdez9HMw0aXJNpIArEDxQXauObcUAD2F1ax/chxgysSbTUg1p95oxKpLshh9q2XcXjPD0aXJNpAAtEgb+8s4JkN2WzIkPtP7q5HmA8LRyfiZVHMvftX/Lj5K6NLEmdJAtEgky6Oo2e4D89/c4TPfio2uhzRRvGBXixMTSLKz8qix37Dxs9XGl2SOAsSiAbxs5mZOSKBgbF+vLItj3/tLjS6JNFGYb5W5qcmcm6IhZemPsLn7/7N6JLEGZJANFDzZN9hSYHsP1ZFQ6OsfXZ3/jYzs0YkMCjOnxWLZ/D+ssUYtRpMnDkZ7jSYxaT4w0UxJ5b3ldc04Gs1yVI/N+ZlMTH5kjhe2XaUj//6J0qLjnHf1PmYLfLr5urkCtEFmJTCajZR29DI9DWZPPeNdMpxd2aT4uELo7mlTxhrP3qHFyaOp6aqyuiyRCskEF2IzWxiVNcgvs4qZ96GbKrrG40uSbSBUoo7+kYwfmAUO9avYsEjd3C8rMTossRpSCC6mLE9Q3lsUDQ78yqYsTaL47XSQMDdXd0jhIkXx3Jo5zbmjLuRwrxO1y7UbUgguqDUc4KZeHEsB4uqeXnrUaPLEU5wcWIgM0fEU3j0CLPuvY6cQz8ZXZL4BRKILmpoQtMv0Lj+kUaXIpykb5Qf8y8Jp6Ekj9njbmD/99uNLkmcRALRhfWN8iPM19q0X8u2o2RKB2631zXUm0WjkwhoqGD+Q7fx7ZdrjC5JtCCB6AaKqurZmH2caasz2F8oI5XuLtrfxsLRSST4wnO/v4eND11odEnCTgLRDUT4WVmYmoifzczTX2Ty/dEKo0sSbRTsbWHeyAT6Rvnyp81H+ffrL8kEbhcggegmov1tLEhNIsrPxpz12WzOLje6JNFGvlYzTw1LYFhSIO+9tIg3l8yisVGmWhlJAtGNhPpYmJ+aSJ8IHwK9zEaXI5zAam5aqXTNuSF8/s7rvHLDebItgYEkEN2Mv83M7MsS6BXhC8BPck/R7ZmUYlz/pu1rv8os57mxfak8Lu8AjCCB6Iaat7/ckXucCWkZ/H1ngdx/cnNKKW7oHcbjQ2L4Mb+SebItgSEkEN3YBVF+jD4niH/uKuTVbXk0Sii6vZFdmrYlyP1pF7Puk20JOpoEohszmxSPXBjNDb1C+exACc9LUwiPMNC+LUFVftO2BId27zS6pE5DAtHNNW+L+ZsLIvgys5zv82RKjidouS3BvPG3sHPjeqNL6hQkED3Ejb3DePGKZFJi/QHknqIHaN6WINpWz7OP38vXn31kdEkeTwLRgySHeAOwu6CSyaszKamS/Z/dXZivlWdGJdIr1MKy6b/j07+/ZnRJHk0C0QPV1mvSi6uZsjqDvOO1Rpcj2sjfvv/O0IQA/v78HN5eOk8mcLcTCUQP1C/GjzkjEymraWDq6kxpCuEBbGYTE4bGclX3YD5561VemfEH6uvkj52zSSB6qJ7hPsxPTaRRa6atziC3XH553J3ZpBg/MIo7+4bz1acf8uzj91JVcdzosjyKBKIHSw72ZuHoJEZ2CSLSz2p0OcIJlFLc3CecxwZFs2vzBp554FZKi44ZXZbHkED0cNH+Nu4bEIXZpCioqGNLjiwJ8wSp5wQz7dJ4svf9wKx7rycvO8PokjyCBGIn8o8fCljwZQ6rD8pGR57gwjh/5o5MpKKshFn3Xs/hPT8YXZLbk0DsRB5IiaZvlB9/2nKUj/YUGl2OcIKe4T4svDgEW1Uxc+/+FT9s/tLoktyaBGIn4m0x8dSweC5ODOCN7wp487t8mcDtARKCvFg0OokoPyvPPvJrvvl/HxtdktuSQOxkrGbFkxfFcnm3YHbkVlAja589QpivlfmpifQM9+GlaY+xetxAo0tySxKInZDZpHgoJYr5qYl4W0zU1DdS1yATfd1d8wTuixICeP3bfJnAfRYkEDsppRS+VjNaa5Z8c4R5G3KorpdfHndnM5uYKBO4z5oEYienlGJwvD878yqY8UUm5TUNRpck2ujkCdzPj+0rE7gdJIEoSO0azORL4jhYXMO0NRkUVtYZXZJooxMTuAdHszOvgnnjb6G0sMDoslyeBKIAYEh8ADOHx5NfUc+CL3Nk9NlDpHYNZvql8eTs/5GZ914nHbhbIYEoTugb7ce8kQk8kBJ1Yt8W4f5S4vyZN7JlB+7vjS7JZUkgip/pHuZD9zAfAD7YXcjugkqDKxLOcG54iw7c91wjHbhPQQJR/KKqukbWHCpl5tosth2RG/KeID7Qi0Wjk4nxt/Hs4/fw5X8/MLoklyOBKH6Rj9XE/NREEgJtzN+Qzfr0UqNLEk4Q6mPhmVGJ9Amz8ecZv+c/b/xZ7he34FAgKqWuUErtU0odUEpN+YWvJyql1iqlvlVK7VRKXeX8UkVHC/a2MG9UIr0ifFm6MZdP9hcbXZJwAj+bmRnD47kkMYB3/jifN5fMkgncdq0GolLKDCwDrgR6A7crpXqfdNhTwPta6/7AbcDLzi5UGMPXambmiHgGxfvja5U3FJ7Cajbx5NBYrjk3hM/feZ2Xpj5CXa10Vrc4cMwg4IDW+hCAUupdYCywu8UxGgi0fxwEHHFmkcJYNrOJqZfEnRh5PlRcTXKwFyYZiXZrJqUY1z+SMB8Lb6z6L2XFRTzx3Gv4BgS2/o89lCN/8uOArBaPs+2fa2kWcKdSKhv4FHjsl55IKTVeKbVNKbWtrFp2hHMnzWF4pLyWiWkZvLAxl/pGuffk7pRSXN8rjD9cFMO+7d8w57c3UVxw1OiyDOOs90C3A29oreOBq4C3lFL/89xa6+Va6xStdUqgtyMXp8LVxPhbue28MNZnlLHgy2xqZP2zRxiRHMSM4QnkH97HrOsvIufwAaNLMoQjgZgDJLR4HG//XEvjgPcBtNYbAW8g3BkFCtfSvCTsoQuj2H6kgplrszheK+ufPUG/GD/mj0qktkEz5/ZRZE662OiSOpwjgbgV6K6U6qKUstE0aLLypGMygVEASqleNAWiLJz0YFd0C2HixbH8VFTFKtmSwGOcE+rN4jFJ+NvMzFibxfb1aUaX1KFaDUStdT3wKPA5sIem0eRdSqk5Sqlr7Yc9CdyvlPoeeAe4R8vkJo93cWIgS8YkM7ZnKIDMZ/MQ0f42Fo1OIinIi6VPjGPNB28bXVKHUUb9EHcL89HPX55syGsL58s7Xsvir4/w2OBokoO9jS5HOEF1fSOLv85h+5EKbj0vjGtXfOsWa9x/PSBhu9Y65Wz+rUwsE05R06Aprqpn2upM9sj6Z4/gbTEx7dJ4RnUJ4r0fC3lt7iQa6j17dogEonCKxCAvFo5OIsjbfu9J1j97BItJ8djgaG7pE8a6j9/l+Sfvp7rKc//gSSAKp4n0s7IgNYn4QBvPbMjmu9wKo0sSTqCU4o6+ETyYEsX3X61m/gO3UVZcZHRZ7UICUThVsLeFeSMTuaJbMD3C5V6iJ7myewiTL4kjc893zLkuhfycTKNLcjoJROF0fjYz41Oi8bWaqa5vJO1giYxAe4gh8QHMuSyBspoGZt17Pel7fzS6JKeSQBTtKu1ACcu2HOW17Xk0Sih6hF4RvixMTcJaWcSc397ED5u/NLokp5FAFO3qmnNDuK5nKJ/8VMJSWf/sMRKCvFg0Ookoaz3PPvJrvvr0I6NLcgoJRNGulFLc0y+Cuy6IYENGGfM3yPpnTxHma2VBaiI9w315+anfeUSzWQlE0e6UUtzUO4xHLozmcEkNJdLpyGP42czMGhHPxfZms28tme3WzWal5YzoMGO6BXNpUiA+VhNaa47XNhLgZTa6LNFGVrOJCUNjCfPJZ+U7f6X4WB4PzVmKzcv9ZhnIFaLoUD72rtv/+OEYT36eTm55rcEVCWcwKcW4AVHc2y+Czav+y7PX9qWi3P324ZFAFIYYFOdPZV0jU1ZncKi42uhyhJNc1yuMJy+KZV9hFbPvu4HCvFyjSzojEojCEN3DmvYJtpgU09dksivfc5eDdTbDkgOZOTyBwowDzLpxKFkH9hpdksMkEIVh4gO9WJiaRIi3hXkbsqXRrAfpG+3HgtREGjXMueNy9mzfaHRJDpFAFIaK8LOycHQiE4fG4m+TARZP0iXEm8WjkwjxsbDggVvZtOq/RpfUKglEYbhALwsDYv0B2JBRxr/3embjgM4o0s/KwtQkuod686cpD/PZP/5qdEmnJYEoXMrWnOO8/m0+b31f4PaTfEWTAC8zsy9LYHCcH28tmcXbS+e57FxFCUThUn4/JIbLuwXzr92FvLz1KA2y1M8jeFlMTLo4jqu6B/PJW6/y8tOPU1/nelOuZGK2cClmk+KhlCgCbGb+tbuQ8tpGJg6NxWxy/db14vTMJsX4gVGE+Vh567OPOb4jjd+9vxXfgECjSztBrhCFy1FKcdcFEYzrH0msv1XC0IMopbipTxi/HxLDrvxK5vz2JooLjhpd1gkSiMJlXdszlN/0iwQgo6SGUlkD7TEu6xLE08MTyD+8j5l3X0fOoZ+MLgmQQBRuoK5BM29DNlNWZ5JfUWd0OcJJ+sf4sWBUIvUlR5l13/Xs/XaL0SVJIArXZzUr/nBRDKXV9UxZlUFmaY3RJQkn6RraNFcxSFex4P6b2LLmU0PrkUAUbqF3hC/zRyXSqDXTVmew71iV0SUJJ4nyt7FodBJdQ715cdKDfP7uG4bVIoEo3EZyiDcLRyfhZzPzkUze9iiBXmbmXpbAoFg/Vix+mn+88IwhcxWVUZNfu4X56OcvTzbktYV7K6mqx9tqwttioqFRyyi0B2lo1Ly2PY/PDpQwPCmQce9/i8VqO6Pn+PWAhO1a65SzeX25QhRuJ9jHgrfFRGVdA5NXZfDJ/mKjSxJOYjYpHkiJ4q6+EazPKGPRY3dTWV7WYa8vgSjcllkpQnwsLN+exzs/yFI/T9E8V/HxwTHs3foVc357E0X5HdNXUQJRuC0vi4kpl8QxqksQ7/5YyKvb82SpnwcZ2fX/5irOvmEo2Yf2t/trSiAKt2Y2KR4bHM31PUP57KcS3vq+wOiShBOdmKvYqJl9+2j27tjcrq8ngSjcnlKKe/pHMn5gFFd2Dza6HOFkXUO9WTQ6iWBvCwsevoPNqz9pt9eSQBQe4+oeIUT522jUmvd+PEZZjSz18xRR/jYWjk7inEATf5z0YLv1VZRAFB4ns6SGf+4qZOrqTApkqZ/HCPQyM+eyBAbH+9v7Ks51+lxFCUThcZJDvJl9WQJFVfVMXp1Bliz18xjNfRWv7h7MJ28t56Vpj1JX67zvrwSi8Eh9IpuW+jU0aqauzmR/oSz18xRmk+L+gVHcfUEEm9L+w8JH7nTaHtASiMJjdQnxZmFqEsHeZmrqXbNlvTg7Silu6B3GExfF8NO3m5r2gD56pM3PK4EoPFpMgI0Xr+zC+VF+ABwpd7229eLsDU8OYubwBIoyDzDznrFk/rSnTc8ngSg8XvNa5205x3nkk0Oy1M/D9I1umquoyo8x584r2vRcEoii0zg/ypeUWH+Wb8/jH7LUz6MkhzTNVewV7tOm55FAFJ1Gy6V+7/1YyKvbZKmfJ4nwszJjREKbnkN23ROdSvNSvyBvMx/uKeLCOH8GxvobXZZwERKIotNRSnF3v0gGxfnTK8IXAK01Sklfxc5O3jKLTqs5DPcXVvHUF1myq5+QQBSirKaB/YVVsqufkEAUIiXWnzmXJVBaXc9k2dWvU3MoEJVSVyil9imlDiilppzimFuUUruVUruUUv9wbplCtK9eEb7MT01EA1NXZ5BdJqHYGbU6qKKUMgPLgNFANrBVKbVSa727xTHdganAxVrrYqVUZHsVLER7SQ72ZlFqIiv3FRPjf2YbGwnP4MgV4iDggNb6kNa6FngXGHvSMfcDy7TWxQBa63znlilEx4jyt3H/wCjMJkVRVT1fZ3bcBkfCeI4EYhyQ1eJxtv1zLfUAeiilvlZKbVJKtW39jBAu4J+7jrH46yP8W/aA7jScNQ/RAnQHRgDxwAal1Pla65KWBymlxgPjASJ8ZQqkcG339Y+kpLqB17/Np6S6nt9cECFzFT2cI1eIOUDL9TDx9s+1lA2s1FrXaa0PA/tpCsif0Vov11qnaK1TAr0lEIVrs5pNTBgay+XdgvlwTxEvbTkqS/08nCOBuBXorpTqopSyAbcBK0865mOarg5RSoXT9Bb6kBPrFMIQZpPioZQobj0vjL3HqqiSvooerdXLNK11vVLqUeBzwAy8rrXepZSaA2zTWq+0f22MUmo30ABM1FoXtmfhQnQUpRS/Pj+CG3qF4W0xUdfQSG2Dxs9mNro04WQOvW/VWn8KfHrS52a0+FgDT9j/E8IjeVua3lD9afNRMkprmDkigVAfufXjSWSlihBnaGSXII4er2XKqgzpwO1hJBCFOEP9YvyYNzKRqvpGpqzK4GBRtdElCSeRQBTiLHQP82FBaiJeFsWCL7Opa5DRZ08gN0CEOEvxgV4sTE3iWGU9VrPMT/QEEohCtEGYr5UwXysAH+wuxNdq4sruIQZXJc6WBKIQTtCoNXsKKtl6pILS6gZuPS9MVrW4IbmHKIQTmJRi6qXxjOwSyDs/HuPV7bKBlTuSK0QhnMRsUvxucAxBXhY+2lvE8ZoGnhwaK1eKbkQCUQgnUkpxT/9Igr3N+FjNEoZuRgJRiHZwXa+wEx//kFdBQpAXwdLQxOXJPUQh2lFVXSOLvz7ClFUZ5B2XVS2uTgJRiHbkYzUx7dI4ymsbmLwqg8PFsqrFlUkgCtHOekX4siA1CZNJMW1NJj/mVxpdkjgFCUQhOkBikBeLUpMI9bGwMavc6HLEKchdXiE6SISflUWjk/CxtxGrrGvA1yo9FV2JXCEK0YH8bWbMJkVxVT2Pf3aY9388RlM7UeEKJBCFMECAl5neEb68/cMxXtuRT6OEokuQt8xCGMBiUjw+JIYgLzP/3ldMWXU9jw+JwWqWaxQjSSAKYRCTUtw3IIpgHwsrvisgws/K3f0ijS6rU5NAFMJgN/QKI8bfxvlRvkaX0unJ9bkQLuCihAD8bWZqGxp5cVOurGoxiGGBKDeRhfhfueV1bMkpl1UtBjEsEDNLavjPviKZciBEC0nBXrKqxUCGBaK3xcRfduTz6rY86qWRphAntFzVMmttFltzjhtdUqdhWCBG+1u5oVconx0okW+4ECeJ8LOyIDWJC6J9ifa3Gl1Op2HcKLNS3N0vksHxAfQM9wGgvlFjMUlDTSEAAr3MPD08AQCtNdtzKxgY4ydNZ9uR4aPMzWGYXlzNQ/89xC65ZyLE/9icc5y567Nlr5Z2ZnggNvOymLCYFDPWZvLFoVKjyxHCpQyO8+f6nqF89lMJS745Ql1Do9EleSSXCcSYABvPjkmid4QvL27O5a3vC2RqjhB2zXu13Nc/km+yypm9LpvKugajy/I4LhOI0NQJZOaIBC4/J5h/7S4k7WCJ0SUJ4VLG9gzlD0Ni2FdYxYFCmafobC63dM9iUjx0YRTnR/lyUUIA0HRDWW4kC9FkRJcgLoj2I8Sn6de3pr4RL4tLXdu4LZf8v6iU4tKkQCwmRWl1PdPWZHKwSP4aCtGsOQw3Z5fzwH8PcUB+P5zCJQOxpbKaBvIr6pi6OkNarwtxkrhAG1YTTF+TyXdHK4wux+25fCAmBHmxZEwyScFeLPwqhw92F8pyPyHs4gO9WJiaRJSflbnrs/gyo8zoktyaywciNL09mDcykUsTA3jz+wI+2ltkdElCuIwwXyvzUxPpEebDc98c4afCKqNLclsuN6hyKl4WE08OjaV7WDEjkgONLkcIl+JvMzNrRAJfZpbRLdTb6HLclltcITZTSjG2ZyhB3hbqGjR/3JRLdlmN0WUJ4RK8LCZSuwajlCKztIbXZFXLGXOrQGwpv6KObUeOMyktg+9y5WayEC19m1vBf/cXs+CrHGrqZVWLo9w2EOMCbSy5PJlwXyuz12fx2U/FRpckhMsY2zOUB1Ki2JZznBlrsyivkVUtjnDbQASI9LOycHQi/WP8eGVbHu/vOmZ0SUK4jKu6hzDpklgOFFUzdXUGxVX1Rpfk8txmUOVUfK1mpl8az993FjAo1t/ocoRwKUMTAgkYYea/+4vxt7n19U+HcPtABDCb1M+2b/xgdyGXJAYQ5W8zsCohXMP5UX6cH+UHQHlNA7nHa+kR5mNwVa7J4/5kHKus44PdhUxMy2BPgfRWFKKlv+zIY/qaTLbkyKqvX+JxgRjua2XxmCR8rSae+iKLdenSW1GIZvf1jyQpyIsFX+awWrpJ/Q+PC0RoWs60eEwyPcO9WboxVwZbhLAL8rYwd2QiF0T58actR/nnrmOyFLYFhwJRKXWFUmqfUuqAUmrKaY67USmllVIpzivx7AR6mZk1IpHR5wSREOhldDlCuAwfq4npw+IZkRxI2sFSKutknmKzVgdVlFJmYBkwGsgGtiqlVmqtd590XADwOLC5PQo9G1az4tFBMSceb8wq59xwH0J9PGIsSYizZjUrHh8SQ2l1A342Mw2NmkatsZo98k2jwxw5+0HAAa31Ia11LfAuMPYXjpsLLAJcsjHb8doG/rQ5lwlp6RwqdskShehQJqVO9FX887ajzF6XTUVt557A7UggxgFZLR5n2z93glJqAJCgtf7EibU5lb/NzLxRiQBMXZ3B5mwZZROi2XmRvuwuqGT6mkyKOvEE7jZfHyulTMDzwJMOHDteKbVNKbWtrLrj/6d3DfFmyZhkEgKbRtk+3FPY4TUI4YpGJAfx1PB4co/XMnlVBjlltUaXZAhHAjEHSGjxON7+uWYBwHnAOqVUOjAEWPlLAyta6+Va6xStdUqgtzH38UJ9LDwzKpGhiQHUNsjomhDNBsT4M29kItX1jcxal0VdJ/z9cCSVtgLdlVJdaArC24BfN39Ra10KhDc/VkqtAyZorbc5t1Tn8bKYmDA0luZtq34qrCLK30agl9nQuoQwWvcwHxaNTqKgog6rufNt7NbqFaLWuh54FPgc2AO8r7XepZSao5S6tr0LbC8mpVBKUdvQyIIvc5iYli69FYUAYgNsXBDdtNTvs5+KWXe48yxucOgeotb6U611D631OVrrZ+yfm6G1XvkLx45w5avDk9nMJiZdEkdVXWNTb0XZqEcIABq1ZmNWOUs35fJxJ7nf3rknHdn1DPf5v96K66S3ohDQ9C7q6eHxXJwYwN++K+D1HXk0eviqFglEu5a9FXfmVcpyJiEAq7npfvvVPUL4975iXtyU69G/G7Jko4Xm3ooNWqOU4lhlHb5WE75WGWwRnZdJKe4fEEmotwWzqWlvI08lgXgSs0lhRtGoNc9syKa+UfPUsHjprSg6NaUUN/UJO/F437EqovysBHvYMlh5y3wKJqW4p18khZX10ltRiBZqGxpZ8FUOk1dnkFvuWRO4JRBP44Jov5/3VuxE0w+EOBWb2cTUS+KoqG1g8qoMDhR5Tm8ACcRWxAd68eyYZHqG+/Dx3iLqZZ9bITg33IeFo5Pwsiimr8n0mK2AJRAdEOBlZtaIBGZdloDFpKiub5S9bkWnFx/oxcLUJKL9razP8Ix3T551R7QdWc2KYHPT/64XNuVSUFHHtEvjCPO1GlyZEMYJ87Uyf1QiNnsfxcq6BreelSFXiGdhZJdAsstqmJiWwUEPun8ixNnws5mxmhXHaxt48vMMt57ALYF4FgbFBbAoNQmTauqtuDFLeisK4WMx0T/al3/vK+aFjblu2S1HAvEsJYd48+yYZJKCvXhl21Eq6zp3p2EhzCbF/QOjuKtvBOszypi3Icvtfi/kHmIbhPhYmDcykbyKOnytZhq1pqGRTtk2SQj4vwncIT5mXtpylL/uyOexwTHsXyo2AAAW9UlEQVSt/0MXIYHYRl4WE4lBTbv6vfPDMX7Mr2TKJXEEGdQAVwhXMKprMGE+VpKD3WvHS3nL7ESJQV4cKKpmYloGmaXSW1F0bv1i/Aj2sVDfqFm68Qg/FVYZXVKrJBCd6NKkQJ4ZlUhtQ1NvxR1HjhtdkhCGK6muZ3dBFU99kcmOXNf+nZBAdLIeYT48OyaZaH8rC77KobgT72AmBEC4r5VFo5OI8bcxb302a114CawEYjuI8LOyIDWJ6ZfGn9j31pN7yAnRmubN3fpE+vLCplw+ddEmzBKI7cTHaqJfTNO+FF9lljFrXTbHO/km4KJz87OZmTE8ntSuQZwb5mN0Ob9IArED1NZrfsyvYFKa57VLEuJMWM0mHhscwzmh3gBsSC+jrsF1+gJIIHaAkV2DmHNZImU1DUxMS+eHPM/oDCJEWxwsqua5jUeYvS6bChd59ySB2EH6RPry7JgkgrwtzFybJVueik7vnFBvfj8kht0FlUxfk0lhZZ3RJUkgdqSYABuLRifxQEo08YHuNWFViPZwWZcgnhoeT+7xWqaszjD8QkECsYP528xc3i0YgENF1Sz5OoeqOte5hyJERxsQ488zo5Koa9CklxgbiLK+zEDpJTV8nVVOVlktTw2LJ8JPeiuKzqlbqDcv/6rriV6KJVX1hmxgJVeIBhrZNYgZwxMoqKhjQlo6+465/tImIdpLcxjuKajk/v8cZNXBkg6vQQLRYP1j/Fg4Oglvi4npazLZ7wbrPYVoT8nB3vSJ8OWlLUd598djHbqoQQLRBSQGefHsmCSuOTeEriHeRpcjhKF8rCaeGh7PyC6BvPPDMf68LY+GDtrcTQLRRQR6Wbi7XyQWk6Kkup7XtufJRlai07KYFL8bHMNNvcP4/EAJ6zPKOuZ1O+RVxBnZebSST/YXs+9YFdOGxRNqwM1lIYymlOKuCyI4L9KXftG+HfKacoXogoYlBzL10jgyS2uYkJbOIdnISnRi/WP8UEqRW17L9DWZ5Fe03wRuwwLRXXfl6iiD4wNYODoJBUxZnSHL/USnV1Jdz+HiaiavyiC9uH0uEgwLxMyyBj7Z75otgFxF1xBvloxJZlCcP0nBMtgiOrdeEb4sSG26SJi6JpOd7XCRYFggevn4sHx7Hsu3d9wIkjsK8bEw4eI4Ar3M1DU08sHuQpfqDiJER0oK9mLR6CTCfCzMXpft9HdOhgViVEIyV97xWz7ZX8wzX2a73XaFRtiRW8Gb3xcwY20WpdXSiVt0ThF+VhaOTmLMOUF0d3JfRUMHVe56cib3TZvPt3nVTFudSUE73iz1BIPjA5gwNFY2shKdnr/NzAMp0XhbTFTWNfCffUVOGZcwfJQ59aa7mPjiGxyttzFxVQYHZET1tC5NCmTeyERqGhqZvCqD74/KYIvo3Nanl/GXHfm8sDGXuoa2haLhgQhwwdARzHr9Q8zB0Uxbn8em7HKjS3Jp54b7sGRMMl2CvU7s2SJEZ3VFt2Du7BvO+owy5q7PatNzuUQgAiR068ncN1eS0K0nC7/M4aM9hbIx02lE+Fl5ZlQiiUFeaK3ZkF5GvQxOiU5IKcXNfcL53eBodhW0rReAywQiQFBYBE8tf49Bo3/FG98V8PLWo/JLfhpKKQD2HKuyt2LPko2sRKc1qmswr17TtU3P4VKBCGDz9uGxBcsYe9+jpB0sZWZ6pPySt6J3hC+PD25qxT4xLYMjspGV6KTCfdvWU9TlAhHAZDJx66OTeWDWc+zZvolJO+DocfklP53mjazK7RtZtcekVSE8nUsGYrPh197C1JffpqQwn4lpGewpqDS6JJfWJ9KXJZcnEeZjlW0JhDgLLh2IAL1TLmLOipX4RCbw9IY8NnRQGyB3Fe1vY+kVyQyODwBgd0GlrAQSwkEuH4gAMUldmbNiJeec14/nvjnCex3cRdfdmE1Ngy3ZZTVMX5PJfFkJJIRD3CIQAQKCQ5j68ttc+qub+McPx3hhU66s6W1FfKAX4wdGsSO3gimrMsmT+7BCnJZDgaiUukIptU8pdUApNeUXvv6EUmq3UmqnUmqNUirJ+aWC1ebFg7Of5+aHJ7AuvYwZa7Moq5E1vadzZfcQZo5I4FhlndyHFaIVrQaiUsoMLAOuBHoDtyulep902LdAita6L/AvYLGzC21RD9f/9nEeXfASP5U2MjHN+M2tXV2/aD8Wj0nC12piV75sYiXEqThyhTgIOKC1PqS1rgXeBca2PEBrvVZr3XzpsQmId26Z/2vo5WOZ/up7VFoDmbihiK2XzWnvl3Rr8YFePH9FMjf2DgUgt7xWmvQKcRJHAjEOaLlAMNv+uVMZB3z2S19QSo1XSm1TSm0rLy5yvMpT6HHBQOa8uZKQ8EgWPHwHqw91/D6u7sTXakYpRUlVPRPS0ln0VQ7VspGVECc4dVBFKXUnkAI8+0tf11ov11qnaK1TAkJCnfKakXGJzHrjI3qnDOFPm4/y5nf5cuXTiiBvM7f2CWdLznGmrs7gWKW0XRMCHAvEHCChxeN4++d+RimVCkwHrtVad+hNPb+AICa+uIJRN97JB3uKePbrI7KF52kopbi2ZyjTh8WTW17HhLQMfiqUe4tCOBKIW4HuSqkuSikbcBuwsuUBSqn+wKs0hWG+88tsncVq5b5p87njiafZmH2cKbt9KK6SEejTSYn1Z9HoJKwm+PQn2d9GiFYDUWtdDzwKfA7sAd7XWu9SSs1RSl1rP+xZwB/4p1LqO6XUylM8XbtSSnH1neP5w3OvkXNwPxPT0kkvkYazp5MU7MWzY5J5MCUagLKaBpn0LjotZdQPf9feffUzb3/abs9/eM8PLPn9vVQVFzDp4lgGxvq322t5iur6RiZ8nk6XEG8eHRSNl8Vt5u0LccLYd/Zu11qnnM2/9dif+C69zmfOm/8h6pxezP3yiGx56gAvs2JElyA2ZJTx1BeZcstBdDoeG4gAYVExzPzrB/S/ZKRseeoApRQ39Q5jyiVxZJTUMCEtnUPttCG4EK7IowMRwNvXjyee+8uJLU+l0UHrLkoIYGFqEhr4y/Y8uacoOo1OsUORyWzmridnEp3YhRWLZzD5ewszetUS4de27rqerGuoN0vGJKO1RilFXUMjFpM6sW2BEJ7I468QWxp982+Y9McVFORmM+Gbcpl714pQHwthvlYaGjWLvj7CHzcflQ5DwqN1qkAE6HvRcGb/7SMsVhvT1mSyMUu2PG2NScE5IV58cbiUGWuzKK2WwRbhmTpdIALEn3Muc99cSWLv/iz8KocPd8uWp6ejlOL28yN4cmgsB4qqmZiWQWapdBgSnqdTBiLYtzx99V2GjLmGFd8X8NIW2fK0NcOSAnlmVCI1DY0s+DJHRuyFx+kUgyqnYvP24dH5LxGT1JWPXnuR/Io6Jl8Sh7/NbHRpLqtHmA9LxiRTWtOA2aROXFnLYIvwBJ32CrGZyWTi5ocm8OCcpewuqGRSWga5sq/xaUX4WekW6g3A33ce48/b8uTqWniETh+IzYb96iamvvpPSk2+TEzLYLe02m9V89Xh5wdKmL0ui/Iamd8p3JsEYgu9Bg5h9oqV+Mck8fQXWaw7XGp0SS5NKcVdF0Tw+JAYdhdUMWlVOjllcnUt3JcE4kliErsw+42P6TFgCEs35bLC/woZgW7FyC5BzBuZQEVtI9PWZEgXbuG2JBB/gX9QCFOW/Z3h197Ch6+9wPMbc6mVCcmn1SvCl2fHJPHQhdF4S5cc4abkJ/cULFYb42cu4bbfTbV3f8miRCYkn1aUv40h8QEAbEgv4zVppiHcjATiaSiluPaeh/n9s69yqLyRCZtrZEKyg9JLqvnv/mLmrs+molYGW4R7kEB0wKBRVzHjL/+irraGyasy+C63wuiSXN5v+kXyyIXR7MyrYNIqmcok3IMEooPO6dOPOStWEpbUg9nrs/hM9iBp1Zhuwcy6LIHS6nompmVQIg1nhYuTQDwD4TFxzHz9Q/pePJJXtuXx1x1yj6w1faP8WDwmmRt7hxLs06kXRgk3IIF4hnz9A3jy+b9y+W33snJfMQu/yqGqTkagTyc2wMb1vcIAOFBUzZvf5csfEuGSJBDPgtli4e5Jc7h70ly2HZHN3s/ElpxyPthTxALpXC5ckARiG1x+2z08+eIKco/XMTEtg4NFsv9Ia359fgTjB0axPbeCKasyya+QPyTCdUggtlH/S0Yy6+3/hykoiqnr89icLQ1nW3N1jxBmDk/gWGUdEz5PlxFo4TIkEJ0gsXsv5ry5kvhu57LgqyO8H32TLPdrRb8YPxaPSWJIQgCRsreNcBESiE4SHB7J08vfZ1Dq1by9dB4vb5WGs62JD/Ti4QujMZsUxVX1fLC7kEb5QyIMZFggHsvN4WjmYaNevl3YvH14bMEyrr33EdIOljJ7XRbHZZWGQ9all/Lm9wUs/ipHmkMIwxgWiBXlpcy4+1r2bN9oVAntwmQycdtjU3hw9vPsKqxjsqzScMh1PUMZ1z+SzTkyai+MY1ggxiZ3IzAknPkP3cH336wzqox2M+yam5n257cpxoeJX5WwRxrOnpZSimt7hjJ9WDy55U2DLeklMmovOpZhgWi12Zi94mNGXHcr3c8fYFQZ7arXwIuY8+Z/8AsM4qkvsliXLg1nW5MS68/iMUkkB3sT7iuDLaJjGTqo4hcQxLhpC/ANCKS2uor3XlpEdZVnXUnFJHZhzop/073/YJZuzOWdHwpkBLoViUFezLosAX+bmbqGRtIOlsj/M9EhXGaUede2jaz82zLm/vYmiguOGl2OU/kHhTD15bcZds3NvPtjoTScPQPr0stYtuUoS745Qo0Mtoh25jKB2P+SkTy59HWOpB/k6buuIX3vj0aX5FQWq40HZj3HbY9NYUNGGU9/kUWpNJxtVWrXIH5zQQRfZ5YzfU0mRdIxR7QjlwlEgAHDUpn1t49QJhOz77vB4wZblFJce+8jPL74FQ4d10xMy5CGs61QSnFj7zCmXBpHVlkNE9LSOVQsgy2ifbhUIAIk9ejNnBUr6TlgMJHxSUaX0y4Gp17NU8vfp9o7mEkbjknDWQcMiQ9gQWoS/lYzVpMyuhzhoVwuEAFCIqKY/NJbxCR2QWvNun+/R32dZ81L63Zef+a++R/Co+OY/eURaTjrgK4h3rxwZTIJQV5ordmac1wGW4RTuWQgtrRn+yaWz57Aokfv4nhZidHlOFV4TByz/vYRfS8aLg1nHWRSTVeH245UMG9DNktlgEo4kcsHYu+Ui3hwzlL2fruFmXeP9bjlfj5+/k0NZ2+/j5X7ilkgDWcdkhLrxx19w1nfvCOiDLYIJ3D5QAQY9qubmP7KO5SXFnvkcj+zxcLdE2dzz+S5bMutlKVrDlBKcUufcCZfEsvh4mompKWTLoMtoo3cIhABeg4YzNwVKwmJiPa4+4nNxtx6D5NefIOjdVYmpmVwQBrOtmpoQiALUpsG345VylWiaBtl1E3prr376mfe/vSM/11jQwMmsxmAvTs206PfhZhMbpPrDsk6sJcl91xJWU0DTwyNPbH5uzi12oZGbOamn4ODRdV0DfFCKRmN7ozGvrN3u9Y65Wz+rdslSXMYpu/bxdz7b+bFSQ963HK/hG49mf3xZhJ69WPhV0f4cE+hjKa2ojkMDxQ1vX3+0+aj1MlgizhDbheIzZJ69OaOPzzNtrX/zyOX+wWHR/LU8vcYnHo1K74r4IXq/tQ1SCi2pmuIFzf3CWPN4VJmrJXVQOLMuG0gKqW46s77eXLp6+RmHPLI5X42bx8eXbCM68Y9xrqP32XGoTBpONsKk1L8+vwInhway4GialkNJM6I2wZiswHDUpn1etNyv73fbjG6HKczmUzc8sgkHpyzlH3fbWVimjScdcSwpECeGZVIbUMjO44cN7oc4SbcblDlVCrLy/DxD0ApRX5OJhGxCR53U33vjs08P+F+VGUpUy+Np0+kr9ElubyymnoCbGaUUhRU1BHua/G4nwvxc+0+qKKUukIptU8pdUApNeUXvu6llHrP/vXNSqnksymmLXwDApt+6I9kMfW2y/nLvMnU13nWlVTPAYOZ88a/8Y/twtPrc/nisDScbU2gl+VEGD7+2WFe3npU7sWKU2o1EJVSZmAZcCXQG7hdKdX7pMPGAcVa627AUmCRswt1VFh0HJffdi9rP3qHhY/cxfFSz1ojHJ3YhdlvfEzP/oN4cVMuf99ZIDvVOSDM18KV3UNIO1jKrHVZlNXIvVjxvxy5QhwEHNBaH9Ja1wLvAmNPOmYssML+8b+AUcqg9yXN99wenvsi+7/fxoy7x5KbcciIUtqNf2Awk196ixHX3cY/dxVK81QHmJTirgsi+MOQGPYeq2JiWjrZZTLYIn7OkUCMA7JaPM62f+4Xj9Fa1wOlQJgzCjxbl1x9A9NeeYeKslLS3nvDyFLahcVq5f6nF3P749P5JruC3QVVRpfkFkZ0CWLeyASq6hpZudez3j2Itmt1UEUpdRNwhdb6t/bHdwGDtdaPtjjmR/sx2fbHB+3HHDvpucYD4+0PzwM8a57Mz4UDx1o9yn158vl58rmB55/fuVrrs1reZXHgmBwgocXjePvnfumYbKWUBQgCCk9+Iq31cmA5gFJq29mOBLkDOT/35cnnBp3j/M723zrylnkr0F0p1UUpZQNuA1aedMxK4G77xzcBX2hZayaEcDOtXiFqreuVUo8CnwNm4HWt9S6l1Bxgm9Z6JfBX4C2l1AGgiKbQFEIIt+LIW2a01p8Cn570uRktPq4Gbj7D115+hse7Gzk/9+XJ5wZyfqdk2EoVIYRwNW6/llkIIZyl3QPRHZb9tYUD5/eEUmq3UmqnUmqNUspt9lZt7dxaHHejUkorpdxq5NKR81NK3WL//u1SSv2jo2tsCwd+NhOVUmuVUt/afz6vMqLOs6GUel0plW+f8vdLX1dKqT/az32nUmqAQ0+stW63/2gahDkIdAVswPdA75OOeRh4xf7xbcB77VmTAed3GeBr//ghdzk/R87NflwAsAHYBKQYXbeTv3fdgW+BEPvjSKPrdvL5LQcesn/cG0g3uu4zOL9hwADgx1N8/SrgM0ABQ4DNjjxve18hutWyv7PQ6vlprddqrZtbem+iaR6nO3Dkewcwl6a16+62AYwj53c/sExrXQygtc7v4BrbwpHz00Cg/eMg4EgH1tcmWusNNM1oOZWxwJu6ySYgWCkV09rztncguuWyvzPgyPm1NI6mv1ruoNVzs78NSdBaf9KRhTmJI9+7HkAPpdTXSqlNSqkrOqy6tnPk/GYBdyqlsmmaRfJYx5TWIc70dxNwcNqNaDul1J1ACjDc6FqcQSllAp4H7jG4lPZkoelt8wiaruw3KKXO11qXGFqV89wOvKG1fk4pdRFNc4nP01p32k4h7X2FeCbL/jjdsj8X5cj5oZRKBaYD12qt3aXFSmvnFkDTevR1Sql0mu7TrHSjgRVHvnfZwEqtdZ3W+jCwn6aAdAeOnN844H0ArfVGwJumdc6ewKHfzf/Rzjc+LcAhoAv/d2O3z0nHPMLPB1XeN/qGrZPPrz9NN7e7G12vs8/tpOPX4V6DKo58764AVtg/DqfpLViY0bU78fw+A+6xf9yLpnuIyujaz+Ackzn1oMrV/HxQZYtDz9kBRV9F01/Wg8B0++fm0HS1BE1/lf4JHAC2AF2N/h/t5PNbDeQB39n/W2l0zc46t5OOdatAdPB7p2i6LbAb+AG4zeianXx+vYGv7WH5HTDG6JrP4NzeAXKBOpqu5McBDwIPtvjeLbOf+w+O/mzKShUhhLCTlSpCCGEngSiEEHYSiEIIYSeBKIQQdhKIQghhJ4EohBB2EohCCGEngSiEEHb/H6CobcTAnn6EAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Apply the decision function on the grid input matrix\n", "y_keras_grid = model.predict(X_grid)\n", "# Reshape the array to recover the squared shape\n", "y_keras_grid = y_keras_grid.reshape(rows.shape)\n", "# Plot the decision\n", "plot_decision_contours(y_keras_grid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Writing the exact same library with numpy\n", "\n", "Now that we have solved this exercice with **just** a dozen lines of `Keras`, let's dive into the inner workings of a neural network by writing one from scratch." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ingredients\n", "\n", "- `numpy`\n", "- [a loss function](#Loss-function)\n", "- [some layers](#Layers)\n", "- [a neural net](#Neural-network)\n", "- [an optimizer](#Optimizer)\n", "- [a batch data provider](#Batch-generator)\n", "- [a training routine](#Training)\n", "\n", "Hopefully by the end of this tutorial you will have an understanding of the building blocks needed for training (deep) neural networks. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Foreword\n", "\n", "We will purely rely on `numpy` for this tutorial. Make sure to import it here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Object Oriented Python\n", "\n", "Object-oriented Python, a.k.a _classes_, will be used in this tutorial. \n", "For those not familiar with Python classes, know that you will only be required to write some definitions and Python code **within the** class **methods** and **not** actually **write any class**. \n", "\n", "If you want to know more about Python classes, here is a step by step [tutorial](https://aboucaud.github.io/slides/2016/python-classes)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loss function\n", "\n", "A loss function measures how good our predictions are, compared to the expected values. It is the cost function that needs to be minimised. The loss function must be differentiable, as its gradient is needed to adjust the parameters of the network through backpropagation.\n", "\n", "Below is generic loss class. It implements \n", "- `loss()` : **the loss** computated from the expected label and the predicted one,\n", "- `grad()` : **the gradient of the loss**, needed for the backpropagation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercice #1 - mean square error loss\n", "\n", "***5 min*** - *Implement the `MeanSquareError` class*\n", "\n", "In this exercise, `predicted` and `actual` are vectors (1-d numpy arrays). \n", "You must implement both the loss and its derivative using these two vectors.\n", "\n", "For info, the mean square error loss function is defined as\n", "\n", "$${\\rm loss_{MSE}}(y_{true}, y_{pred}) = \\sum \\left(y_{pred} - y_{true}\\right) ^ 2$$\n", "and its gradient\n", "$$\\nabla {\\rm loss_{MSE}}(y_{true}, y_{pred}) = 2 \\cdot (y_{pred} - y_{true})$$" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "class MeanSquareError:\n", " def loss(self, y_predicted, y_true):\n", " return np.sum((y_predicted - y_true) ** 2)\n", " \n", " def grad(self, y_predicted, y_true):\n", " return 2 * (y_predicted - y_true)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Layers\n", "\n", "\n", "### Neuron\n", "\n", "A neuron $\\mathscr{N}$ has multiple input values (vector $\\mathbf{x}$ of size $m$) and a single output $z$. Each neuron is characterised by its weights (vector $\\mathbf{w}$ of size $m$) and a constant bias $b$ to perform the linear operation\n", "\n", "$$\\begin{aligned} \n", "\\mathscr{N}_{\\mathbf{w}, b}(\\mathbf{x}) &= \\sum_i w_i.x_i + b \\\\\n", " &= \\begin{bmatrix} w_{0} & \\cdots & w_{m}\\end{bmatrix} \\begin{bmatrix} x_0 \\\\ \\vdots \\\\ x_m \\end{bmatrix} + b\\\\\n", " &= \\mathbf{w}^T\\mathbf{x} + b \\\\\n", " &= z\n", "\\end{aligned}$$\n", "where $m$ is the input size, and output a single value $z$.\n", "\n", "\n", "### Linear layer\n", "\n", "A linear layer $\\mathscr{L}$ is a set of neurons, and can therefore be represented by a matrix of weights $\\mathbf{W}$ and a vector of constants $\\mathbf{b}$. \n", "For a layer of $n$ neurons, the matrix $\\mathbf{W}$ is therefore $(m,n)$ and the vector $\\mathbf{b}$ is of size $n$. \n", "\n", "The operation realized by the layer on the input vector $\\mathbf{x}$ of size $m$ is\n", "\n", "$$\\begin{aligned} \n", "\\mathscr{L}_{\\mathbf{W}, \\mathbf{b}}(\\mathbf{x}) \n", "% \\begin{bmatrix} y_0 \\\\ \\vdots \\\\ y_n \\end{bmatrix}\n", " &= \\begin{bmatrix} \\sum_i W_{i, 0}.x_i + b_0 \\\\ \\vdots \\\\ \\sum_i W_{i, n}.x_i + b_n \\end{bmatrix} \\\\\n", " &= \\begin{bmatrix} W_{0,0} & \\cdots & W_{m,0} \\\\ \\vdots & & \\vdots \\\\ W_{0,n} & \\cdots & W_{m,n} \\\\\\end{bmatrix} . \\begin{bmatrix} x_0 \\\\ \\vdots \\\\ x_m \\end{bmatrix} + \\begin{bmatrix} b_0 \\\\ \\vdots \\\\ b_n \\end{bmatrix} \\\\\n", " &= \\mathbf{W}^T\\mathbf{x} + \\mathbf{b} \\\\\n", " &= \\mathbf{z}\n", "\\end{aligned}$$\n", "\n", "which is a matrix multiplication and an addition, that produces an output vector $\\mathbf{z}$ of size $n$.\n", "\n", "### Activation layer\n", "\n", "After the layer forward pass, there might be an activation layer whose role is to break the linearity of the network. The so-called activation layer is thus a non-linear fonction $f$ acting on the output $\\mathbf{z}$ of the linear layer. \n", "\n", "$$\\begin{aligned}\n", "\\mathbf{a} &= f(\\mathbf{z}) \\\\\n", " &= f(\\mathbf{W}^T\\mathbf{x} + \\mathbf{b})\n", "\\end{aligned}$$\n", "\n", "The activation layer conserves the shape.\n", "\n", "### Backpropagation (computing of the layer gradients)\n", "\n", "For the backward pass, each layer receives a gradient vector for the preceding layer.\n", "\n", "The ***chain rule*** connects the the loss $\\mathscr{C}$ (for cost) to the weights and biases of layer $i$ and yields the following relations :\n", "\n", "$$\\begin{aligned}\n", "\\dfrac{\\partial \\mathscr{C}}{\\partial \\mathbf{W}^{i}} \n", " &= \\dfrac{\\partial \\mathscr{C}}{\\partial \\mathbf{a}^{i}} \\cdot \\dfrac{\\partial \\mathbf{a}^{i}}{\\partial \\mathbf{z}^{i}} \\cdot \\dfrac{\\partial \\mathbf{z}^{i}}{\\partial \\mathbf{W}^{i}} \\\\\n", " &= \\mathbf{\\nabla}\\mathscr{C}^{i} \\cdot f'(\\mathbf{z}^{i}) \\cdot \\mathbf{x}^{i} \\\\\n", " &= \\mathbf{\\nabla_W^i}\n", "\\end{aligned}$$\n", "\n", "and \n", "\n", "$$\\begin{aligned}\n", "\\dfrac{\\partial \\mathscr{C}}{\\partial \\mathbf{b}^{i}} \n", " &= \\dfrac{\\partial \\mathscr{C}}{\\partial \\mathbf{a}^{i}} \\cdot \\dfrac{\\partial \\mathbf{a}^{i}}{\\partial \\mathbf{z}^{i}} \\cdot \\dfrac{\\partial \\mathbf{z}^{i}}{\\partial \\mathbf{b}^{i}} \\\\\n", " &= \\mathbf{\\nabla}\\mathscr{C}^{i} \\cdot f'(\\mathbf{z}^{i}) \\\\\n", " &= \\mathbf{\\nabla_b^i}\n", "\\end{aligned}$$\n", "\n", "where $\\mathbf{\\nabla}\\mathscr{C}^{i}$ is the gradient vector of the loss propagated at layer $i$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial, we define sequential neural nets, made of one or more layers.\n", "The layer will pass its inputs forward\n", "and propagate gradients backward. \n", "\n", "For example, a neural net might look like `inputs -> Linear -> Tanh -> Linear -> output`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The base class for a layer has a dictionary to store parameters ($\\mathbf{W}$, $\\mathbf{b}$) and gradients ($\\mathbf{\\nabla_W}$, $\\mathbf{\\nabla_b}$) and implements a forward and a backward method." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercice #2 - linear layer\n", "\n", "***10 - 15 min*** - *Implement the `forward` and `backward` methods of the linear layer.*\n", "\n", "The mathematical recap above is here to help you. \n", "\n", "Be aware that neural networks are generally trained in batches (see [batch generator part](#Batch-generator)), essentially in order to \n", "- save some foward / backward computing steps\n", "- reduce the noise produced by extreme input vectors at the optimisation step.\n", "\n", "We therefore introduce the concept of ***batch_size***, which is the number of simultaneous trained inputs.\n", "For this reason, the input and output arrays of the layers are not actual vectors but **matrices** whose shape of one dimension is the ***batch_size***.\n", "\n", "Hints:\n", "- matrix products can be written either with `np.dot(m1, m2)` or `m1 @ m2` with recent Python versions (3.5+)\n", "- pay a specific attention to the shape of the input and output matrices for the matrix product\n", "- $\\mathbf{W}^T\\mathbf{x}$ is written `x @ W`" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "class LinearLayer:\n", " \"\"\"\n", " Inputs are of size (batch_size, input_size)\n", " Outputs are of size (batch_size, output_size)\n", " \"\"\"\n", " def __init__(self, input_size, output_size):\n", " self.params = {}\n", " self.grads = {}\n", " # Initialize the weights and bias with random values\n", " self.params[\"w\"] = np.random.randn(input_size, output_size)\n", " self.params[\"b\"] = np.random.randn(output_size)\n", "\n", " def forward(self, inputs):\n", " \"\"\"\n", " inputs shape is (batch_size, input_size)\n", " W shape is (input_size, output_size)\n", " b shape is (output_size)\n", " \"\"\"\n", " self.inputs = inputs\n", " W = self.params[\"w\"]\n", " b = self.params[\"b\"]\n", " # Compute here the feed forward pass\n", " return inputs @ W + b\n", " \n", " def backward(self, grad):\n", " \"\"\"\n", " grad shape is (batch_size, output_size)\n", " return shape is (batch_size, input_size)\n", " gradW shape is the same as W shape\n", " gradb shape is the same as b shape\n", " \"\"\"\n", " X = self.inputs\n", " W = self.params[\"w\"]\n", " # Compute here the gradient parameters for the layer\n", " self.grads[\"w\"] = X.T @ grad\n", " self.grads[\"b\"] = np.sum(grad, axis=0)\n", " # Compute here the feed backward pass\n", " return grad @ W.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Activation layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercice #3 - tanh\n", "\n", "***5 min*** - *Implement the hyperbolic tangent and sigmoid layers and their derivatives.*\n", "\n", "Look for the definitions in the lecture.\n" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "class Tanh:\n", " def __init__(self):\n", " self.params = {}\n", " self.grads = {}\n", "\n", " def forward(self, inputs):\n", " self.inputs = inputs\n", " return np.tanh(inputs)\n", " \n", " def backward(self, gradients):\n", " f_prime = 1 - np.tanh(self.inputs) ** 2\n", " return f_prime * gradients" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Neural network\n", "\n", "A neural net is a collection of layers. It takes care of sequentially calling the layers `forward` and a `backward` methods in the right order.\n", "\n", "In addition, it implements a getter method `params_and_grads` that will be used by the optimizer to update the values of the weights and bias of each layer." ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "class NeuralNet:\n", " def __init__(self):\n", " self.layers = []\n", " self.loss = None\n", " self.optimizer = None\n", " \n", " def add(self, layer):\n", " self.layers.append(layer)\n", " \n", " def compile(self, loss, optimizer):\n", " self.loss = loss\n", " self.optimizer = optimizer\n", " \n", " def predict(self, inputs):\n", " \"\"\"\n", " The forward pass takes the layers in order\n", " \"\"\"\n", " for layer in self.layers:\n", " inputs = layer.forward(inputs)\n", " return inputs\n", "\n", " def backprop(self, grad):\n", " \"\"\"sequential gradient computation and backward pass to the next layer\"\"\"\n", " for layer in reversed(self.layers):\n", " grad = layer.backward(grad)\n", " return grad" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimizer\n", "\n", "The role of the optimizer is to adjust the network parameters (weights and biases of the linear layers here) based on the gradients computed during backpropagation.\n", "\n", "The main attribute of an optimizer is the _learning rate_ (a.k.a. `lr`), which defines the size of the jump taken in the direction of the gradients. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercice #4 - Stochastic Gradient Descent\n", "\n", "***5 min*** - write the optimizer step\n", "\n", "Here we have a very basic implementation of a _Stochastic Gradient Descent_ (a.k.a. `SGD`). \n", "\n", "The step that needs to be written iterates over the neural network layers and updates the layers parameters in the direction _opposite_ to the gradient." ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "class SGD:\n", " def __init__(self, lr=0.01):\n", " self.lr = lr\n", "\n", " def step(self, layer):\n", " for name, param in layer.params.items():\n", " grad = layer.grads[name]\n", " param -= self.lr * grad\n", " " ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "# Add update method to the NeuralNet class to take a step\n", "def update(self):\n", " for layer in self.layers\n", " self.optimizer.step(layer) \n", "\n", "NeuralNet.update = update" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Batch generator\n", "\n", "It can be costly to compute the gradients and update the weights after every entry of the training dataset. In order to minimize such computational cost, the inputs of the network are traditionally fed in batches and the gradients are thus averages over those batches of data.\n", "\n", "A batch size of 32 is a default in multiple training sets. Some recent [study](https://arxiv.org/abs/1804.07612) claims this number is the perfect balance between computing efficiency and training stability.\n", "\n", "During an epoch the network will iterate over the whole dataset. Adding some shuffling in the process ensures the batches are not fed exactly in the same order at each epoch." ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "class BatchIterator:\n", " def __init__(self, batch_size=32, shuffle=True):\n", " self.batch_size = batch_size\n", " self.shuffle = shuffle\n", "\n", " def __call__(self, inputs, targets):\n", " starts = np.arange(0, len(inputs), self.batch_size)\n", " if self.shuffle:\n", " np.random.shuffle(starts)\n", "\n", " for start in starts:\n", " end = start + self.batch_size\n", " batch_inputs = inputs[start:end]\n", " batch_targets = targets[start:end]\n", " yield batch_inputs, batch_targets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training\n", "\n", "The training routine uses all objects defined above and executes actions **in the right order** to train the neural network.\n", "\n", "The dataset being usually small with respect to the number of free parameters of the neural net, going through the dataset multiple times during the training is a necessity. This corresponds to the number of epochs, which has to be specified.\n", "\n", "### Exercise #5 - build the training routine\n", "\n", "***10 min*** - write the sequential steps needed for training at each epoch\n", "\n", "_Hints_:\n", "- feed forward\n", "- compute the loss and the gradients\n", "- feed backwards\n", "- update the net" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "def train(self, inputs, targets, batch_size=32, epochs=2000):\n", " iterator = BatchIterator(batch_size=batch_size)\n", " for epoch in range(epochs):\n", " epoch_loss = 0.0\n", " for (batch_inputs, batch_targets) in iterator(inputs, targets):\n", " X = batch_inputs\n", " y_true = batch_targets\n", " # Compute the predictions of the current network\n", " y_predicted = self.predict(X)\n", " # Compute the loss\n", " batch_loss = self.loss.loss(y_predicted, y_true)\n", " epoch_loss += batch_loss\n", " # Compute the gradient of the loss\n", " grad = self.loss.grad(y_predicted, y_true)\n", " # Backpropagate the gradients\n", " self.backprop(grad)\n", " # Update the network\n", " self.update()\n", " \n", " # Print status every 100 iterations\n", " if epoch % 100 == 0:\n", " print(epoch, epoch_loss)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "# Add the function to the NeuralNet class\n", "NeuralNet.fit = train" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise #6 - solve XOR with our neural net - 1st attempt\n", "\n", "***5 min*** - Initialise the loss, the optimiser and create a neural net with a single linear layer. Then try it on the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an attempt at solving the XOR problem using a single linear layer" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 1.3300664685785624\n", "100 1.0000002203093972\n", "200 1.0000000000005151\n", "300 1.0\n", "400 1.0\n", "500 1.0\n", "600 1.0\n", "700 1.0\n", "800 1.0\n", "900 1.0\n", "1000 1.0\n", "1100 1.0\n", "1200 1.0\n", "1300 1.0\n", "1400 1.0\n", "1500 1.0\n", "1600 1.0\n", "1700 1.0\n", "1800 1.0\n", "1900 1.0\n", "\n", "X => y => y_pred => round(y_pred)\n", "[0 0] => [0] => [0.5] => [1.]\n", "[1 0] => [1] => [0.5] => [1.]\n", "[0 1] => [1] => [0.5] => [1.]\n", "[1 1] => [0] => [0.5] => [0.]\n" ] } ], "source": [ "# Initialize loss and optimizer\n", "sgd = SGD(lr=0.05)\n", "mse = MeanSquareError()\n", "\n", "# Create empty neural network\n", "net1 = NeuralNet()\n", "# Add layers\n", "net1.add(LinearLayer(input_size=2, output_size=4))\n", "net1.add(LinearLayer(input_size=4, output_size=1))\n", "# Add loss and optimizer\n", "net1.compile(loss=mse, optimizer=sgd)\n", "# Train the model\n", "net1.fit(X, y, batch_size=32, epochs=2000)\n", "\n", "y_pred_linear = net1.predict(X)\n", "\n", "print_xor_results(X, y, y_pred_linear)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's have a look at the decision contours." ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEzCAYAAABJzXq/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHeZJREFUeJzt3Xl8VPX97/HXZ5YkJCTsIZAFtOCC1lZEQaCK4O+q7b1y7y0gUGqtkICIiiuuoFlA0WLVUiutu1VE64Lbj1arVVtQUCoKuCCghE1QBGXLDH7vHzn1RgQyJDNzMpP38/Hg8Zhz5jvnvL9MeHPmnJmJOecQEREI+B1ARKSpUCGKiHhUiCIiHhWiiIhHhSgi4lEhioh46i1EM7vHzD4zs/f2c7+Z2e1mtsLMlphZz/jHFBFJvFiOEO8DTj/A/WcA3b0/ZcCdjY8lIpJ89Raic+5V4IsDDBkMPOBqLQBam1mneAUUEUmWeJxDLATW1Fmu9taJiKSUUDJ3ZmZl1L6sJitoxxW2ykzm7mUve75xfPLlbjJbZNOx5BACAV1jk9S3avm7m51zHRry2HgU4lqguM5ykbfue5xzs4BZAN3atXAzTusah91LY7z2yTZmvLGRjMwsJt3xANm5eX5HEmmUkT2LP2noY+NxSDAXONu72twH2OqcWx+H7UoS/KRLHlecWMCq995i6nkj+Hrbl35HEvFNLG+7eQSYDxxuZtVmNtrMxpnZOG/I88BKYAXwR2B8wtJKQpxYnMuV/Yv49P13qRo7nG1bDnQNTSR9mV9f/6WXzE3P2+u/Zur8TRQUd+XqOx+hVbsGnYYR8dXInsVvOed6NeSxOosu3+rZqSWT++bzWfWnVJQOY8umDX5HEkkqFaJ8xzEFOUzp14EvqldSPmYon29Y53ckkaRRIcr3HJWfzQ2nFPPVhk8pLx3KpnVr6n+QSBpQIco+HdG+BeWnFLN921Yqxgxl45rVfkcSSTgVouxX93YtqOrXht1fbKB8zBDWrf7Y70giCaVClAM6tE0WVYNK+GbbZipKh1K98kO/I4kkjApR6tWldSZVg0oI7NhCxZihfPrhcr8jiSSEClFiUtwqk6mDSghnZlI5dhirlr/rdySRuFMhSsw652Yw7YQWZEe/pmrscFa8u9jvSCJxpUKUg1LQMoOpg7qQx06mjR/J+4vf9DuSSNyoEOWgdcgJM3VQCa3b53PThF+ybNF8vyOJxIUKURqkXXaYaccGyA9HmX7h2by74FW/I4k0mgpRGqxNixBVg0ronOW4ZeK5LH79735HEmkUFaI0SqusEBUDSyjJgRmXjGHRK/P8jiTSYCpEabS8zCDlA0voesRR3HbFOBb87Vm/I4k0iApR4qJlRpDKI7ZzWOswd1x1Pv984Um/I4kcNBWixE12OMiUAcUc1T6L3197Ea8+85jfkUQOigpR4qpFOMDkk4s4Jr8Fd025hL8/8bDfkURipkKUuMsMBbj25CJ6ds7hT5WT+Ouj9/kdSSQmKkRJiIxggKv6F9K7sCX33XQdzz/0R78jidRLhSgJEw4GuKJ/IX2Lc3loRjlz753pdySRA1IhSkKFAsZlfTtzUpc8Zt9xI0/M+i1+/aZHkfqE/A4g6S8YMCb26UQoAI//4TdEIzUMHX85ZuZ3NJHvUCFKUgQDxgW9OxEKGE/dfQeRmhpGTrxGpShNigpRkiZgxnnHFxAOGM89eBfRaISzL7tepShNhgpRkipgRulxHQkFjKcfuYdoTQ2/vqqKQECns8V/KkRJOjPj18fmEw4GePwvDxGN1FB63XQCwaDf0aSZUyGKL8yMUce0JxwwHpk7h2gkwrgbZhAM6UdS/KOfPvGNmTH8h+0JBYwHX3iSPdEI4ytvJxQO+x1NmikVovhuyFHtCAXg3r89SzQa4cIbf08onOF3LGmGdCZbmoT/fWQ7So/LZ9HL87j10jJqdu/yO5I0QypEaTL+52FtOe/4jix+/SV+c/Fodu/c6XckaWZUiNKknN6tDRf2LuC9Ba9y80XnsGvnDr8jSTOiQpQmZ9ChrZl4YieWv72Am84fxc7tX/sdSZoJFaI0SQO6tuKyEwv46N23mTb+F2z/aqvfkaQZUCFKk9W/JI9JfQtYvfRtpo4bwddbt/gdSdKcClGatD5FuVzVv4jqD96jauxwtm353O9IksZUiNLk9SpsyTUnFbHuk4+pLBvG1s83+R1J0pQKUVLCsZ1ymNwvn02rP6K8dChbNm3wO5KkIRWipIxjOuZw/SnFbKleRfmYoXy+YZ3fkSTNqBAlpfTokE35KcV8teULyscM4bO1n/odSdKIClFSzuHtW1DRry07vt5GRelQNny6yu9IkiZUiJKSurXNorJvGyJbNlJROpS1q1b4HUnSgApRUtahbbKoHFjCN9s2U1k2jDUr3vc7kqQ4FaKktC6tM5l6aglmRmXZWXzy4TK/I0kKUyFKyivKy2Ran5Zk1GyjsmwYK5e943ckSVExFaKZnW5mH5jZCjO7ch/3l5jZy2a22MyWmNlP4x9VZP8652YwbVAJOXt2UDVuBB8tedvvSJKC6i1EMwsCM4EzgB7ACDPrsdewa4E5zrljgeHA7+MdVKQ+HVtmMHVQCa3ZxbTxI3n/7Tf8jiQpJpYjxBOAFc65lc65GmA2MHivMQ7I8263AvSOWfFFh5wwVYNKaJNfwE0TfsnShf/yO5KkkFgKsRBYU2e52ltX1/XAKDOrBp4HLtjXhsyszMwWmdmibbuiDYgrUr922WFu/LHRMXMP0y88myXz/+F3JEkR8bqoMgK4zzlXBPwUeNDMvrdt59ws51wv51yvvCz9fitJnNYtQlQOLKGoBdwy8VwWv/aS35EkBcRSiGuB4jrLRd66ukYDcwCcc/OBLKB9PAKKNFSrrBAVA0vokhvg1ovPYeHfX/A7kjRxsRTiQqC7mR1iZhnUXjSZu9eYT4FBAGZ2JLWFqO9oEt/lZgYpP6WYH7TJ4rZJ57Hgr8/4HUmasHoL0TkXBSYA84Dl1F5NXmpm5WZ2pjfsUqDUzN4BHgHOcc65RIUWORgtM4Jcf0oxR7TN4I6rJ/D680/6HUmaqJhO5Dnnnqf2YknddZPr3F4G9ItvNJH4yQ4HmTKgmMp/VHPndRcRjdQwYPBZfseSJkafVJFmIysU4LqTizi690+YdcNlvPT4Q35HkiZGhSjNSmYowOSSDfTqnMPdU69i3ux7/Y4kTYgKUZqdjGCAK/sX0buoJfdPn8xzD97ldyRpIlSI0iyFg8YV/QrpV5zLn2+t5Km77/A7kjQBene0NFuhgHFp386EFqxnzszpRCMRfj72YszM72jiExWiNGvBgHFRn04EA8YTs24lGqnhrAmTVIrNlApRmr1gwLigdwEZQWPuvTOJRmr4xcXXqRSbIRWiCBAwY1yvjgQDxnMP/ZFoJMLZl99AIKDT7M2JClHEY2aU9swnHDCeevQ+IjU1jL5mmkqxGVEhitRhZpzz4w6EAsbjTz7MnmiEssk3EwgG/Y4mSaBCFNmLmTHqmPaEA8YjzzxGNBrhvBtuJRjSP5d0p2dYZB/MjOE/bE8oaDz4wlNEIxEmVN1BKBz2O5okkE6OiBzAkB7tOPfYfN588TlumzSOSM1uvyNJAqkQReox+Ii2lB3Xkbde+StzZt7sdxxJIBWiSAx+dlgbin5wGOs/+djvKJJAKkSRGB1y5DG8/eqLvPSXP/sdRRJEF1VEYnRh9hJ2dM7h7qoriUYinDb8HL8jSZzpCFEkRhnBAFf1L6R3YUvun34dzz00y+9IEmcqRJGDEA4GuKJ/IX2Lc/nzjAqevud3fkeSOFIhihykUMC4rG9nTuqSx6O/u4m/3HUr+p1q6UHnEEUaIBgwJvbpRChg/OWuGUQjNQw7/wp9Q06KUyGKNNB/vjYsFICn7/kd0UgNIydeq1JMYSpEkUYImHHe8QWEAsZzD8769mvDVIqpSYUo0kgBM8qO60g4YDw9+14iNTWce/VUfW1YClIhisSBmfHrY/MJBwM8/sSfiUZq9LVhKUiFKBIn//nasFAAZj/zGHuiUcbdMENfG5ZC9EyJxJGZMeKHHQgHAjz4wpNEoxHOr7xdXxuWIlSIIgkw5Kh2hIPGPX97lj3RCBfe+HtC4Qy/Y0k9dNZXJEH+87Vhi16ex62XllGze5ffkaQeKkSRBPrZYW0Yf3wBi19/id9cPJrdO3f6HUkOQIUokmCndWvNBb0LeO+N17j5onPYtWO735FkP1SIIklw6qGtmdingOVvL+DGCb9kx9df+R1J9kGFKJIkA7q24vITC1jxzkJuHP8Ltn+11e9IshcVokgS9SvJY1L/QlYvW8zUcSP4eusWvyNJHSpEkSTrU5TLVf2LqP74Q6rGDmfbls/9jiQeFaKID3oVtuSavvms++RjKsuG8eXmz/yOJKgQRXxzbKccJvfLZ9Pqj6goG8YXn633O1Kzp0IU8dExHXOYMqCYLdWrqBgzlM3r1/odqVlTIYr47Kj8bMpPKearL7dQUTqUz9Z+6nekZkuFKNIEHN6+BRX92rJz81rKxwxhw6er/I7ULKkQRZqIbm2zqBxYQvTLzygfM4S1q1b4HanZUSGKNCGHtKktReccFaVDWbPifb8jNSsqRJEmpkvrTKb2aUkwGKSibBirP1jqd6RmQ4Uo0gQV5WUyrXcOWTVfUTX2LFYue8fvSM2CClGkieqUm8HUQSXk7NlB1bgRfLTkbb8jpb2YCtHMTjezD8xshZlduZ8xw8xsmZktNbOH4xtTpHnq2LK2FPPatGPa+JG8v/hNvyOltXoL0cyCwEzgDKAHMMLMeuw1pjtwFdDPOXcUMDEBWUWapQ45Yab1DNIuGOGm80exdOG//I6UtmI5QjwBWOGcW+mcqwFmA4P3GlMKzHTObQFwzumDmSJx1C47zNRBJXTM3MP0C89myfx/+B0pLcVSiIXAmjrL1d66ug4DDjOzf5rZAjM7PV4BRaRW6xYhqgaVUNQCbpl4Lotfe8nvSGknXhdVQkB3YAAwAvijmbXee5CZlZnZIjNbtG1XNE67Fmk+8jJDVAwsobjb4cy4tJSFL/+335HSSiyFuBYorrNc5K2rqxqY65yLOOdWAR9SW5Df4Zyb5Zzr5ZzrlZel34Aq0hC5mUEqe+yiW6sgt10xjgV/fcbvSGkjlkJcCHQ3s0PMLAMYDszda8xT1B4dYmbtqX0JvTKOOUWkjpYZQa4/pZgj2mZwx9UTeP25J/yOlBbqLUTnXBSYAMwDlgNznHNLzazczM70hs0DPjezZcDLwOXOOX0NsEgCZYeDTBlQzNHts7jzuot45elH/Y6U8sw558uOu7Vr4Wac1tWXfYukk93Rb5j22loWb9jO6KunMWjIKL8j+Wpkz+K3nHO9GvJYfVJFJMVlhgJcfVIhx3fO4e6pVzFv9r1+R0pZKkSRNJARDDCpfxG9i1py//TJPPvAH/yOlJJUiCJpIhw0ruhXSL+SXB7+bRVP3X2H35FSjt77IpJGQgHj0hM7Ew6sZ87M6UQjEX4+9mLMzO9oKUGFKJJmggHjwt6dCJrxxKxbiUZqOGvCJJViDFSIImkoGDAm9C4gHDTm3juTSM1uRl0yWaVYDxWiSJoKmDGuV0dCAePZP/+JaCTCr64oJxDQpYP9USGKpDEzY0zPfEIB46k59xONRBh9zTSV4n6oEEXSnJlxzo87EAoYjz/5MNFIDWOn3EIgGPQ7WpOjQhRpBsyMX/6oAxlB4+FnH2dPNMJ55b8lGFIF1KW/DZFm5Kyj2xMKGA/899NEo1EmVN1BKBz2O1aToRMJIs3Mz3u049xj83nzxee4bdI4IjW7/Y7UZKgQRZqhwUe0pey4jrz1yl+59bIyanbv8jtSk6BCFGmmfnZYG8YfX8A7/3yZWyaey+6dO/2O5DsVokgzdlq31lx4QgFL33ydmy/6Fbt2bPc7kq9UiCLN3MBDW3FxnwLef2s+N074JTu+/srvSL5RIYoIJ3dtxWV9O/PxOwuZNn4k27/a6nckX6gQRQSAfiV5TOpfyOr3l1I1dgRffbnF70hJp0IUkW/1Lsrlmn4dWbvyQ6rGnsXWLzb7HSmpVIgi8h3HdW7Jtf3y2bDyfSpLh7Fl00a/IyWNClFEvufHBTlMGVDM52tWUFk2jM83rvc7UlKoEEVkn47Oz+b6AcV8ufkzKkqHsGldtd+REk6FKCL7dWSHbG7o247tG6upKB3KxupP/I6UUCpEETmgw9u3oGJgCbs/X0fFmCGs/2Sl35ESRoUoIvX6QdssKgeWsGfrJipKh7J25Ud+R0oIFaKIxKRrmywqB5UAUFE6lE8/Wu5zovhTIYpIzEpaZVLVO4fQrq1Ujj2L1e+/53ekuFIhishBKcrLZOqgElrUfEXl2OF8vPTffkeKGxWiiBy0TrkZTD21Czl5rZh63kg+fOctvyPFhQpRRBokPyfMtOPCtLFd3Hj+L1j+1gK/IzWaClFEGqxDTpjKgSW0D0WYfsHZvPfG635HahQVoog0SrvsMFUDSyjI/IabJ57DO/96xe9IDaZCFJFGa90iROWgYjp37cZvLh7NW//4m9+RGkSFKCJxkZcZouqoGrrmBvjt5WW8+dILfkc6aCpEEYmb3Mwg5acU0611mNuvPI/58+b6HemgqBBFJK5yMoJcP6CII9tmMPPq83nt2b/4HSlmKkQRibvscJDJA4o5Oj+bP0y5mFeemu13pJioEEUkIbJCAa49qYhjC7KZVX45f3vsAb8j1UuFKCIJkxkKcPVPCjm+sCX3TruGFx6+2+9IB6RCFJGECgcDTOpXyIlFLXnwlut55r47/Y60XypEEUm4cNC4rF8h/UtyeeT2qTz5p9v8jrRPIb8DiEjzEAoYl5zYmVBgPY/9/haikQhDxl2Kmfkd7VsqRBFJmmDAuKhPJ0IB48k/3ka0pobhF17VZEpRhSgiSRUw4/wTCggHjGfuv5NopIZRl05pEqWoQhSRpAuYMbZXR0IB45mH7yYSqeGcSZUEAv5e1ohp72Z2upl9YGYrzOzKA4z7uZk5M+sVv4giko7MjNE98/k/R7Tlxcce5E+Vk/jmm298zVTvEaKZBYGZwH8B1cBCM5vrnFu217hc4CLgjUQEFZH0Y2b86scdCAWNx56azZ5olLFTbiEQDPqSJ5YjxBOAFc65lc65GmA2MHgf4yqAm4BdccwnImnOzBh1TAdG/rA9rz37ODOvvZBoJOJLllgKsRBYU2e52lv3LTPrCRQ7556LYzYRaUbOOro9v/pRB+bPm8vvrp5ANFKT9AyNPoNpZgFgBnBpDGPLzGyRmS3ativa2F2LSJr5vz3aMaZnPm++9Dy/vXwskZrdSd1/LIW4Fiius1zkrfuPXOBo4BUzWw30Aebu68KKc26Wc66Xc65XXpYucIvI9/2vw9syrldH3n71RWZcMoaaXTuTtu9YCnEh0N3MDjGzDGA48O23Pjrntjrn2jvnujrnugILgDOdc4sSklhE0t4Z3dsw4YQClsz/BzdPPJddO3ckZb/1FqJzLgpMAOYBy4E5zrmlZlZuZmcmOqCINE//9YPWXNS7gGWL/sX0C85m5/avE75Pc84lfCf70q1dCzfjtK6+7FtEUsern2zj1vnr6HZML664/X6yc/MOOH5kz+K3nHMNei+0vu1GRJq0k7rkcXm/zny8ZBFTx4/k621fJmxfKkQRafL6Fudx5U8K+fTD5VSNHc62LV8kZD8qRBFJCScU5nJN33zWr1hG1dhhbP1ic9z3oUIUkZTRs3NLrj2piI2rPqSydBhbNm2M6/ZViCKSUn5UkMOUAcVs3rCWitKhfL5xfdy2rUIUkZRzdH42N/Rrz9YvNlNROoRN66rjsl0VooikpCM7ZFPety3bN1ZTMWYIG9esbvQ2VYgikrIOa9eCioEl7P5iPRWlQ1m3+uNGbU+FKCIp7Qdts6gcVEI0EqGidGijtqVCFJGU17V1FlUn5pEb2dqo7agQRSQtlLTK5PYzDmnUNlSIIpI2goHG/eY+FaKIiEeFKCLiUSGKiHhUiCIiHhWiiIhHhSgi4lEhioh4VIgiIh4VooiIR4UoIuJRIYqIeFSIIiIeFaKIiEeFKCLiUSGKiHhUiCIiHhWiiIhHhSgi4lEhioh4VIgiIh4VooiIR4UoIuJRIYqIeFSIIiIeFaKIiEeFKCLiUSGKiHhUiCIiHhWiiIhHhSgi4lEhioh4VIgiIh4VooiIR4UoIuKJqRDN7HQz+8DMVpjZlfu4/xIzW2ZmS8zsJTPrEv+oIiKJVW8hmlkQmAmcAfQARphZj72GLQZ6OeeOAR4Hpsc7qIhIosVyhHgCsMI5t9I5VwPMBgbXHeCce9k5t8NbXAAUxTemiEjixVKIhcCaOsvV3rr9GQ28sK87zKzMzBaZ2aJtu6KxpxQRSYJQPDdmZqOAXsDJ+7rfOTcLmAXQrV0LF899i4g0ViyFuBYorrNc5K37DjM7FbgGONk5tzs+8UREkieWl8wLge5mdoiZZQDDgbl1B5jZscBdwJnOuc/iH1NEJPHqLUTnXBSYAMwDlgNznHNLzazczM70ht0MtAQeM7N/m9nc/WxORKTJiukconPueeD5vdZNrnP71DjnEhFJOn1SRUTEo0IUEfGoEEVEPCpEERGPClFExKNCFBHxqBBFRDwqRBERjwpRRMSjQhQR8agQRUQ8KkQREY8KUUTEo0IUEfGoEEVEPCpEERGPClFExKNCFBHxqBBFRDwqRBERjwpRRMSjQhQR8agQRUQ8KkQREY8KUUTEo0IUEfGoEEVEPCpEERGPClFExKNCFBHxqBBFRDwqRBERjwpRRMSjQhQR8agQRUQ8KkQREY8KUUTEo0IUEfGoEEVEPCpEERGPClFExKNCFBHxqBBFRDwqRBERjwpRRMQTUyGa2elm9oGZrTCzK/dxf6aZPerd/4aZdY13UBGRRKu3EM0sCMwEzgB6ACPMrMdew0YDW5xz3YBbgZviHVREJNFiOUI8AVjhnFvpnKsBZgOD9xozGLjfu/04MMjMLH4xRUQSL5ZCLATW1Fmu9tbtc4xzLgpsBdrFI6CISLKEkrkzMysDyrzF3YMfef+9ZO4/ydoDm/0OkUDpPL90nhuk//wOb+gDYynEtUBxneUib92+xlSbWQhoBXy+94acc7OAWQBmtsg516shoVOB5pe60nlu0Dzm19DHxvKSeSHQ3cwOMbMMYDgwd68xc4FfebeHAH93zrmGhhIR8UO9R4jOuaiZTQDmAUHgHufcUjMrBxY55+YCdwMPmtkK4AtqS1NEJKXEdA7ROfc88Pxe6ybXub0LGHqQ+551kONTjeaXutJ5bqD57Zfpla2ISC19dE9ExJPwQkz3j/3FML9LzGyZmS0xs5fMrIsfORuivrnVGfdzM3NmllJXLmOZn5kN856/pWb2cLIzNkYMP5slZvaymS32fj5/6kfOhjCze8zsMzPb51v3rNbt3tyXmFnPmDbsnEvYH2ovwnwMHApkAO8APfYaMx74g3d7OPBoIjP5ML9TgGzv9nmpMr9Y5uaNywVeBRYAvfzOHefnrjuwGGjjLef7nTvO85sFnOfd7gGs9jv3QczvJKAn8N5+7v8p8AJgQB/gjVi2m+gjxHT/2F+983POveyc2+EtLqD2fZypIJbnDqCC2s+u70pmuDiIZX6lwEzn3BYA59xnSc7YGLHMzwF53u1WwLok5msU59yr1L6jZX8GAw+4WguA1mbWqb7tJroQ0/1jf7HMr67R1P6vlQrqnZv3MqTYOfdcMoPFSSzP3WHAYWb2TzNbYGanJy1d48Uyv+uBUWZWTe27SC5ITrSkONh/m0CSP7rXnJnZKKAXcLLfWeLBzALADOAcn6MkUojal80DqD2yf9XMfuic+9LXVPEzArjPOfcbMzuR2vcSH+2c+8bvYH5J9BHiwXzsjwN97K+JimV+mNmpwDXAmc653UnK1lj1zS0XOBp4xcxWU3ueZm4KXViJ5bmrBuY65yLOuVXAh9QWZCqIZX6jgTkAzrn5QBa1n3NOBzH92/yeBJ/4DAErgUP4/yd2j9przPl896LKHL9P2MZ5fsdSe3K7u9954z23vca/QmpdVInluTsduN+73Z7al2Dt/M4ex/m9AJzj3T6S2nOI5nf2g5hjV/Z/UeVnfPeiypsxbTMJoX9K7f+sHwPXeOvKqT1agtr/lR4DVgBvAof6/Rcd5/m9CGwE/u39met35njNba+xKVWIMT53Ru1pgWXAu8BwvzPHeX49gH96Zflv4H/4nfkg5vYIsB6IUHskPxoYB4yr89zN9Ob+bqw/m/qkioiIR59UERHxqBBFRDwqRBERjwpRRMSjQhQR8agQRUQ8KkQREY8KUUTE8/8AxEcutRX5jmwAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Apply the decision function on the grid input matrix\n", "y_linear_grid = net1.predict(X_grid)\n", "# Reshape the array to recover the squared shape\n", "y_linear_grid = y_linear_grid.reshape(rows.shape)\n", "# Plot the decision\n", "plot_decision_contours(y_linear_grid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A single linear layer does not work, as expected. XOR is a typical non-linear problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise #7 - solve XOR with a Neural Net - 2nd attempt\n", "\n", "***5 min*** - Write a more advanced neural net (using additional linear and activation layers) until the predictions match the target values." ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 1.5909322284602154\n", "100 0.781992483663568\n", "200 0.6258952466537688\n", "300 0.4283373434041924\n", "400 0.2177336026642543\n", "500 0.08181187313441049\n", "600 0.024396985375692515\n", "700 0.006296720599011931\n", "800 0.0015022465010545297\n", "900 0.000344443628963212\n", "1000 7.746439185520821e-05\n", "1100 1.726138427065125e-05\n", "1200 3.8295631446871435e-06\n", "1300 8.478645246994355e-07\n", "1400 1.8753467659609788e-07\n", "1500 4.1460841273162615e-08\n", "1600 9.164341033861374e-09\n", "1700 2.0254450356827815e-09\n", "1800 4.4762984777258344e-10\n", "1900 9.892542095055063e-11\n", "\n", "X => y => y_pred => round(y_pred)\n", "[0 0] => [0] => [1.03452335e-06] => [0.]\n", "[1 0] => [1] => [0.99999817] => [1.]\n", "[0 1] => [1] => [0.99999734] => [1.]\n", "[1 1] => [0] => [3.22153239e-06] => [0.]\n" ] } ], "source": [ "sgd = SGD(lr=0.01)\n", "mse = MeanSquareError()\n", "\n", "net2 = NeuralNet()\n", "net2.add(LinearLayer(input_size=2, output_size=4))\n", "net2.add(Tanh())\n", "net2.add(LinearLayer(input_size=4, output_size=1))\n", "\n", "net2.compile(loss=mse, optimizer=sgd)\n", "\n", "net2.fit(X, y, batch_size=32, epochs=2000)\n", "\n", "y_pred_tanh = net2.predict(X)\n", "\n", "print_xor_results(X, y, y_pred_tanh)" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEzCAYAAABJzXq/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd0VNXax/Hvnsmk95CQnoAgXQFBwUovomK7KiqKIrZroQpK79K5ehWxYkEs2FBpijSRjiBSRUggBQJJCAnpyX7/yIQXuSiBZHIyZ57PWqyVmZzMPMeJv5yz9zn7UVprhBBCgMXoAoQQoqaQQBRCCDsJRCGEsJNAFEIIOwlEIYSwk0AUQgi7CwaiUupdpVSaUur3v/m+Ukq9opQ6oJT6TSnVsurLFEIIx6vIEeI8oNs/fL87UN/+73FgTuXLEkKI6nfBQNRarwEy/mGTnsAHuswGIFApFVFVBQohRHWpijHEKODIWY+T7M8JIYRTcavON1NKPU7ZaTWeVnVVVIBHdb69a9KapFOFFJZoavva8HG3Gl1RtSkuKSUxqxB3qyLC14YlsoHRJTlU0sE/KC4qJCwqFm9fP6PLMcyhPTtPaK1DL+VnqyIQk4GYsx5H25/7H1rrN4E3AeqFeOmZXeOr4O3FhZzILWLy2mQOZOTTq1kA9zQJwaKU0WVVi20pOUz/JYWMvGKeee5Fml1zg9ElOczxlCRmDe5H4r5d3PXkQG7v+xwWi+tdSHJ/y5jES/3ZqvivtQh4yD7b3AbI0lqnVsHriipSy9vGpI6xtIv3Z8HOE0z5OZmC4lKjy6oWLSN9mdE1nmAvG1Oevp/vPngDsy5oEhoZzZh3v+S67newcM4MZg95nIK8PKPLcioXPEJUSi0A2gG1lFJJwGjABqC1fgNYDNwMHABygUccVay4dB5uFvq3iaBukCfbj57GzeIaR4gAEX7uTOkcx382pvLx7Ikk7NtFvxFT8fDyMrq0Kufu6cVT42dTp1Ez9mzbgM1DhqUuhjLqr6WcMhunVGssSpGZV0xiVgHNw32MLqlaaK35fHc6H+9MJ/byxgyc8RahkTEX/kEnpbVGKUX6sVQO/7GHFtd3MLqkanF/y5itWutWl/KzrjfAIM6MH87feZyxq47w1Z50055Gnk0pxT1NajHixihOHNzD8Ad7sGvTOqPLchhl/5y/mDuT6c/34cu3ZlNa6hpDJZdKAtGFPdayNm2j/Zi3/Tgz16e6zLhiq0hfpneNJ6j0NJOevp/FH71l6j8IfV4Yd2ZccdbgfuTmZBtdUo0lp8wuTmvNwt3pzP/tBHWCPBh+YzS1vG1Gl1UtcotK+M+GVDYk5dAu3p8+H2/B3dN844pQ9jkvXfAu82eNJzymDi+8+j5hUbFGl+UQcsosLplSin81qcWIG6OxKIWnm+v8SnjbrAy9PooHmtVidcIpxjx6F8dTkowuyyGUUnS/vy8vzVmAf3AIvgFBRpdUI8kRojijfBC+qKSUDUk5XB/rd2Ycyuw2J+cwc30KVp9Anp/6Bo1btTW6JIcp/5wL8vJY893ndLzrQVNdryhHiKJKlIff8j+zmP5LCv/ddJSiEtcYV2wd5cv0LvEElOYw6Yl7WPLxO6YdVyz/nNct+Yr3Jg9n1qDHZFzRTgJR/I/u9QP5V5MQfjyYxfAVh0nPLTK6pGoR5e/OtC5xtI705cPpY3hj9EAK8817YXP7O3rx0JCx/PrzT4x66DZSEv40uiTDSSCK/2FRigevCGXo9ZEkZhUwaHkif6SbNxjO5m2zMuyGKHo1rcXa7xYytu9dpB9NMbosh1BK0a3Xo7w0ZwE5WZmMfOhWdm5ca3RZhpJAFH/r2hh/pnSOI9DD6lKTLRaluK9ZLYbfGMWxA7sYcce17Nm63uiyHKZxq7ZM+Oh7LmvSnNCIaKPLMZTr/JaLSxIf6MmsbvHEBHigtWZVQhbFpeYcWzvX1VF+TOsSh6+7lYmP38OyT94z7bhirYgoXprzMeGxddBas/ijt8jNPmV0WdVOAlFcUPkg/J4Tecxan8rolYfJyi82uKrqEe3vwfSucVwV6cv7U0cxd8xACgvyjS7LoRL37WLBK5MY+fBtJB86YHQ51UoCUVRY41BvBrSJYH96PoOWJXAww9zBUM7bZuWlG6K4r2kIa75dyDgTjysCxDdsyktzFnA66ySjHrqVLauWGV1StZFAFBelXZ0AJnWMpRQY+mMi6w67xmmVRSl6NQvlxRuiSNn/O8Mf7MHebRuNLsthGl3VhonzFxMRV5eZAx/juw/eMLqkaiGBKC5a/RAvZnaJ5/IQTzysrvUr1Cbaj+ld4vAtOsXEfnez/NN5ph1XDAmPZNQ7X9Cu573ENWhidDnVQu5UEZes/I4HgPVHsmlW2xtfF2lRcLqwhJnrU9iScpp2Pe+lz7AJuHt4Gl2Ww/305cc0aN6aqLr1jS7lb8mdKsIQ5WF4Mq+YmetTGLQsgcNZBQZXVT183K0MvzGae5qEsOqbTxnf719kpJl7ofjcnGwWvjGDUQ/fZtpxRQlEUWmBXm6Max9DfnEpQ5YnsiHJNW4DsyjFA1eEMuz6KJL3/saI29uy99dNRpflMN6+foz/4Fsi4i9j5sDHWDhnhunWV5RAFFWiUag3M7rGE+PvzuS1yXz6+wmjS6o2bWPKrlf0crMw8bG7+OHzD8w9rvj2Qm667R6+fGs2/xnyhKn2tVrbkApzq+VtY1KnWOZsPupSPVsAYgM8mN41npnrU3hv8nAO7dnJI8MmYHM3X08Tdw9PHh89nTqNmqE1ploRSSZVRJUr/51SSvF7Wi7BXm5E+rkbXFX1KCnVfLzzBAt3p1OvWUsGTJ9LUGi40WU53LY1P1JSUkzr9t2MLkUmVUTNopRCKUVJqea1TakMXpbAtpQco8uqFlaLoveVZQtjJO35leEP9GD/ji1Gl+VQ5atxzxrUj89en+bU44oSiMJhrBbFmHYxhPrYGL8miS93u0YzKyhbGGNq53g88zOZ0PcO1j5xSQcsTkEpxeDZ73LTbffw9duvMKP/o5zOzjK6rEsigSgcqrZvWU/ktjF+vL/jODN+SaGoxDVCMS7Qgxld47mitg9zNh/jrfFDKSo052VJ5eOKjwybwG8bVjOy963kZGUaXdZFk0kV4XCebhaGXBtJ3aAMjmQV4EIrieHrbmXEjdHM/+04X3z1MUl/7qP/tLkEhdY2urQqp5Si8z0PE1O/EdtW/4CPf6DRJV00mVQR1ar87paU7ELSc4toVtvH6JKqzbrDp/jPhlS83a30n/MF9a9oaXRJDpd0cD8bf/ieO/o9X219W2RSRTiN8ks0PtiexqiVR/h+f6bLjCteF+vP1C5xeFgV4x67m5++/Njokhxuw7Jv+WLuTKcZV5RAFIZ4rk0ErSJ9eXPrMV7deJRCF2lmFR/oyfQu8TSr5c7bE4byzqQXKS4qNLosh7nryYFnxhVHPHgLSX/uM7qkfySBKAzhbbPy4g1R3Ns0hBWHyppZnXSRRWf9PKyMuimaOxsFs2LhR0zu0ZSTJ9KMLsshyscVh8/9lPzc04x6uCcHd+8wuqy/JYEoDGNRivubld0L7G614OVCsy1Wi+Lh5mEMvjaSg5n5jOh5DQd2/mp0WQ7TsMXVTJz/PTf0uIuYeg2NLudvuc5voKix2sb4MaFDDB5uFk4XlrAm0TUWnQW4Ic6fqZ3jsFnKxhVXff2J0SU5THBYBI+8OBGbuwc5WZm8PWFYjRtXlEAUNUL5ZMu3+zOZ8UsKc7ccdZlmVvFBnkzvGk+TYDfeHDeE9yYPN/W4IsD+HVtZveizGjeuKIEoapR/NQ7h9obBLP7jJKN+cp1mVv4eVka3i+H2hsH88PkHTHziPrLSjxtdlsO0vLETI978lIK8XEY+dBubViw2uiRAAlHUMFaL4pEWYQxoG8EfGWXNrBJOukYzq/J9H9Q2kkM7tzD8gZv5c9d2o8tymAbNWzNx/vfEXNaA2UOeqBHDBRKIokZqFx/A5E5xBHq54ecibQnK3Rjvz5TOcVjdbIzrezerF31mdEkOExQazsi3P+fWh5+ixQ0djS5HAlHUXPWCPZnWOY4QbxslpZof/jxJiYuMK9YN8mRmGw8aBVmZO2YQ86aMpLioyOiyHMLm7kGv518iICSUkuJi3hw7mCMH9hpSiwSiqNHKJ1u2pOTw301HGbc6iZzCEoOrqh7+Hm6MaRdDzwZBLP90HlNuaUJWhrlXIj+RmsT2dSsZ9XBPNv74fbW/vwSicArXRPvx76vD+T3ttEs1s7JaFI+2rM1A+5jqiNta1+gLmyurdkx82bhivYb854Un+eTVlyktqb4/gBKIwml0uSyQCR1iKbA3s9qc7BqLzgLcFB/AlE5xWICxj97Fmu8WGl2SwwSFhjPyrc9of8f9LHrvNd6Z9GK1vbcEonAq5c2s6gZ54O/hWpMtdYM9mdE1noaBVt4YNYD3p4029bhiv5FT6Dv8ZTrd3bva3leW/xJOqXwZMYBVh7K4OtoXb5trBGRJqea97Wl8uy+Txq3a8tyUOfgHhRhdlsMteGUydRo1o03nW/5xO1n+S7ic8jBMzS7klY2pDFmeSEq2ue/uKGe1KB5rWZv+bSL4Y+c2RjzQg0N7dhpdlkMVFuSzd9tGXhn6lEPHFSUQhVOL8HNnbPsYsvJLGLwsga0u0swKoH2dAF5uFwGnjjH2oR6s/e4Lo0tyGHcPT0a8+Skd7nyARe+9xtTnHnZIiwIJROH0mtX2YUbXOMJ8bIxfncQ3ezOMLqna1LOPK14e4smcUf35cPoYSorNebujzd2Dx0a8zGMjprBr8y+M73dPlR8pSk8VYQq1fd15uXMcr25Mdbk7WwI93RjbPpb3fk3ju4/f4fAfe3n25dfxDwo2ujSH6HDn/URf1oCs9DQs1qr9rGVSRZjK2ZMtm5NziA1wp7avu8FVVZ+fDmbx+uajBNSOYuCMt4hv2NTokhzupy8/5tiRBO59ZigWq1UmVYQoVx6GBcWlvLYplcHLE9l57LTBVVWfDnUDmNwpFp11lDGP3sG6JV8ZXZLDHflzH9++P4cpzz5U6XHFCh0hKqW6Af8BrMDbWuuXz/l+LPA+EGjfZpjW+h/X85EjROFoyacKmbQ2iZTsQvq2DKNH/aAzgWl2J/OLmfpzMruO53F7w2Du+mArVjfzjpCt/GoB7708guDa4aQlHXbcEaJSygq8BnQHGgO9lFKNz9lsBPCZ1roFcB/w+qUUI0RVivJ3Z1qXOFpF+vLW1jRe3XiUUhfp8Bfo6ca4DrHcXD+Qr/dmMO3WJmSfdL7G8RXV/o5ejHzrczw8vSr1OhU5Zb4aOKC1Pqi1LgQ+AXqes40G/O1fBwAplapKiCpydjMrPw8rFhc5QgRwsyieaBXOs9eEs/t4HiMe7EHi/t1Gl+Uw9a9oyeRPllfqNSoSiFHAkbMeJ9mfO9sY4EGlVBKwGHj2fC+klHpcKbVFKbXllIushCyMV97Mqk/zUAD+SM9j74k8g6uqPp3qBjK5UyylmamMfqAbvyz7xuiSHMZiqdy0SFVNqvQC5mmto4GbgQ+VUv/z2lrrN7XWrbTWrfw9zTueIWqm8vHDedvTGL4ikR/+PGlwRdXn8hAvZnSNp16wJ/998Rnmz5pg2usVK6MigZgMxJz1ONr+3Nn6Ap8BaK3XA55AraooUIiqNuz6aJqG+fDfTUddqplVkJcb49rH0r1eIN9/OJepz1V+VtZsKhKIm4H6Sqk6Sil3yiZNFp2zzWGgI4BSqhFlgWjeDjnCqZU3ir/jrGZWp11k0VmbVfFk63CeuTqcPVs3MvzBWzi8f4/RZdUYFwxErXUx8AywDNhD2WzyLqXUOKXUbfbNBgH9lFI7gAVAH23UFd9CVIDVoujTIoyBbSPw9bDi6eZal+R2viyQSe0iKMlIYfQDXdmw/FujS6oR5E4V4fLK727JyCtm9/Fcro/1v/APmURGXjFTfk5m74k87mwUzJ0fbKvy2+Gqm9ypIkQllE+2fLk7nWnrUnjv1zSXaWYV7OXGhA6xdK0XyJd7MspWkTnlOpNN55JAFMKuT4swutsvZB63OonsAtcZV3y6dTj/bh3O7o2rGfngLRz+wzXHFSUQhbBzsyiebBV+ppnV4OUJLrPoLECXeoFM7BhHYXoyo/vcbkjXO6NJIApxjvJmViFebi7Xt6VhLS9mdIkj3kcb0vXOaBKIQpxHo1BvJnaMxdfdSmFJKUsPZLrMfdAh3jYmdoihy2UBLHrvNab1f8RlxhUlEIX4G+WTLWsSTzFn8zFeXptMbpFrHC3ZrBb+fXUET7Wuza71Kxl9S0uS/txndFkOJ4EoxAV0rBPAYy3D2JyS41LNrAC61QtiQodY8otLGdWrM5tWLDG6JIeSQBTiApRS3Nog+C/NrH5zoUVny3thxwZ4MHvI43z22lTTjitKIApRQVfYm1nVCfIg2Mu1FicJ8bYxqWMsnS8L4Ot3XmX6gEc5nZ1ldFlVTgJRiItQ29edCR1iifb3QGvNkj8yKSguNbqsamGzWvh363CebFWbnet+YmTvW0k6uN/osqqUBKIQF6l8suVARj5ztxxj6A+JHMtxjXFFpRTd6wcxvkMseccOM+q+zux8vo3RZVUZCUQhLlH9EC9G3hRN2ukiBi9PdKlxxSZh3szsFk+MvzuT1yazcM4MSkud/0hZAlGISrgq0pdpXeLx97AyeuURlh1wjev1AGp525jUKZaOdQL48q3ZzBzYl9zsU0aXVSkSiEJUUnkzq9ZRvoR4u9Zki7vVwrPXhPP4VbXZsfZHRj50K8mHDhhd1iWTQBSiCnjbrLx0QzStIn0BWJNwivTcIoOrqh5KKXpcXjaumHs0kVH3dmTr6so1ezKKBKIQVSy7oIQ3thxl0LIEl2pm1SSs7HrFSH93ZgzoyxdzZznduKIEohBVzM/DyuROcXi4WRi+4jA/ulAzq1AfG5M7xtKhjj9fzJ3JrMH9yM3JNrqsCpNAFMIB4gI9mN4lnqZhXry66ShvbT1mdEnVxsPNwnPXRNDvqjB+Xb2c0T2ak5Lwp9FlVYgEohAOUtbMKobbGwa73GSLUopbLg9mXPsYsgtLGHVvB7at+dHosi5IAlEIB7JaFI+0COPORiEA7Dh6mj8z8g2uqvo0q+3DzK7xRPjamN7/Eb58a3aNHleUQBSimpRqzTvb0hj2YyJrEpz7er2LEepjY3KnONrF+7NwzgxmD3mcvNM5Rpd1XhKIQlQTi1KMax9DvWBPZqxPYZ4LNbPycLPQv00Ej7UMY9uqZYx66DZSEw8aXdb/kEAUohoFerkxrn0s3esF8tXeDMavTnKZxSHOLKPWLoZTyQcZdU97fl27wuiy/kICUYhqZrMqnrR3uQvxdsPdqowuqVpdEV62jFqYT9m44tdvv4JR/eHPJYEohEG61Avk2WsiUEqRdKqAjUnOc71eZdX2dWdK5zhuiPXjs9enMXvIEzViXFECUYga4PNd6Uxam8yCncddppmVh5uFgW0jeLRFGFtXLmHsLVdy9PAhQ2uSQBSiBvj31eF0qBPAJ7+nu1QzK6UUPRsGM6ZdDJl5xYzofSvb1600rB4JRCFqAHerheeuCf9LM6sTLrI4BMCV4T7M6BpPbWs+0557mEXvvWbIuKIEohA1xNmzsLV9bfh7WI0uqVqVjyteH+PLJ6++zCvDniY/t3oX3ZVAFKKGuSLch1E3xeButZBTWMLSA5k1ZhbW0TzcLAy6NpI+zUPZ/ON3jO1xJceOJFTb+0sgClGD/fDnSeZsPsbM9akudb3iHY1CGH1TDOl5RQx/8BZ2/LKqWt5bAlGIGuz2hsH0vjKUtYmnGPpDImmnXWdcsXlE2bhimDWfac/2ZtG81x1+pCyBKEQNppTi7sYhjLA3sxq0LIH96a6z6Gy4fVyxbYwfn7wymVeHPU1+Xq7D3k8CUQgn0MrezCo+0IMQL9daSszTzcKQayN5+MpQNv7wHeN6XMGxpESHvJcEohBOIsrfnfEdYgnxtlFSqvl2XwZFJa4zrnhn4xBGtYvm+OkiRva+hZ0b1lT5+0ggCuGEfjuWy9vb0hi+4jAZecVGl1NtWkb4MqNrPCHk8fIzvfn2/TeqdFxRAlEIJ9Qiwoeh10eSmFXAwGUJ7HOhZlYRfvZxxSgfFvxnIm/c0YSCvKrZfwlEIZzUtTH+TOkch4dV8dKKwy616KyXzcIL10XS+4pQfj6czbgezUhLPlzp15VAFMKJxQd6Mr1LPC3CvQn3sxldTrVSSnF3kxBG2mfgR/S+hd83/ly51zTqCvh6IV56Ztd4Q95bCDNbfuAk10T7EuDpOrPRqdmFTFqbRNKpQko1W7XWrS7ldeQIUQgTSTtdxFvbjjF4eQIHM12nmVX5uGJ5M69LJYEohImE+diY1DGWEg1Df0hkTaLrjCt626z0vjK0Uq8hgSiEydQP8WJml/iyZla/pPDxzuNGl+Q0JBCFMKHyZlbd6gUS4etudDlOw3VGXYVwMTar4qnW4Wcerz+STZS/O7EBHgZWVbNV6AhRKdVNKbVPKXVAKTXsb7a5Rym1Wym1Syn1cdWWKYSojKKSUt799RhDlie6VDOri3XBQFRKWYHXgO5AY6CXUqrxOdvUB14ErtNaNwH6O6BWIcQlslktTO4UR7S/O5PWJvPp7ydcppnVxajIEeLVwAGt9UGtdSHwCdDznG36Aa9prTMBtNZpVVumEKKyanmXzUC3j/fn450nmPJzMiWlEopnq8gYYhRw5KzHScA152xzOYBSah1gBcZorZdWSYVCiCrj4Wbh+TYR1A3y5GR+MVaLMrqkGqWqJlXcgPpAOyAaWKOUaqa1Pnn2Rkqpx4HHAUK9ZT5HCCMopbitYfCZx/tO5JFbVEqLCB8Dq6oZKnLKnAzEnPU42v7c2ZKARVrrIq31IWA/ZQH5F1rrN7XWrbTWrfxd6LYiIWqyT34/wbjVR/hqT7rLNLP6OxUJxM1AfaVUHaWUO3AfsOicbb6m7OgQpVQtyk6hD1ZhnUIIB3nhuijaRPsxb/txl2pmdT4XDEStdTHwDLAM2AN8prXepZQap5S6zb7ZMiBdKbUbWAkM0VqnO6poIUTVKV9K68ErarE28RQv/pjIqYISo8syRIXOW7XWi4HF5zw36qyvNTDQ/k8I4WSUUvyrSS3iAz1ZnZCFj801b2KTgTwhxBmto3xpHeULQHpuEdtST9OpbgBKucZstGv+GRBCXNB3+zP576ajvL75qMs0s5IjRCHEeT14RSgWpVi4O53DWYUMuz6KIJO3QJUjRCHEeVktit5XhvLCdZEcysxn4LIEDmcVGF2WQ5k77oUQlXZdrD9Rfu68v+M4IXKEKIRwdfFBnoxuF4OPu5WC4lK+3pNuyvugJRCFEBdlY1IO720/zphVRzhVUGx0OVVKAlEIcVFujPfn+Wsi2HM8j0HLEjlkomZWEohCiIvWoW4AkzvFUlyqGfpDIluSc4wuqUpIIAohLkn9EC9mdo2naW1vIvzM0bdFAlEIccmCvNwYdVMMUf7uaK1ZtDeDnELnvQ9aAlEIUSUSThYwb3saQ5YnknTKOa9XlEAUQlSJOkGejO8Qy+nCEgYvS2RTsvM1s5JAFEJUmSZh3szoGk+kn41Ja5L5Zm+G0SVdFAlEIUSVCvWxMblTHO3i/YkJcK7JFnPfhyOEMISHm4X+bSPPPP7pYBaNw7wI963ZASmBKIRwqJzCEt7bnkap1gy5Lorm4TW3mZWcMgshHMrX3crUznEEe7kxdtURvtmbUWObWUkgCiEcLsLPnSmd47g6ypd3f03j1Y1Ha2QoyimzEKJaeNusDL0+is93pePpZqmRbQkkEIUQ1caiFPc2rXXm8bbUHDzdLDQO9Tawqv8np8xCCENorZn/2wlG/nSYpQcyjS4HkEAUQhhEKcXY9jFcUduHOZuPMWfzUYpKjB1XlEAUQhjG193KiBujubNRMEsPnGTUysPkFxvX4U/GEIUQhrJaFA83D6NukCe70nLxsBo32SKBKISoEW6I8+eGOH8AjmQV8GdmPu3iA6q1BglEIUSN8/XeDH48mMXBjHwebh6G1VI9R40SiEKIGuep1uF4WBXf7Msk4WQBg6+Lwt/D6vD3lUkVIUSN42ZRPN4qnGevDmfX8TwGL0vgWE6h49/X4e8ghBCXqNNlgcQEePDVnnSCvRwfV3KEKISo0RrU8mLYDdHYrBZyCkv4Zm8GpQ66D1oCUQjhNFYlZPHur2lMWptMblHVN7OSQBRCOI0e9YN4/KrabE3JYcjyRJJPVe24ogSiEMJpKKXocXkQ49rHcKqghCHLE9iVlltlry+BKIRwOs1q+zCjazyNQr2I9Ku6tgQSiEIIpxTmY2PkTTEEeblRUqpZuDu90vdBy2U3Qgint+t4Lh/tOM7Piacq9TpyhCiEcHpX1PZh1E3RnCqo3MyzBKIQwhRaRvoy99a6lXoNCUQhhGnYrJWLNAlEIYSwMywQ84tKKSmteW0IhRCuy7BATM4uZNTKw6TnFhlVghBC/IVhgRjmY+OP9Hz6L62eZX2EEOJCDAtEPw8rM7vF07FOAGE+NqPKEEKIMyoUiEqpbkqpfUqpA0qpYf+w3V1KKa2UalWR143296BPizCUUhzNKWTMyiOknZZTaCGEMS4YiEopK/Aa0B1oDPRSSjU+z3Z+wPPAxksp5Gh2EfvS8+i/5BDrj2RfyksIIUSlVOQI8WrggNb6oNa6EPgE6Hme7cYDU4D8SymkeYQPs7rFE+nnzss/J/PmlqMUlhjXn1UI4XoqEohRwJGzHifZnztDKdUSiNFaf1+ZYsJ93ZncKY6eDYL4/o+TfLk7ozIvJ4QQF6XSizsopSzATKBPBbZ9HHgcINT7/G9tsyoebVmbFhG+NAr1AuB0YQk+7o7vuCWEcG0VOUJMBmLOehxtf66cH9AUWKWUSgDaAIvON7GitX5Ta91Ka93K3/Ofs7hFhA+ebhbyi0t54YdEXt2YSkEll/YRQoh/UpFA3AzUV0rVUUq5A/cBi8q/qbXO0lrX0lrHa63jgQ3AbVrrLVVRoM2iaBPtx4qDWQxensDhrIKqeFkhhPgfFwxErXX8fa7iAAAZeElEQVQx8AywDNgDfKa13qWUGqeUus3RBVotit5XhjKmXQxZ+SUMWpbA8j9Poh3UdUsI4bqUUcFSL8RLz+waf1E/k5lXzMz1KeQVlfJy5zjcLMoxxQkhnFbPBXu3aq0rdC30uZxqxewgLzfGtIvhdFEJbhZFdkEJx08XUTfY0+jShBAm4HTLf1ktCn+Pshz/YEcaQ35I5Pv9mXIKLYSoNKcLxLP1vjKU5uHevLn1GJN/TiansOobVwshXIdTB6K/hxsjbozm0RZhbEnOof+SQxzKvKQbZYQQwrnGEM9HKUXPhsE0CvXi7W1pBF7g+kYhhPg7Tn2EeLbLQ7yY0in2TI/WD3ccJyu/2OiyhBBOxDSBCGVHiwAHM/P5Zm8Gzy9NYOex0wZXJYRwFqYKxHL1Q7yY2iUOLzcLo1Ye4ZOdJ6R/ixDigkwZiAB1gzyZ2TWeG+P8WfD7Cf67KdXokoQQNZypZyC8bBYGtI3kitrexAR4AKC1PnNqLYQQZzN1IJbrWDfwzNfvbEvDZlU8cEWo3PonhPgL054yn0+p1hSWar7ck8FLKw5L/xYhxF+4VCBalOLp1uEMvjaSI1kF9F9yiA1J0r9FCFHGpQKx3A1x/szsGk+4nzszf0nhpFyvKITARcYQzyfCz50pnWI5mFlAoKcbWmtO5pcQ5OWy/0mEcHkueYRYzma10KBWWd+WlYdO8dR3B1mTcMrgqoQQRnHpQDxbs9rexAd6MGN9Cv/dJP1bhHBFEoh2oT42JnaM5e7GIfz4p/RvEcIVSSCepbx/y2h7/5aU7EKjSxJCVCOZQTiPFhE+vHFrXbxtZb2gt6bk0CjU68xjIYQ5yRHi3ygPv5N5xbz8czKDliVwMEMWnxXCzCQQLyDQ3tiqoFhL/xYhTE4CsQKahHkzu3s8V9r7t0xbl0KphKIQpiNjiBVU3r/lm70ZFJVqLLJijhCmI4F4ESxKcUejkDOPt6ee5tDJfHo2DJaAFMIE5JS5EtYnZTNv+3EmrE6S/i1CmIAEYiU82ao2T7SqzY5jufRfmsDvablGlySEqAQJxEpQSnFz/SCmdYnD000x8qfD7E/PM7osIcQlkjHEKlA3yJMZXeNZcTCL+sGeAJSUaqyyIrcQTkWOEKuIt83KrQ2CUUpxNKeQZxYfZFtqjtFlCSEuggSiAxSXamwWC2NXJfHB9jSKpQWqEE7BsEBMzS7kN5M2kY/292Balzi6XBbAF3syGL7iMMelf4sQNZ5hgVikbIxalWzaJvIebhb+fXUEg66NJPFkAV/tSTe6JCHEBRg2qRJZpx4xlzVgweIv+c2rAUOijply+f4b4/ypH+x5Zt+Ony4i0NOKzSqjFULUNIb9X2mxWHhq/GweHzWNP3Zu5fklh9h+1Jyn0BF+7ni6WSgu1YxZdYShPySSKmstClHjGHqYopSi3e33MeHD7/CNuowxK48w/7fjpjyFBnCzKB66MpRjp4sYsDSBtYnSv0WImqRGnLdFX9aA8R9+y4233cNnu9IZfiCI9FxzTkJcE+3H7G51iAv0YPovKby2KZXCEunfIkRNUCMCEcDTy5snxszgyXGzOLhrB8+vzjTtdXzl/VvubBRMwskCFHIBtxA1QY0JxHI33nI3E+YvJrBWGGNXJfHhDnOeQrtZFA83D2NSxzhsVkV2QQmrE7Jk8VkhDFTjAhEgqk49xn+wiPZ39GLh7nRG/HSYEyY9hbZZy44Ov9+fycz1qczekEpekZxCC2GEGhmIAO6eXvQbOZV/T3yFg6ct9F+SwJYUc55CA/yrSQi9mtViTeIpBi5L4FCm9G8RorrV2EAsd133O5g4fzHB8Q0YvzqJt2ztTHkrnNWiuK9pLcZ3iCW/uJQhyxPZlJxtdFlCuJQaH4gAEXF1GTvvazre/SDfffAGL5n4VrimYd7M7hbPdbF+1A/2MrocIVyKUwQilJ1C931pMs+9/DqH86z0X3rItEdQAZ5uDGgbSZCXGyWlmtnrU9h3QtZZFMLRnCYQy7XpcisT5y+mVt0mTFyTzLvbjlFUYr5T6HLpecXsOp7Liz8m8tWedOn2J4QDOV0gAoTH1mHsvK/ocm8fvtmXyYsrEjmWY85b4cJ8bMzqVofWUb5l/VvWJHGqQPq3COEIFQpEpVQ3pdQ+pdQBpdSw83x/oFJqt1LqN6XUCqVUXNWX+lc2dw/6DB1P/2lzSS5wp/+qdNYfMecptK+7lWHXR/H4VbXZcTSXyWuTjS5JCFO6YCAqpazAa0B3oDHQSynV+JzNfgVaaa2vABYCU6u60L9zdcebmfjxYsJj43n552Te3HqMIhPeCqeUosflQUzrHEfflrUBKCrRprxoXQijVOQI8WrggNb6oNa6EPgE6Hn2BlrrlVrr8pZzG4Doqi3zn9WOjmPMu1/S/f6+fL8/k6E/HjbtajJ1gz2pZ+/bMm97GmNWHSEzT06hhagKFQnEKODIWY+T7M/9nb7AkvN9Qyn1uFJqi1JqS3ZmRsWrrAA3mzu9B49hwIy3OJpdyMBlCaw7bO7VZOoEerD3RF7Z0mmp5lw6TYjqVKWTKkqpB4FWwLTzfV9r/abWupXWupVfUHBVvvUZrdt3Y9JXPxPZ4EqmrkvhjS1HTbuaTKfLApnRNZ4ATytjVh0x7X3fQlSXigRiMhBz1uNo+3N/oZTqBAwHbtNaF1RNeZcmNDKGUe8spEfvJ1jyx0kG77CRYtJT6NgAD6Z3iafTZQF8uy+DNJNesC5EdahIIG4G6iul6iil3IH7gEVnb6CUagHMpSwM06q+zIvnZnPngQEjGDTrXdKPJjNwZRprTLogq4ebhWeujuDVm+sQ4eeO1poDGXIvtBAX64KBqLUuBp4BlgF7gM+01ruUUuOUUrfZN5sG+AKfK6W2K6UW/c3LVburburM5AXLiKnXkBm/pPD6pqMUFJvzFLq2rzsA65OyGbQsgbdNftG6EFVNGbX+Xt3GV+iJ8xdX2/sVFxXx+ZzpfDvvdeIDPRhyXSTR/h7V9v7VqaiklHnbj/Pd/kzqBXsy+NpIIvzcjS5LiGrRc8HerVrrVpfys055p8qlcLPZ6PXci7zw6gek48PAn46x6lCW0WU5hM1qod9VtRl2fRSp9hl3s160LkRVcplALNf8uvZM/mQpdRo2ZdaGVGblXmnaU+i2MX7M7l6HmAAPlHQpEOKCXC4QAYLDIhg+91N6PvoMqxd9xuDlCRzOMnRi3GHCfGy83CmWNtF+AKw8lEXSKXPuqxCV5ZKBCGB1c+PeZ4Yy9L8fctLix+BlCaw4eNLoshzCYj88LCgu5YMdxxm4NIGfDppzuECIynDZQCx3RdubmPzJUi5r0YZXNh5lenZT8k16Cu3hZmF6lzjqh3jxn42pzFqfIv1bhDiLywciQFBoOC/NWcCd/frz8/dfMGhZAoknzXlaGeJtY1z7GHo1LevfMnh5gmnv5BHiYkkg2lmsVu5+ahAvvv4x2bYABi9P4Ic/T5qyLajVorivWS3GtY+hW71A3K3yayAESCD+j6bXXM/kBUupf9V1/HfTUWatN29b0Ga1fbi1Qdk95duPnmbqumROF5YYXJUQxpFAPI/AWmG8+NpH3P3UINYcyWHAlhISTN4WNDW7kPVHshmwNIE/0qV/i3BNEoh/w2K1cme//oyY+wl5OdkM+SGRZQfMeQoN0L1+EJM7xVGqNcN+TOSbvRmm3Vch/o4E4gU0uqotkz9ZRsOrb+T1zUeZ8UsKuUXmPK1sWMuL2d3r0CrSl3d/TWNjco7RJQlRrSQQKyAguBYvvPoB9z4zlHXJuQxcmsBBk64mU96/ZfiNUVwT5Qsg44rCZUggVpDFYqHno88wYu6nFHiHMOSnFL6Oe9CUp5VKKa6O8kMpxbGcQp749iCf7Tohi88K0zMsEI+nJJGw93ej3v6SNWx5DZMWLKPJ1dfx3ssjmLouxdRHUH4eVlpE+DD/txOMlf4twuQMW/7LzWbTFouVPkPH0/6OXobUUBmlpaV8/8FcPn1tCmFeFgZfG0n9EC+jy3IIrTU/HMzira3H8HKzMLBtJM0jfIwuS4jzcsrlv6LqXk7jVm2xuTvnmoQWi4Vb+zzFqLcXUuQbytCfUvh2nzlnZpVSdLkskOld4vH3sLLW5M27hOsydIHYCR99j7IvPLBh+bfUjomnTqNmhtRTGdknM3lj9EB+XfsjbaJ9efaaCHzdrUaX5RAFxaVowNPNQtKpAjysFkJ9bEaXJcQZTnmECJwJw+KiIj57fRqj+9zO8k/nOd1Rll9gEINnv8sDA0eyOTmHAUsT2HfCnBc3e7hZ8HSzoLVm9oZUBiw9xKZkWXxWmEONmGV2s9kY897XNLvmBuZNGcl/hj5FbrZznZYppejx4OOMnreIUv/avLgy2dQXNyulGNg2klAfGxPXJPOO9G8RJlAjAhHAPyiYQbPfpdfzw9mycinDH7iZvNPOd2FwvWYtmLRgCS1u6Mi7v6YxaW0y2QXmnIWO9HNnSuc4bq4fyKJ9mbz4Y6LMQgunVmMCEewTFQ8/yai3F9Lu9vvw8vE1uqRL4usfyIDpb9F78Bi2peYwYOkh9pr0FNrdauGJVuEMuz4KX3eracdOhWuoUYFY7vIrW9Hz0WcA2L9jK68Me5rT2c61wrNSiu7392X0+99CUCQv/ZjIl3vSKTXpKXTbGD9Gt4vGZlVkF5Tw4Y7jpu1VI8yrRgbi2ZIP7mfzT0t46f6bObh7h9HlXLTLmjRn0seLuarDzby//TjjkqI5VWDO08rySbKtKTks3J3OkOWJ0r9FOJUaH4jt7+jFyLcWUlpSzOg+d7B0wbtON1Hh4xfA81PfoM/Q8ezcuJbn151m9/Fco8tymHZ1Ahh1UzSZ+cUMWpbATyZt9yrMp8YHIsDlV17F5AVLufLam/hg2mg2LP/W6JIumlKKLvf2Yey8r7HZ3Bm+4jALd5v3FPqqSF9md4unXrAn/9mQyjd7M4wuSYgLMvTC7InzF1/Uz2it2bD8W67p1AOL1Uphfh7uns53u1xuTjZvj3+BDT98R8sIH/q3iSDA083oshyipFTz1d4MOtYJIMjLDa31mVNrIRzBaS/MvlhKKdp2vQ2L1UpGWioDb7+RJR+/43Sn0N6+fjz78us8+tIkdqYX8/zP2exKM+cptNWiuLtxCEFebpSUasatTmLpgUyn+8yEa3CqQDybu6cXdRpfwYfTxzBrcD9yTjlXT2WlFJ3u7s3Y97/Bw8ub4SuT+CjoFlMvsZVfXIrWMGfzMab9Yu5VgoRzctpA9PUPZOCMt+k9aDS/rl3BS726c2Dnr0aXddHiGzRh4vzFtO16G5+/Pp2xq45w0qQXN/u4WxnVLpqHrgxl/ZFsBi6T/i2iZnHaQAT7tX4PPMbod74EYMnHbxtc0aXx8vHl3xNeod/Iqew+WUL/pYf47ehpo8tyCItS3NU4hEkdYykp1cxan2rqo2LhXJxqUuWf5Jw6iUVZ8Pbz50RqMp7e3vgGBFXZ61eXw3/s4ZWhT5OacIB7m4ZwT5NaWC3mnITILighM7+Y2AAPikpKySvW+HvInS6iclxmUuWf+PoH4u3nj9aaV4Y9zUu9uvPHb9uMLuuixdZvxISPvuO6Hnfxye/pjF55hAyTnkL7eViJDShbD/P9Hcfpv/SQqa/PFDWfaQKxnFKKh18Yi7JYGPfYXXz/4Vynm9H09Pbh6fGzeWLMDPadKqX/kkNsN+kpdLl28QHYLIrhKw7z+a4Tpr0+U9RspgtEsN8ut2AJLW/oxPxZE5je/1Gnuxca4Kbb7mHCR9/jG3UZY1YlMf+346Ydb6sX7MmsbvFcF+PHR7+dYMxK804uiZrLlIEIZbfL9Z/+Jg8NGUvm8WO4ubkbXdIlia57ORM+/I4bb/0Xn+1KZ+RPh0nPLTK6LIfwtlkZdG0k/24dzsHMAk7JZTmimplmUuWflBQXY3VzIz/3NOsWf0X7O+/HYnG+vwVrv/uCd8cOwN0viIHNvWkZ4ZzLo1VEXlEpXraylbl/OZJNm2g/004uiaolkyoXYHUruy1u9aLPeGfSi8wY0Jfsk5kGV3XxbrjlLiZ89hOBtUIZuyqJD7anmfYU2stW9qu5+3geU9elMOKnw5ww6ZGxqDlcIhDLdbm3Dw+/MI6dG9bw0v3d2L9ji9ElXbSoOvUY/8G3tL+jF1/syWD4CnMHRZMwbwa0jeBgZgH9lxxiS7LzraIunIdLBaJSiq73PcKY977C6ubGuMfuZt2Sr4wu66K5e3rRb+RU/j3xFQ7lWnh+dYapg6JdfAAzu8ZTy9vG+DVJfLLzhNElCZNyqUAsV7fxFUyav4QbetzF5Ve2NrqcS3Zd9zuYOH8xIWERjF+TxLxf0yg26Sl0lL87U7uU9W+JD3LOXt6i5nPJQATw9vPniTEzCI2MRmvN2xOGsW/7ZqPLumgRcXUZO+9rOt79IF/tzWDYHl+OnzbnKXR5/5Y20X4ALP4jk3WHnas7o6jZXDYQz5aVfpxdm35mfL9/sWje65SWOlcvEHdPL/q+NJlnJ79G0p/7eH7VCTYmmbtXcqnWrE08xdR1Kbyx+SiFJc71mYmaSQIRCKwVxsT5i2ndoTufvDKZac/34VSm863w3LbrbUycv5jQyBgmrTV3r2SLUoxrH8vtDYNZcuAkL0j/FlEFJBDtvP38ee7l13nkxYns2rSOqc895HS3/AGEx9ZhzHtf0eXePmW9klckciyn0OiyHMJmVTzSIoyRN0VzIreYIcsTOWXSHtiierjEhdkXK2Hv7xTk59GgeWtKS0pAKae8kHvjj9/z5rghKGXh+ea+Z8bezCg9t4gdx3LpUCcAKGtdIBdyuyaHX5itlOqmlNqnlDqglBp2nu97KKU+tX9/o1Iq/lKKqSniGzalQfOy2eeFc2cy7bmHOZWZbnBVF++aTj2Y9PESwmPimLw2mTlcZ9pT6BBv25kw3J56mueXHCLhZL7BVQlnc8FAVEpZgdeA7kBjoJdSqvE5m/UFMrXW9YBZwJSqLtQoIWER7N6ynhfv68qerRuMLuei1Y6OY/S7X9KtV1+WLniHYT8mctSkp9Dl3KyKnMIShixPZNmBk0459CGMUZEjxKuBA1rrg1rrQuAToOc52/QE3rd/vRDoqEzSWq3j3Q+e6Xsy4Yl7+fqdV51uFtrm7sFDQ8YwYMZbpBR5MGBpgqkvV2ka5s3sbnVoHOrF65uPSv8WUWEVCcQo4MhZj5Psz513G611MZAFhFRFgTVBfIMmTPjoe9p0voUv5s4iNeFPo0u6JK3bd2PygiVENriSt7elkV/sXMF+MQK93BjdLobe9v4t646Y+zIkUTUuOKmilLob6Ka1fsz+uDdwjdb6mbO2+d2+TZL98Z/2bU6c81qPA4/bHzYFfq+qHamBagFmvsfMzPtn5n0D8+9fA631Jc0gVqQ7ejIQc9bjaPtz59smSSnlBgQA/zMLobV+E3gTQCm15VJngpyB7J/zMvO+gWvs36X+bEVOmTcD9ZVSdZRS7sB9wKJztlkEPGz/+m7gJy0j2UIIJ3PBI0StdbFS6hlgGWAF3tVa71JKjQO2aK0XAe8AHyqlDgAZlIWmEEI4lYqcMqO1XgwsPue5UWd9nQ/86yLf+82L3N7ZyP45LzPvG8j+/S3D7lQRQoiaxvnuRxNCCAdxeCCa/ba/CuzfQKXUbqXUb0qpFUqpOCPqvBQX2reztrtLKaWVUk41c1mR/VNK3WP//HYppT6u7horowK/m7FKqZVKqV/tv583G1HnpVBKvauUSrNf8ne+7yul1Cv2ff9NKdWyQi+stXbYP8omYf4E6gLuwA6g8TnbPA28Yf/6PuBTR9ZkwP61B7ztXz/lLPtXkX2zb+cHrAE2AK2MrruKP7v6wK9AkP1xmNF1V/H+vQk8Zf+6MZBgdN0XsX83Ai2B3//m+zcDSwAFtAE2VuR1HX2EaPbb/i64f1rrlVrrXPvDDZRdx+kMKvLZAYyn7N51Z1tJoSL71w94TWudCaC1TqvmGiujIvunAX/71wFASjXWVyla6zWUXdHyd3oCH+gyG4BApVTEhV7X0YFo9tv+KrJ/Z+tL2V8tZ3DBfbOfhsRorb+vzsKqSEU+u8uBy5VS65RSG5RS3aqtusqryP6NAR5USiVRdhXJs9VTWrW42P83gQpediMqTyn1INAKuMnoWqqCUsoCzAT6GFyKI7lRdtrcjrIj+zVKqWZa65OGVlV1egHztNYzlFJtKbuWuKnW2rw3uV+Ao48QL+a2P/7ptr8aqiL7h1KqEzAcuE1r7Szr3F9o3/woux99lVIqgbJxmkVONLFSkc8uCViktS7SWh8C9lMWkM6gIvvXF/gMQGu9HvCk7D5nM6jQ/5v/w8EDn27AQaAO/z+w2+Scbf7NXydVPjN6wLaK968FZYPb9Y2ut6r37ZztV+FckyoV+ey6Ae/bv65F2SlYiNG1V+H+LQH62L9uRNkYojK69ovYx3j+flKlB3+dVNlUodeshqJvpuwv65/AcPtz4yg7WoKyv0qfAweATUBdo/9DV/H+/QgcA7bb/y0yuuaq2rdztnWqQKzgZ6coGxbYDewE7jO65irev8bAOntYbge6GF3zRezbAiAVKKLsSL4v8CTw5Fmf3Wv2fd9Z0d9NuVNFCCHs5E4VIYSwk0AUQgg7CUQhhLCTQBRCCDsJRCGEsJNAFEIIOwlEIYSwk0AUQgi7/wN3YqWbGMMCjQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Apply the decision function on the grid input matrix\n", "y_tanh_grid = net2.predict(X_grid)\n", "# Reshape the array to recover the squared shape\n", "y_tanh_grid = y_tanh_grid.reshape(rows.shape)\n", "# Plot the decision\n", "plot_decision_contours(y_tanh_grid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Congratulations you did it !" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Acknowledgements\n", "\n", "The idea and the code for this tutorial have been for the most part inspired by the video \"Deep Learning Madness\" https://youtu.be/o64FV-ez6Gw by [Joel Grus](https://twitter.com/joelgrus)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 2 }