Workshop Eager mode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dYUH0qf8Dqkm"
   },
   "source": [
    "# Goal: Graph to Eager mode migration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6hH4wd81A_P8"
   },
   "source": [
    "## Introduction\n",
    "In this workshop, we implement and train a feed-forward neural network (also known as an \"MLP\" for \"multi-layer perceptron\") on a dataset called \"Fashion MNIST\", consisting of small greyscale images of items of fashion.\n",
    "\n",
    "## Learning Objectives \n",
    "* Understand how to use **Tensorflow Eager** and **Keras Layers** to build a neural network architecture\n",
    "* Understand how a model is trained and evaluated\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NxSD3yPDfEIj",
    "outputId": "6c6beb36-83c9-4658-c4d0-08dbdbbc6a0f"
   },
   "outputs": [],
   "source": [
    "#@title Imports (RUN ME!) { display-mode: \"form\" }\n",
    "from __future__ import absolute_import, division, print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.eager as tfe\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sU33KeDUDomX"
   },
   "source": [
    "## The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nL0dFs-DD6sM"
   },
   "source": [
    "In this practical, we use the Fashion MNIST dataset consisting of 70,000 greyscale images and their labels. The dataset is divided\n",
    " into 60,000 training images and 10,000 test images. The idea is to train a **classifier** to identify the class value (what type of fashion item it is) given the image. We train and *tune* a model on the 60,000 training images and then evaluate how well it classifies the 10,000 test images that the model did not see during training. This task is an example of a **supervised learning** problem, where we are given both input and labels (targets) to learn from. This is in contrast to **unsupervised learning** where we only have inputs from which to learn patterns or **reinforcement learning** where an agent learns how to maximise a reward signal through interaction with its environment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3lEM3aBa2N63"
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9VRxcYWHfZJ5"
   },
   "outputs": [],
   "source": [
    "# Tensorflow has convenient modules for loading a number of standard datasets\n",
    "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
    "(train_and_validation_images, train_and_validation_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
    "\n",
    "text_labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Z_Ks31qEfUM"
   },
   "source": [
    "### Construct a validation set\n",
    "Next, we remove 10,000 images and labels from the training set to use as a validation set. We will come back to the validation set later! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dZKEjjGM8I4t"
   },
   "outputs": [],
   "source": [
    "# Construct a validation set from the last 10000 images and labels from \n",
    "# train_and_validation_images and train_and_validation_labels\n",
    "validation_images = train_and_validation_images[-10000:, :, :]\n",
    "validation_labels = train_and_validation_labels[-10000:]\n",
    "\n",
    "# Construct a training set from the first 50000 images and labels.\n",
    "train_images = train_and_validation_images[:50000, :, :]\n",
    "train_labels = train_and_validation_labels[:50000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fbWk7qJOEkl6"
   },
   "source": [
    "### What does the data look like?\n",
    "Each image in the dataset consists of a 28 x 28 matrix of greyscale pixels. The values are between 0 and 255 where 0 represents black, 255 represents white and there are many shades of grey in-between. Each image is assigned a corresponding numerical label, so the image in ```train_images[i]``` has its corresponding label stored in ```train_labels[i]```. We also have a lookup array called ```text_labels``` to associate a text description with each of the numerical labels. For example, the label 1 is associated with the text description \"Trouser\".\n",
    "\n",
    "The 2 cells below demonstrate how to visualise the input data. Make sure you understand what's happening, particularly how the indices correspond to individual items in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 280
    },
    "colab_type": "code",
    "id": "PgfvNeX2gcAL",
    "outputId": "4b853a99-0389-4dc8-d163-0714ad00507b"
   },
   "outputs": [],
   "source": [
    "# We use the Matplotlib plotting library to visualise an image selected at random from the training set \n",
    "plt.figure()\n",
    "random_index = np.random.randint(0, len(train_images))\n",
    "plt.imshow(train_images[random_index], cmap='gray')\n",
    "plt.colorbar()\n",
    "numerical_label = train_labels[random_index]\n",
    "text_description = text_labels[numerical_label]\n",
    "plt.title('True Class: {} (\"{}\")'.format(numerical_label, text_description))\n",
    "\n",
    "plt.gca().grid(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 589
    },
    "colab_type": "code",
    "id": "-utfMlxhhL3b",
    "outputId": "18781e96-bf98-4a18-8e3e-3149fac0df80"
   },
   "outputs": [],
   "source": [
    "# Another view, showing 25 randomly selected images at a time\n",
    "plt.figure(figsize=(10,10))\n",
    "for i in range(25):\n",
    "    plt.subplot(5,5,i+1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid('off')\n",
    "    \n",
    "    img_index = np.random.randint(0, 50000)\n",
    "    plt.imshow(train_images[img_index], cmap=plt.cm.gray)\n",
    "    plt.xlabel(text_labels[train_labels[img_index]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S6U63sTpGYvg"
   },
   "source": [
    "### Preparing the data with TensorFlow\n",
    "At the moment, our training data consists of two large tensors. The images are stored in a tensor of shape $[50000, 28, 28]$, consisting of all the $28 \\times 28$ images matrices stacked together. The labels are stored in a 1D vector of shape $[50000]$. We wish to train a model using **mini-batch stochastic gradient descent**. In order to do so, we need to shuffle the data and split it into smaller (mini-)batches. We also convert the data from numpy arrays to TensorFlow Tensors.\n",
    "\n",
    "\n",
    "In order to do this batching (and shuffling) we will use the Tensorflow [Dataset API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset), which is a set of simple reusable components that allow you to build efficient data pipelines. Data is said to \"stream\" through the pipeline, meaning that when something at the output of the pipeline wants data, the pipeline will provide that data as soon as it has enough, rather than waiting to process all the data. This allows you to easily build pipelines that work on large datasets without having to load it all into memory in one go!\n",
    "\n",
    "We build this pipeline step-by-step:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HO8Z6SImg_7W"
   },
   "source": [
    "We being by defining the ```batch_size``` hyperparameter of our model. This hyperparameter controls the sizes of the mini-batches (chunks) of data that we use to train the model. The value you use will affect the memory usage, speed of convergence and potentially also the performance of the model. It also interacts with the *learning rate* used in gradient descent. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x7VwqreOhbfv"
   },
   "outputs": [],
   "source": [
    "batch_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3CaT9bWFheRw"
   },
   "source": [
    "The first component we add will group the image and label tensors together into a tuple and then split them into individual entries - ie. 50000 tuples containing a (28, 28) dimensional image and 1D label associated with that image. The following line adds this splitting component to the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7wITDBK8h5Z6"
   },
   "outputs": [],
   "source": [
    "train_ds = tf.data.Dataset.from_tensor_slices((train_images, train_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VNVG8Kb3h_CW"
   },
   "source": [
    "The next thing we do is apply a map operation. This lets us run an arbitrary function on each element. The function we provide returns the image values divided by 255 and converted ('cast') to a float and the label converted to a 32-bit integer.\n",
    "\n",
    "**NOTE**: \"Lambda\" functions are just one-line, anonymous functions. In this case, it defines a function that takes arguments x and y (before the colon), and outputs their manipulated values (after the colon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4fcE7eTviBOi"
   },
   "outputs": [],
   "source": [
    "# Divide image values and cast to float so that they end up as a floating point number between 0 and 1\n",
    "train_ds = train_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gm5iOdU0iD_0"
   },
   "source": [
    "Then we add a **shuffle** component. This returns a random element from the pipeline. Can you spot a potential problem here? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bvr5r8k6iJGI"
   },
   "outputs": [],
   "source": [
    "# Shuffle the examples.\n",
    "train_ds = train_ds.shuffle(buffer_size=batch_size * 10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OlGI65QIjOgp"
   },
   "source": [
    "The final component in our pipeline is the batch component. This just requests `batch_size` elements from the previous pipeline component, groups them together into a single tensor and returns that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PB8CZrsmjR7Z"
   },
   "outputs": [],
   "source": [
    "# Now \"chunk\" the examples into batches\n",
    "train_ds = train_ds.batch(batch_size)  \n",
    "\n",
    "# The output of this pipeline will be tuples of tensors containing images and labels. \n",
    "# The images will be of shape (batch_size, 28, 28) and the labels of shape (batch_size, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v04ibRFSALQu"
   },
   "source": [
    "## Define the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KItI--nMHFRQ"
   },
   "source": [
    "In this section we'll build a classifier. A **classifier** is a function that takes an object's characteristics (or \"features\") as inputs and outputs a prediction of the class (or group) that the object belongs to. It may make a single prediction for each input or it may output some score (for example a probability) for each of the possible classes. Specifically, we will build a classifier that takes in (a batch of) 28 x 28 Fashion MNIST images as we've seen above, and outputs predictions about which class the image belongs to. \n",
    "\n",
    "For each (batch of) input images, we will use a **feed-forward neural network** to compute un-normalised scores (also known as **logits**) for each of the 10 possible classes that the image could belong to. We can then **classify** the image as belonging to the class which receives the highest score, or we can quantify the model's \"confidence\" about the classifications by converting the scores into a probability distribution. \n",
    "\n",
    "A feed-forward neural network consisting of $N$ layers, applied to an input vector $\\mathbf{x}$ can be defined as:\n",
    "\n",
    "\\begin{equation}\n",
    "\\mathbf{f_0} = \\mathbf{x} \\\\\n",
    "\\mathbf{f_i} = \\sigma_i(\\mathbf{W_if_{i-1}} + \\mathbf{b_i}) \\ \\ \\ i \\in [1, N]\n",
    "\\end{equation}\n",
    "\n",
    "Each layer has a particular number, $m_i$, of neurons. The parameters of a layer consist of a matrix $\\mathbf{W_i} \\in \\mathbb{R}^{m_i \\times m_{i-1}}$ and bias vector $\\mathbf{b_i} \\in \\mathbb{R}^{m_i}$. Each layer also has a non-linear activation function $\\sigma_i$. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tHy4bjbTVCzV"
   },
   "source": [
    "### Aside: Activation functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ox7VKs3BVCFV"
   },
   "source": [
    "\n",
    "\n",
    "Activation functions are a core ingredient in deep neural networks. In fact they are what allows us to make use of multiple layers in a neural network. There are a number of different activation functions, each of which are more or less useful under different circumstances. The four activation functions that you are most likely to encounter are, arguably, [ReLU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/ReLU), [Tanh](https://www.tensorflow.org/api_docs/python/tf/keras/activations/tanh), [Sigmoid](https://www.tensorflow.org/api_docs/python/tf/keras/activations/sigmoid), and [Softmax](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Softmax). \n",
    "\n",
    "Read more about activation functions [here](https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bDRoliDeAShd"
   },
   "source": [
    "### Configure the feed-forward neural network\n",
    "We configure the feed-forward neural network part of our classifier using the [Keras Layers API](https://www.tensorflow.org/api_docs/python/tf/keras/layers). This API consists of various reusable building-blocks that allow us to define many different neural network architectures (similar to how we defined a data pipeline earlier!). \n",
    "\n",
    "Here we use the [Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) component which allows us to wrap together a sequence of layers. An important point to note here is that we are **configuring** our neural network architecture as a pipeline. We can think of the resulting ```model`` variable as a *function* that takes a batch of images as inputs and outputs a batch of logits. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cDPrTMSahRBu"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential( #Define your model layers here\n",
    "\n",
    "\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W3AU_94YP1st"
   },
   "source": [
    "The following summary shows how many parameters each layer is made up of (the number of entries in the weight matrics and bias vectors). Note that a value of ```None``` in a particular dimension of a shape means that the shape will dynamically adapt based on the shape of the inputs. In particular, the output shape of the ```flatten_input``` layer will be $[N, 784]$ when the batch of inputs passed to the model has shape $[N, 28, 28]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "fSW4V3HuQd1-",
    "outputId": "f605f661-f91e-4ee5-ee7f-1df1d691038b"
   },
   "outputs": [],
   "source": [
    "# Summary of your model \n",
    "### Your code should be here :\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "After running the summary of your model, it should represent the following architecture : <img src=\"workshop.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VwWoSlpDMzNZ"
   },
   "source": [
    "### Define the loss\n",
    "As we did in the previous practical, we need to specify a loss function for our classifier. This tells us how good our model's predictions are compared to the actual labels (the targets), with a lower loss meaning better predictions. The standard loss function to use with a **multi-class classifier** is the **cross-entropy loss** also known as the \"negative log likelihood\". Suppose we have a classification problem with $C$ classes. A classifier is trained to predict a probability distribution $p(y | X_i)$ for each input $X_i$ from a batch of $N$ examples. The vector $p(y|X_i)$ is $C$ dimensional, sums to one, and we use $p(y|X_i)_c$ to denote the $c$th component of  $p(y|X_i)$. The true class for example $i$ in the batch is $y_i$ and we define the indicator function $\\mathbb{1}[y_i=c]$ to be 1 whenever $y_i = c$ and $0$ otherwise. This classifier has a cross-entropy loss of\n",
    "\n",
    "$- \\frac{1}{N}\\sum_{i=1}^N \\sum_{c=1}^C log( p(y|X_i)_c) \\mathbb{1}[y_i=c]$\n",
    "\n",
    "**NOTE**: The indicator is one for the true class label, and zero everywhere else. So in that sum, the indicator just \"lifts out\" the $log(p(y|X_i))$ values for all true classes. So the above expression is minimised (note the negative at the front) when the model places all its probability mass on the true labels for all the examples. Remember that  log(1)=0 , thus the closer all probabilities of $y_i = c$ are to one, the lower the loss will be and the better the model will be performing.\n",
    "\n",
    "\n",
    "Fortunately we don't need to write this function ourselves as Tensorflow provides a version called \n",
    "\n",
    "```tf.nn.sparse_softmax_cross_entropy_with_logits```. \n",
    "\n",
    "**NOTE**: This function actually computes the cross entropy loss directly from the un-normalised logits, rather than from the probability distribution for numerical stability.\n",
    "\n",
    "By the way, for training data in which the labels are themselves distributions rather than exact values, this definition of cross-entropy still works, where the indicator function is replaced with the corresponding probability of each class for that example. This might be important when we are not sure whether the training data has been labelled correctly, or when the data was labelled by a human who gave their answer along with a degree of confidence that the answer was correct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7OHi-HftJy9_"
   },
   "source": [
    "## Train the model\n",
    "Now that we have our data, data processing pipeline, our neural network architecture and the corresponding loss that we want to minimise, we need to **train** the model using batched stochastic gradient descent. We do this in multiple **epochs**, which is a single iteration through the entire training dataset. Briefly, in each epoch we loop over all the batches of images and labels, and for each batch we perform the following steps:\n",
    "* Get the **predictions** of the model on the current batch of images\n",
    "* Compute the **average loss** values across the batch, telling us how good these predictions are / how close they are to the true targets.\n",
    "* Compute the **gradient of the average loss** (or the average gradient of the losses in the batch) with respect to each of the model's parameters: This tells us the direction to move in \"parameter space\" to decrease the loss value\n",
    "* **Adjust the parameters** by taking a small step in the direction of each component of the gradient (where the learning rate controls the size of the step)\n",
    "\n",
    "During training we also track some metrics, such as the loss and accuracy to see how well the classifier is doing. Note that the cell below may take a few minutes to run! \n",
    "\n",
    "### Hint: Computing gradients in Eager mode\n",
    "\n",
    "In graph mode, TensorFlow builds up a \"computation graph\" which captures all operations of the model and their dependencies. For training, the gradient can then be computed by traversing backwards from every node through its dependents and applying the \"chain rule\" of differentiation. \n",
    "\n",
    "However, in Eager mode, we don't have the concept of the computation graph anymore. Operations are performed imperatively (in the order in which they were executed). TensorFlow therefore uses a mechanism called the \"GradientTape\" for computing gradients in Eager mode. Basically, the gradient tape records the order of all operations as they are executed, and can then be \"run backwards\" (traversed from last to first operation) for computing the gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5hpiGc40hYkK"
   },
   "outputs": [],
   "source": [
    "# Create an optimizer\n",
    "# The optimizer is responsible for controlling the learning rate\n",
    "optimizer = \n",
    "\n",
    "step_counter = tf.train.get_or_create_global_step()  # Just a variable that keeps track of how many training steps we've run\n",
    "num_epochs = 50  # The number of epochs to run\n",
    "\n",
    "# Lists to store the loss and accuracy of every epoch\n",
    "epoch_losses = []\n",
    "epoch_accuracies = []\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    # Tensorflow provides a convenient API for tracking a number of metrics during training/evaluation\n",
    "    loss_avg = \n",
    "    accuracy = \n",
    "\n",
    "    # Loop over our data pipeline\n",
    "    for step, (image_batch, label_batch) in enumerate(train_ds):\n",
    "\n",
    "    # Initialise a GradientTape to track the operations\n",
    "    with ... as ....:\n",
    "        # Compute the logits (un-normalised scores) of the current batch of examples \n",
    "        # using the neural network architecture we defined earlier\n",
    "\n",
    "        logits = \n",
    "        \n",
    "        # Compute the cross-entropy loss of the classification outputs on this batch\n",
    "        loss_value =  \n",
    "        \n",
    "        # Compute the average loss over the batch\n",
    "        loss_value =  \n",
    "\n",
    "        # Add current batch loss to our loss metric tracker - note the function call semantics\n",
    "        \n",
    "        \n",
    "        # Compare most likely predicted label to actual label\n",
    "        accuracy(...................................)\n",
    "\n",
    "    # Play the tape backwards and get the gradient of the loss of the current batch\n",
    "    # Note we're now outside the scope of the with-block above\n",
    "    grads = \n",
    "    # Use the optimizer to apply the gradients to the model parameters along with\n",
    "    # its internal learning rate\n",
    "    optimizer.apply_gradients(zip(........, ...........), global_step=step_counter)\n",
    "\n",
    "    # Get the average loss and accuracy for the epoch\n",
    "    epoch_loss = loss_avg.result()\n",
    "    epoch_losses.append(epoch_loss)\n",
    "    epoch_accuracy = accuracy.result()\n",
    "    epoch_accuracies.append(epoch_accuracy)\n",
    "    print(\"Epoch {:03d}: Loss: {:.3f}, Accuracy: {:.3%}\".format(epoch, epoch_loss, epoch_accuracy))\n",
    "\n",
    "# Plot the loss for all epochs using Matplotlib\n",
    "plt.figure()\n",
    "plt.plot(range(num_epochs), epoch_losses)\n",
    "plt.title('Loss vs epochs')\n",
    "\n",
    "# Plot the accuracy for all epochs using Matplotlib\n",
    "plt.figure()\n",
    "plt.plot(range(num_epochs), epoch_accuracies)\n",
    "plt.title('Accuracy vs epochs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zMwZVX1T3zWP"
   },
   "source": [
    "## Testing\n",
    "\n",
    "After 50 epochs of training on our training set, we obtain a loss of around 0.061 and accuracy of 97.7%. That's pretty good, right? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SmBt_X4B4g3-"
   },
   "outputs": [],
   "source": [
    "# We need to apply the same pre-processing to the test set as we did to the training set\n",
    "# Since we don't need batching or shuffling, we can do this directly instead of \n",
    "# building a tf.Dataset pipeline\n",
    "\n",
    "tf_test_images = tf.convert_to_tensor(test_images, dtype=tf.float32) / 255.0\n",
    "tf_test_labels = tf.convert_to_tensor(test_labels, dtype=tf.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "j0BaJidP46Bm",
    "outputId": "a2cb8a2e-7dd7-4eee-8f7a-9838cecd414a"
   },
   "outputs": [],
   "source": [
    "test_accuracy = tfe.metrics.Accuracy()\n",
    "\n",
    "test_logits = model(....)\n",
    "\n",
    "# Compute the average cross-entropy loss of the classification over the entire test set\n",
    "test_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
    "    logits=test_logits, labels=tf_test_labels)  \n",
    "test_loss = tf.reduce_mean(test_loss)\n",
    "\n",
    "# Compare predicted labels to actual labels\n",
    "test_accuracy(tf.argmax(test_logits, axis=1, output_type=tf.int32), tf_test_labels)\n",
    "\n",
    "print('Completed testing on', tf_test_images.shape[0], 'examples...')\n",
    "print('Loss: {:.3f}, Accuracy: {:.3%}'.format(test_loss, test_accuracy.result()))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "colab": {
   "collapsed_sections": [
    "tHy4bjbTVCzV",
    "SIUsSjNwVk_n",
    "ot-FjimzV6op"
   ],
   "name": "Practical 1: Deep Feedforward Networks",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}