notebooks/Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ML@ Cezeaux - by Emille Ishida\n",
    "\n",
    "### Machine Learning Tutorial\n",
    "\n",
    "# Section 1.b - Supervised learning: classification\n",
    "by [Emille Ishida](https://www.emilleishida.com/)\n",
    "\n",
    "### *Take home message 2: It is crucial to understand the algorithm you are using*\n",
    "\n",
    "**Goal:** 1. Get acquainted with a basic machine learning algorithm    \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2. Discuss hyper parameters  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3. Write your classifier  \n",
    "\n",
    "**Task**: Star-Galaxy Classification  \n",
    "\n",
    "**Data**: Clean data resulting from [Notebook 1](EDA_SDSS_answers.ipynb)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;~9100 objects (lines - depends on how you extracted the outliers)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11 features (columns)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Features we are interested in:  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "$ug$: u-g SDSS color  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "$gr$: g-r SDSS color   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "$class$: source classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import some basic libaries \n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by loading the data and checking if our variables of interest are given.\n",
    " \n",
    "<div class=\"alert alert-info\"> \n",
    "PS1: as we already cleaned this data, we will not perform an extensive EDA here, but remember that you should always make similar tests to those shown in <a href='EDA_SDSS.ipynb'>Notebook 1</a> every single time... and again, just to be sure.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data\n",
    "data = pd.read_csv('../data/SDSS_star_galaxy_clean.csv')\n",
    "\n",
    "# check columns\n",
    "data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the data we want is there, but there is a lot of other stuff as well.  \n",
    "Select only the columns that are interesting to you. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make a new data frame with the columns you need\n",
    "\n",
    "\n",
    "# confirm remaining columns\n",
    "data_use.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take a look at the data you just separated\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Separate your data samples\n",
    "\n",
    "All supervised learning tasks are made of (at least) 4 phases:  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1) **train**: training samples (requires features and labels)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2) **optimize**: validation sample (requires features and labels)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Repeat 1-2 until you are happy with the results**  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3) **evaluate results**: test sample (requires features and labels)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4) **predict**: target sample (there are only you features, you should believe your code at this point)  \n",
    "      \n",
    "        \n",
    "        \n",
    "This means that your labelled sample needs to be divided in at least 3 samples: training, validation and test.  \n",
    "So, in order to continue we must construct these samples.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# separate 60% for training and 20% for validation and 20% test\n",
    "# WARNING: there is probably a smarter way to do this\n",
    "X_train, X, y_train, y = train_test_split(data_use[['ug', 'gr']], data_use['class'], test_size=0.4, random_state=1)\n",
    "X_test, X_validation, y_test, y_validation = train_test_split(X, y, test_size=0.5, random_state=1)\n",
    "\n",
    "# check your samples (size, features, etc.)\n",
    "print('training sample:    ', X_train.shape, y_train.shape) \n",
    "print('validation sample:  ', X_validation.shape, y_validation.shape)\n",
    "print('test sample:        ', X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Build a Nearest Neighbor classifier\n",
    "\n",
    "As a start, we will construct a k-nearest neighbor algorithm (kNN) with k = 1.  \n",
    "This means that the class of a test point will be given by the class of its nearest neighbor.  \n",
    "We can describe this strategy as:  \n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; For all objects in the unlabelled sample:  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1) Calculate the distance to all points in the training sample  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2) Identify its closest neighbor  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3) Assign the class of its nearest neighbor  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a 1-nearest neighbor classifier\n",
    "\n",
    "def my_NearestNeighbor(train_features, train_labels, unlabelled_features):\n",
    "    \"\"\"\n",
    "    Classify an unlabelled using a k=1 nearest neighbor algorithm.\n",
    "    \n",
    "    input: train_features - array, dim=(number of objects, number of features)\n",
    "           train_labels - array, dim=(number of objects, 1)\n",
    "           unlabelled_features - array, dim=(number of objects, number of features)\n",
    "           \n",
    "    output: estimated classes for all lines in test_features\n",
    "            array, dim(number of objects, 1)       \n",
    "    \"\"\"\n",
    "\n",
    "    # calculate distances\n",
    "   \n",
    "    \n",
    "    # assign for each element in the test sample the class of its nearest neighbor\n",
    "\n",
    "    \n",
    "    return \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimate the classes of all objects in the test sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_estimate = my_NearestNeighbor( )\n",
    "\n",
    "# quick look in the first 10 estimated classes\n",
    "class_estimate[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4:  Evaluate your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a classifier and already applied it, let's quantify the results.  \n",
    "Create a metric function that calculates the fraction of correct classifications. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def metric(estimated_classes, true_classes):\n",
    "    \"\"\"\n",
    "    Calculate the fraction of correct classification results. \n",
    "    \n",
    "    input: estimated_classes, array - dim=(number of objects,)\n",
    "           true_classes, array - dim=(number of objects, )\n",
    "           \n",
    "    output: fraction of correct classifications\n",
    "    \"\"\"\n",
    "    \n",
    "    # get the number of objects in the sample\n",
    "\n",
    "    \n",
    "    # get the number of correct classifications\n",
    "\n",
    "    \n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate metrics\n",
    "accuracy = metric(class_estimate, y_validation)\n",
    "\n",
    "print('accuracy: ', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Optimize the classifier\n",
    "\n",
    "The procedure used above aimed to illustrate how you can construct your own classifier, but it is not very practical. Whenever possible, we should make use of available libraries.  \n",
    "\n",
    "Let's reproduce what we did below using [scikit-learn](https://scikit-learn.org/stable/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import neighbors \n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# determine number of neighbors\n",
    "nn = 1\n",
    "weights = 'uniform'              # this can be 'uniform' or 'distance'\n",
    "\n",
    "# create an instance of the classifier\n",
    "classifier = neighbors.KNeighborsClassifier(nn, weights=weights)\n",
    "\n",
    "# train (or fit) the classifier\n",
    "classifier.fit(X_train, y_train)\n",
    "\n",
    "# predict the classes of the te st samtple\n",
    "class_estimate_sklearn = classifier.predict(X_validation)\n",
    "\n",
    "# calculate metrics\n",
    "accuracy_sklearn = accuracy_score(class_estimate, y_validation, normalize=True)\n",
    "\n",
    "print('accuracy give by sklearn: ', accuracy_sklearn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using scikit-learn allows us to easily manipulate the parameters of our algorithm.  \n",
    "Before we can move forward, we need to optimize these results to the best of our abilities.  \n",
    "Ask yourself: **Can the results above be improved ?**  \n",
    "\n",
    "*Try changing the number of neighbors and weights and see how the results change*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Only go to the next step once you are satisfied with the results!!!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Calculate final results\n",
    "\n",
    "In order to ensure some generality to the results of the trained classifier, you must always report its performance in an independent data set, which was not used for training. The aim of the `test` sample is to mimic the results which you would find in the target sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict the classes of the test sample\n",
    "\n",
    "\n",
    "# calculate metrics\n",
    "accuracy_test = accuracy_score(class_estimate_test, y_test, normalize=True)\n",
    "\n",
    "print('accuracy calculated on test sample : ', accuracy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is the accuracy of your optimized classifier.. and the numbers you should report when quoting any output from it!\n",
    "\n",
    "However, this number is not the reason your classifier was built!  \n",
    "Now that you have a working classifier your desired output is a list of classes for each object in a completely new sample for which no labels exist.  \n",
    "\n",
    "So let's read still another data set for which no labels are known."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data\n",
    "data_target = pd.read_csv('../data/SDSS_star_galaxy_target.csv')\n",
    "\n",
    "# check features\n",
    "data_target.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice that for saving time, this data set has already been cleaned. In a realistic situation, all the pre-processing performed in the labelled data which does not involve the labels themselves should also be applied to the target sample!**\n",
    "\n",
    "Now, let's use our classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the classifier you optimized above to estimate the classes for the target sample\n",
    "class_estimate_target = \n",
    "\n",
    "# quick look into the first classes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can now say that you have a new catalog of stars and galaxies, given by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_estimate_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### whose classes can be trusted to the level of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(round(100*accuracy_test), '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can know use this catalog to do your science!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "PS2: In this notebook used only 2 features due to time constraints, but feel free to try it with all collumns. Try accessing, for example, how your results change when you add more information. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------------------------------------------------------------------------------\n",
    "Summary:\n",
    "\n",
    "## Machine Learning Model    \n",
    "\n",
    "**Task**:  Identify stars and galaxies based on their photometric measurements.  \n",
    "\n",
    "       input: 2 different colours in SDSS broad-band filters: u-g and g-r\n",
    "       output: estimated classes\n",
    "       \n",
    "**Task Category**:   Classification\n",
    "\n",
    "**Data**: Extract from SDSS-DR14 as available through [Kaggle](https://www.kaggle.com/lucidlenn/sloan-digital-sky-survey#Skyserver_SQL2_27_2018%206_51_39%20PM.csv)  \n",
    "        \n",
    "        2 Features, x \n",
    "              u-g       u - g colour in SDSS system\n",
    "              g-r       g - r colour in SDSS system\n",
    "    \n",
    "        1 response variable (label), y    \n",
    "            class\n",
    "            \n",
    "**Machine Learning category**:  \n",
    "\n",
    "        Supervised Learning\n",
    "\n",
    "**Set of possible samples**:  \n",
    "\n",
    "        ~9100 objects\n",
    "        \n",
    "**Set of possible labels**:\n",
    "\n",
    "        2 classes: STAR or GALAXY\n",
    "        \n",
    "**Learner**:\n",
    "    \n",
    "        1 - Nearest neighbor    \n",
    "        \n",
    "**Loss function**:\n",
    "\n",
    "        1 - Distance-based"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}